search

microsoft / farmvibes-ai

FarmVibes.AI: Multi-Modal GeoSpatial ML Models for Agriculture and Sustainability
https://microsoft.github.io/farmvibes-ai/
MIT License
680 stars 118 forks source link

Rerun of workflows took significantly longer time. #113

Closed click2cloud-sagarB closed 11 months ago

click2cloud-sagarB commented 1 year ago

Dear Farmvibes Team,

When I attempted to rerun the workflow in timelapse_visulization notebook, it took more than an hour to complete. Earlier same entire workflow only takes less than 5 minutes to rerun.

Attached to this issue, you will find the following files for your reference: log files for: 1) terravibes-worker 2) terravibes-orchestrator

You might also need these details: 1) file system used : Root filesystem is used and is formatted with the "ext4" filesystem type. 2) No of entries under the ~/.cache/farmvibes-ai is 205735.

log_files.zip

rafaspadilha commented 1 year ago

Hello, @click2cloud-sagarB. Thank you for raising this issue and sharing the logs. We have investigated the increase in timeouts since last release's migration to async.io. We have improved the stability of the cluster and haven't experienced similar issues internally. The fix will be available on the next release.

click2cloud-sagarB commented 1 year ago

Thank you @rafaspadilha, I'm glad to hear that the stability of the cluster has been improved, and you haven't encountered similar issues internally. Looking forward to the next release with the fix.

rafaspadilha commented 1 year ago

Hi, @click2cloud-sagarB. Are you still experiencing this issue?

click2cloud-sagarB commented 1 year ago

Hi @rafaspadilha. We are unable to estimate the time required for a complete rerun of workflows since rerun of complete workflow fails due to Cache Retrieval Errors in Workflows since the most recent commit (issue #121).

rafaspadilha commented 1 year ago

Thanks for letting us know. We are working on fixing #121 and will keep you updated.

rafaspadilha commented 12 months ago

Hi, @click2cloud-sagarB. We have just released a new version of FarmVibes fixing the caching issues reported on #121 . Please, could you verify if the problem was solved?

rafaspadilha commented 11 months ago

Closing this issue, as @click2cloud-sagarB reported offline that it was fixed after the release.