search

microsoft / farmvibes-ai

FarmVibes.AI: Multi-Modal GeoSpatial ML Models for Agriculture and Sustainability
https://microsoft.github.io/farmvibes-ai/
MIT License
685 stars 118 forks source link

Chip_size Parameter Issue: Farm Size Dependency in spaceeye_interpolation Workflow #180

Open click2cloud-sagarB opened 4 months ago

click2cloud-sagarB commented 4 months ago

In which step did you encounter the bug?

Workflow execution

Are you using a local or a remote (AKS) FarmVibes.AI cluster?

Local cluster

Bug description

Dear Farmvibes Team,

During the execution of the data_ingestion/spaceeye/spaceeye_interpolation workflow, we encountered an issue with the chip_size parameter "ValueError(\nValueError: dim_size=421 cannot be smaller than chip_size=448. Please consider reducing the step/chip size or increasing the input geometry." Specifically, the workflow failed for a 115.5-acre farm while a 0.5-acre farm ran successfully. So many times it is observed that larger farms fail due to the smaller chip_size, whereas smaller farms are unaffected. Could you please let us know the precise cause of this problem?

Can your team please look into this matter and if possible can your team expose step_size parameter just like crop_cycle workflow so that we can decrease step size from 448 to less than 448 to avoid chip_size issue.

I've attached the log files detailing the errors encountered with both the 115.5-acre and 0.5-acre farms for your review.

log_files.zip

Steps to reproduce the problem

No response

rafaspadilha commented 3 months ago

Hi, @click2cloud-sagarB. Thanks for opening this issue.

Looks like that issue happens when the input geometry falls very close to the border of a tile and yields a region that is smaller (in image size) than our chip size. When that happens, we try to increase the size of the input geometry/window in order to run SpaceEye, but because we are close to the border of the tile, sometimes that is not possible.

I've opened PR #183 with a fix for that issue. I've tested with the geometries that you shared with me and SpaceEye completed successfully. Once the PR merges, let me know if you are able to test it in your geometries.

rafaspadilha commented 3 months ago

PR merged on dev. @click2cloud-sagarB, let me know if you are able to try building the cluster with the dev branch.

I'll close this issue for now. Feel free to reopen if the problem persists.

click2cloud-sagarB commented 3 months ago

Hello, @rafaspadilha I noticed that the dev cluster is having some problems. I was unable to determine whether the chip_size issue had been resolved for the provided geometries because the sapeceeye pipeline is failing for dev cluster with the reason " INVALID_PROTOBUF : Load model from /opt/terravibes/ops/resources/cloud_models/cloud_model2_cpu.onnx failed:Protobuf parsing failed. " When I am running same farm with building main branch cluster it runs sucessfully. I am attaching the log file for failure and ss for same. andrew_dev_cluster_9ed5227e-0b84-47aa-bae9-fd86cbffd015.json

1

rafaspadilha commented 3 months ago

Hi, @click2cloud-sagarB. Let me see if I can reproduce this error.

Did you deleted your old cluster and recreated a new one using the dev branch?

click2cloud-sagarB commented 3 months ago

Yes @rafaspadilha , I deleted old cluster and recreasted a new one using dev branch. Edit: Task spaceeye.preprocess.cloud.cloud fails in dev cluster