microsoft / farmvibes-ai

FarmVibes.AI: Multi-Modal GeoSpatial ML Models for Agriculture and Sustainability
https://microsoft.github.io/farmvibes-ai/
MIT License
680 stars 118 forks source link

Error while running crop_cycles notebook #76

Closed Regen231 closed 1 year ago

Regen231 commented 1 year ago

Hi Farmvibes Team,

I am trying to run the notebook crop cycles but I keep getting the following error:

eam, timeout, verify, cert, proxies)
    561     if isinstance(e.reason, _SSLError):
    562         # This branch is for urllib3 v1.22 and later.
    563         raise SSLError(e, request=request)
--> 565     raise ConnectionError(e, request=request)
    567 except ClosedPoolError as e:
    568     raise ConnectionError(e, request=request)

ConnectionError: HTTPConnectionPool(host='172.18.0.3', port=32259): Max retries exceeded with url: /v0/system-metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcb48d233a0>: Failed to establish a new connection: [Errno 111] Connection refused'))

I have taken the latest code from your github and installed the cluster as per the instructions in the quickstart.md document. I have done the reinstallation twice already. The workflow starts sometimes but eventually gives this errror. The space on my VM shows the following:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       2.0T  743G  1.3T  38% /
devtmpfs        7.9G  4.0K  7.9G   1% /dev
tmpfs           7.9G  4.0K  7.9G   1% /dev/shm
tmpfs           1.6G  1.5M  1.6G   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/loop0      128K  128K     0 100% /snap/bare/5
/dev/loop1       56M   56M     0 100% /snap/core18/2708

Could you please let me know where I am going wrong?

rafaspadilha commented 1 year ago

Hi, @Regen231. Thank you for using FarmVibes.AI and reporting the issue.

Please, could you provide the output of ~/.config/farmvibes-ai/kubectl get pods ?

Is it failing on a specific task within the crop cycle workflow?

Regen231 commented 1 year ago

Hi @rafaspadilha , Thanks for the response. I get the following output of the command ~/.config/farmvibes-ai/kubectl get pods:

NAME READY STATUS RESTARTS AGE redis-master-0 1/1 Running 4 (59m ago) 5d20h rabbitmq-0 1/1 Running 9 (59m ago) 5d20h terravibes-worker-9fc897475-p7vzw 2/2 Running 20 (59m ago) 5d20h terravibes-rest-api-6f7c8d76d7-5zn7t 2/2 Running 15 (59m ago) 5d20h terravibes-worker-9fc897475-v525s 2/2 Running 16 (59m ago) 5d20h terravibes-cache-647dcc8664-jzlbm 2/2 Running 15 (59m ago) 5d20h terravibes-worker-9fc897475-hr7rj 2/2 Running 15 (59m ago) 5d20h terravibes-orchestrator-7666c96cb-hhkwg 2/2 Running 16 (59m ago) 5d20h

It gives the error when I run this cell of the notebook:

run = client.run(wf_dict, "Crop cycles", geometry=geometry, time_range=time_range)

Thanks and Regards, MH

rafaspadilha commented 1 year ago

Hi, @Regen231. Your pods seem to be working fine from the output of get pods and you have plenty of disk space. The error might be related to a miscommunication between orchestrator and the rest api.

We have a few improvements in the last release on how we set up these components. May I ask you to get the latest version of the repo (git pull) and update the library and images (bash farmvibes-ai.sh update)?

Let me know if the updates fix your issue.

rafaspadilha commented 1 year ago

Closing this issue for now. @Regen231, feel free to reopen it if you are still having problems.