search

fieldsoftheworld / ftw-baselines

Code for running baseline models/experiments with the Fields of The World dataset
https://fieldsofthe.world/
MIT License
55 stars 4 forks source link

Clean-up dependencies for pip deployment #3 #36

Closed m-mohr closed 3 weeks ago

m-mohr commented 4 weeks ago

The pip dependencies were a bit messy. Some were unused alltogether, some were installed via conda, some were installed via pyproject/pip directly. Trying to resolve it, but not 100% sure I captured it all correctly.

Also, provides a bit more info in the README.

I ran a couple of commands and it looks like they still work in a fresh environment, but I couldn't test all ftw model commands...

cholmes commented 4 weeks ago

Ah, great - thanks for taking a crack at it. I had wanted to try to see if we could get everything in pip, but we got a bit hung-up on the fact that torchgeo used fiona, and it doesn't include geoparquet support out of the box.

@calebrob6 - how did that resolve? And what's the status of the latest torchgeo? I know we pin to a 0.5 now since 0.6 doesn't work, is there a 0.7 soon that will work?

@m-mohr - I'll try out a fresh build too, I think I can try to test the model commands. Also did you test out ftw data download on windows, with the wget stuff? I want to be sure that does work out of the box - if it doesn't let's make a ticket and swap the library out.

m-mohr commented 4 weeks ago

@m-mohr - Also did you test out ftw data download on windows, with the wget stuff? I want to be sure that does work out of the box - if it doesn't let's make a ticket and swap the library out.

No, I realized I don't have conda installed yet on Windows, so did everything on WSL/Ubuntu. But it looks like wget on PyPi is a pure Python library, i.e. pretty much just a "wget"-like implementation. So I thnk it should be fine...

On the other hand, wget on PyPi seems dead. Last update 9 years ago, so might not be the ideal candidate anyway?!

cholmes commented 4 weeks ago

No, I realized I don't have conda installed yet on Windows, so did everything on WSL/Ubuntu. But it looks like wget on PyPi is a pure Python library, i.e. pretty much just a "wget"-like implementation. So I thnk it should be fine...

Oh great. Yeah, I had the same experience on mac - thought it required, but didn't.

On the other hand, wget on PyPi seems dead. Last update 9 years ago, so might not be the ideal candidate anyway?!

Or it's just a small stable library that doesn't need more? It seems to work pretty great now, the download animations are nice, so I lean towards just sticking with it, at least until we need something new / more. But don't feel strongly, if you have something you want to replace it with. Just want some sort of progress animation, as these files are big enough that you start to wonder if it's actually working...

cholmes commented 4 weeks ago

For model fit I got:

File "/Users/cholmes/miniforge3/envs/ftw-c2/lib/python3.12/site-packages/geopandas/_compat.py", line 64, in import_optional_dependency raise ImportError(msg) from None ImportError: Missing optional dependency 'pyarrow.parquet'. pyarrow is required for Parquet support. " "Use pip or conda to install pyarrow.parquet."

pip install pyarrow worked well. Now doing the full operation to see if it misses anything at the end (will take awhile, since it trains the model and my GPU isn't configured).

(unpack, download worked, and inference started to work - will try to run it all the way through)

m-mohr commented 4 weeks ago

We can keep wget for the time being, I guess...

Thanks for spotting the pyarrow issue, I've added it as a dependency again. Maybe next step is to add some very basic tests that run all things step by step on a small sample to check we don't break anything.

m-mohr commented 3 weeks ago

Keeping this open until I have the tests ready...