search

Clay-foundation / model

The Clay Foundation Model (in development)
https://clay-foundation.github.io/model/
Apache License 2.0
374 stars 48 forks source link

EPIC: Support running on macOS #161

Closed weiji14 closed 7 months ago

weiji14 commented 8 months ago

To allow Clay to run on not just Linux operating systems, but OSX as well!

This comes from user feedback during DevSeed's Team Week, where we found out that the folks developing the Clay model are predominantly Linux-based, but users were using other operating systems like macOS.

Non-OS specific:

OSX (Intel chips):

OSX-ARM64 (M1/M2/M3 chips):

mikeskaug commented 8 months ago

I would be happy to help with this if there is anything I can do. A few weeks ago I started working on getting things set up and running on my Mac. I was interested in experimenting with the project and my typical approach is to start local. My first goal was to get to the point where I could train and predict on a small data set and then move on from there. I haven't finished but I would be happy to share my approach or issues I've encountered.

I've made a few minor changes but the most significant departure has been to use pip instead of mamba to install requirements. I don't know if that's useful or just a bad idea, but I thought it would be worth a try and it seems to work. (Although I guess I'm not 100% sure until I have the various cli commands running, which currently seem to be failing because of data type support on cpu.)

weiji14 commented 8 months ago

Hi @mikeskaug, thanks for offering to help! If you can post some of the error messages you're getting (open a bug report issue), that would be great for us to keep track. There's been lots of dtype related bugs (e.g. https://github.com/Clay-foundation/model/issues/134) due to different operating systems and GPU/CPU devices being used, and we'd like to iron those out as much as possible!

If you're keen to contribute, we're open to help with improving our documentation, or you can also jump in and review some of our outstanding PRs (e.g. #164 and #166 are macOS related ones). Or if you have a use-case/dataset you'd like to share and collaborate on, we can discuss more too at https://github.com/Clay-foundation/model/discussions/categories/ideas :smile:

ritwikvashistha commented 8 months ago

Hi, I tried using the model on my Mac M1 Pro as well as on Binder. But I ran into the same issue with both of them. When I was trying to install locally, I got an error "The following package could not be------installed: pytorch ~=2.1.0 cuda12 does not exist (perhaps a typo or a missing channel)." With Binder, I got the same error as it was loading. I would appreciate any suggestions on how to resolve this issue.

weiji14 commented 8 months ago

"The following package could not be------installed: pytorch ~=2.1.0 cuda12 does not exist

Ah yes, that's because Mac M1 devices don't have CUDA. You'll need to install from the lock-file instead from #164 which adds support for ARM OSX devices. Gimme a few minutes to merge that PR, and then if you do a git pull on the main branch, you should be able to install with:

mamba create --name claymodel python=3.11 conda-lock=2.5.1
conda-lock install --mamba --name claymodel conda-lock.yml

Which reminds me, I should update the installation instructions on that branch before merging.

weiji14 commented 8 months ago

@ritwikvashistha, I've merged #164 now, so installation macOS ARM devices should work :crossed_fingers:. Can you try following the conda-lock installation instructions I mentioned above (or at https://clay-foundation.github.io/model/installation.html#advanced) and see if it installs on your device? Let us know if you still get any errors!

ritwikvashistha commented 8 months ago

I followed the steps you mentioned and got the following error after the conda-lock line:

"(claymodel) ritwikvashistha@ritwiks-mbp model % conda-lock install --micromamba --name claymodel conda-lock.yml Traceback (most recent call last): File "/Users/ritwikvashistha/micromamba/envs/claymodel/bin/conda-lock", line 10, in sys.exit(main()) ^^^^^^ File "/Users/ritwikvashistha/micromamba/envs/claymodel/lib/python3.11/site-packages/click/core.py", line 1157, in call return self.main(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ritwikvashistha/micromamba/envs/claymodel/lib/python3.11/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/Users/ritwikvashistha/micromamba/envs/claymodel/lib/python3.11/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ritwikvashistha/micromamba/envs/claymodel/lib/python3.11/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ritwikvashistha/micromamba/envs/claymodel/lib/python3.11/site-packages/click/core.py", line 783, in invoke return __callback(args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ritwikvashistha/micromamba/envs/claymodel/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ritwikvashistha/micromamba/envs/claymodel/lib/python3.11/site-packages/conda_lock/conda_lock.py", line 1483, in click_install install( File "/Users/ritwikvashistha/micromamba/envs/claymodel/lib/python3.11/site-packages/conda_lock/conda_lock.py", line 1518, in install _conda_exe = determine_conda_executable(conda, mamba=mamba, micromamba=micromamba) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ritwikvashistha/micromamba/envs/claymodel/lib/python3.11/site-packages/conda_lock/invoke_conda.py", line 59, in determine_conda_executable if determine_micromamba_version(str(candidate)) < LooseVersion("0.17"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: '<' not supported between instances of 'Version' and 'LooseVersion' "

weiji14 commented 8 months ago

Error message looks the same as https://github.com/conda/conda-lock/issues/596, looks like an upgrade to conda-lock>=2.5.3 is needed.

weiji14 commented 8 months ago

Ok, trying to fix that TypeError: '<' not supported between instances of 'Version' and 'LooseVersion' " in #183. Could you try:

mamba env remove -n claymodel
mamba create --name claymodel python=3.11 conda-lock=2.5.6
conda-lock install --mamba --name claymodel conda-lock.yml

Note the new version of conda-lock=2.5.6 is used.

ritwikvashistha commented 8 months ago

I just did that, and I didn't get any errors. But it seems like the dependencies didn't get downloaded since, on trying to run some basic codes, I am getting errors related to missing modules. I think I will just download all those missing modules manually since they don't seem like a lot, but I just wanted to mention the issue for reference.

weiji14 commented 8 months ago

I am getting errors related to missing modules. I think I will just download all those missing modules manually since they don't seem like a lot, but I just wanted to mention the issue for reference.

Could you file a separate bug report (https://github.com/Clay-foundation/model/issues/new) for those missing modules? Would be good for us to keep track of what dependencies are missing and needs to be added to environment.yml/conda-lock.yml.

chuckwondo commented 8 months ago

Ok, trying to fix that TypeError: '<' not supported between instances of 'Version' and 'LooseVersion' " in #183. Could you try:

mamba env remove -n claymodel
mamba create --name claymodel python=3.11 conda-lock=2.5.6
conda-lock install --mamba --name claymodel conda-lock.yml

Note the new version of conda-lock=2.5.6 is used.

Note that you need to activate the env before running conda-lock, so the steps should be the following:

mamba env remove -n claymodel
mamba create --name claymodel python=3.11 conda-lock=2.5.6
mamba activate claymodel
conda-lock install --mamba --name claymodel conda-lock.yml
ritwikvashistha commented 8 months ago

It works now, and I just tried the 'Burn scar analysis using embeddings from partial inputs' tutorial. I was able to reproduce it. I will continue experimenting with it to get embeddings from partial inputs. Thanks, @chuckwondo and @weiji14 for the help and the promptness in your replies!