Netflix / metaflow

:rocket: Build and manage real-life ML, AI, and data science projects with ease!
https://metaflow.org
Apache License 2.0
8.04k stars 752 forks source link

micromamba complain from @pypi_base about package incompatibility in intel macos #1930

Open Shuaijun-Gao opened 1 month ago

Shuaijun-Gao commented 1 month ago

I use decorator @pypi_base and @batch for cloud computing a certain step.

If i understand correctly, @pypi_base utilizes micromamba to test environment compatibility locally first. The problem is when using @batch it will download pip files for linux, and my local computer is intel macos. Then micromamba will complain that some pip wheel files is not supported for my intel macos.

Do you have a work around to solve this? Micromamba only does a dry-run right, is there a way to work around?

savingoyal commented 1 month ago

@Shuaijun-Gao can you help me with the error message?

deepthi-kk commented 1 month ago

Can someone assign this to me? I can take a look.

Shuaijun-Gao commented 1 month ago

It always says greenlet -universal2.whl is not a supported wheel on this platform.

savingoyal commented 1 month ago

We don't create the environment on your local Mac for Linux - we just do a dry run to get a list of dependencies, download those packages, and upload them to S3 if they don't already exist there.

@pypi(packages={"greenlet": "3.0.3"})
@batch 
def start...

this works well for me, but I am on an M1 Mac

Shuaijun-Gao commented 1 month ago

yeah, I tried on my friend's m1 mac, it worked. But on intel mac, the dry run will fail

romain-intel commented 1 week ago

Did you check if all your steps are indeed decorated with @batch?