search

Netflix / metaflow

:rocket: Build and manage real-life ML, AI, and data science projects with ease!
https://metaflow.org
Apache License 2.0
8.02k stars 752 forks source link

Conda support for linux-aarch64 #1281

Open DeanKezurer opened 1 year ago

DeanKezurer commented 1 year ago

Hi, I hope you are well. I am currently trying to use Metaflow to build some machine learning pipelines with its AWS compatibility. Due to certain requirements of the models that are being run, I have set up a linux-aarch64 instance in AWS that supports the vCPU and RAM requirements of the model. Furthermore, I have set up a docker image that has Conda pre-installed so that I can run the conda_base package with it. Despite this, the dev team are using local linux-64 workstations to trigger the Metaflow pipelines and as such the Conda dependency file (./metaflow/FlowName/conda.dependencies) is generated with respect to a linux-64 instance. This file is then rendered in the remote environment and causes a conflict as the linux-64 and linux-aarch64 environments do not sync up. Is there a known fix for this issue or is Metaflow not able to run on a remote linux-aarch64 machine, given that the local machine runs linux-64? Thanks, Dean

savingoyal commented 1 year ago

@DeanKezurer we should be able to support linux-aarch64 instances in the next release or so.