search

mlflow / mlflow-export-import

Apache License 2.0
132 stars 78 forks source link

Can't import-run in single mode #128

Open FlorentMeyer opened 1 year ago

FlorentMeyer commented 1 year ago

Hello,

Thanks for the great package, which I'm trying to use for the first time.

I would like to copy runs -- logged locally using Pytorch Lightning and MLFlowLogger -- from a local experiment on a machine that can't access our mlflow server (name=NAME, ID=XXX) to our mlflow server where the "same" experiment exists already (name=NAME, ID=YYY). Is that an intended use case?

Process:

  1. export-run on local machine (works)
  2. copy resulting folder to the machine which runs the server
  3. import-run on server machine (having set MLFLOW_TRACKING_URI, giving --experiment-name=NAME and --input-dir)

I'm getting a load of mlflow.exceptions.MissingConfigException: Yaml file '/DATA/CALCUL/mlflow_artifacts/<exp_id>/meta.yaml' does not exist. errors -- basically one for each existing project (they indeed do not exist, maybe because the server was setup very long ago) -- then a final mlflow.exceptions.MlflowException: Experiment 'NAME' already exists.

Leaving the experiment.json in the resulting folder as is, a new folder XXX gets created on the server machine, containing only a meta.yaml file. Modifying experiment.json to match ID YYY, nothing seems to happen (the runs in experiment YYY on the server machine are intact). Both throw the same chain of errors.

Additional info: importing an experiment with a name and an ID that don't exist on the server gives the same error.

Also, I am curious whether this fits into "Importing from a file-based --backend-store-uri implementation is not supported"? Since I'm using Lightning without running a server locally, I'd say it belongs to the "file-based" category?

Could you please help with that? Thanks a lot!

amesar commented 1 year ago

Hey there, sorry for the delay.

where the "same" experiment exists already (name=NAME, ID=YYY). Is that an intended use case?

Yes, this should be OK. If experiment exists import will use it. If t doesn't it creates it.

maybe because the server was setup very long ag

The tool doesn't address MLflow version discrepancies. I would recommend you upgrade your old tracking server to the latest MLflow, and then export it. Every now and then there are major API and schema changes, so you have to run the schema update script. Tools stays current of the latest MLflow version.

Importing from a file-based

File-based is not supported. From General Limitations "Importing from a file-based --backend-store-uri implementation is not supported"

Hope I got all the questions.