mlflow / mlflow-export-import

Apache License 2.0
132 stars 78 forks source link

Runs imported but models not after using import-models #42

Closed jswtraveler closed 1 year ago

jswtraveler commented 1 year ago

Hi, thank you so much for developing this package. I am trying to migrate experiments, runs, and models from our old mlflow server to our new one. I exported the models using export-models --output-dir mlflow_model_output_dir2 --models shortage_lgb_resource_count_3dp_all_21mth,shortage_lgb_open_duty_ct_3dp_all_21mth --export-source-tags True --export-all-runs True sucessfully. I then changed the mlflow uri to our new location and used import-models. The experiment and runs are all imported and when looking at the model registry the model names are both there. However, under the registry/model name there are no versions. In the experiment UI window there are no links to models, but I can find the artifacts along with the "register model" button if I click through the runs into a single runs details. I have attached screenshots of the original model registry for the model I am moving, the new model registry, and the output of the import-models run.

original_mlflow_server_model_registry new_mlflow_server_model_registry terminal_import_model_finished_
amesar commented 1 year ago

I'd need more info. Can you send a zipped of the exported model directory?

jswtraveler commented 1 year ago

I'd need for info. Can you send a zipped of the exported model directory?

Apologies for what might be a basic question. Is there a way to send them without making them publicly available on the issues page?

jswtraveler commented 1 year ago

export_models_single_ex.zip Hi, so so so sorry for the late reply. I've attached a zip file that has the output. It's a large model so I deleted all but a single run with the artifacts etc but the file structure etc is what the export-models output created.

amesar commented 1 year ago

The zip file contains the output of export-experiments and not export-models. The one experiment was successfully imported.

jswtraveler commented 1 year ago

Apologies, I didn't compress the model folder as well for some reason. I've attached the correct compressed folder this time. I dropped all but one of the run folders like before because of file size of attachments. I've also attached the terminal screenshot that created the folder that I then compressed. Thank you again so much for your help. export-models-terminal-screenshot sdr_export_model_example.zip

amesar commented 1 year ago

I have successfully imported the zip file. I don't see any issues.

If you manually delete runs from an experiment in the export directory, you have to adjust the experiment sdr_export_model_example/experiments/304/manifest.json file and adjust export_info.ok_runs otherwise import will fail.

jswtraveler commented 1 year ago

The deletions were only for uploading to git since the total file size was ~250mb due to artifacts. When I try to use import-models for the original folder without any deletions it imports the experiment, runs, and artifacts but doesn't register the model or any of the versions. But it sounds like you don't see anything wrong with the exported folder structure. I made a workaround to migrate manually in a loop just calling the original server to get each model version then manually registering/re-saving the model versions to the new directory. Thank you for your help.

amesar commented 1 year ago

Cool. I'll close the ticket.