Use prefect to run ML jobs within latent space explorer

[taxe10]

• [x] Launch sleep jobs through prefect
• [x] Assign a name to the dimension reduction job, use a user input for this + a timestamp
• [x] Launch dimension reduction jobs through prefect
• [x] Use either podman or docker to launch jobs
• [x] If time allows, add dropdowns? to go back to previous runs

Note: @taxe10 needs to add a docker flow to mlex_prefect_worker

[runboj]

[runboj]

In "TRAIN_PARAMS_EXAMPLE ", why there are two items in "params_list"? each item in "params_list" is an individual job? @taxe10

[taxe10]

If you are referring to:

TRAIN_PARAMS_EXAMPLE = {
    "flow_type": "podman",
    "params_list": [
        {
            "image_name": "ghcr.io/mlexchange/mlex_dlsia_segmentation_prototype",
            "image_tag": "main",
            "command": 'python -c \\"import time; time.sleep(30)\\"',
            "params": {
                "io_parameters": {"uid_save": "uid0001", "uid_retrieve": "uid0001"}
            },
            "volumes": [f"{RESULTS_DIR}:/app/work/results"],
        },
        {
            "image_name": "ghcr.io/mlexchange/mlex_dlsia_segmentation_prototype",
            "image_tag": "main",
            "command": 'python -c \\"import time; time.sleep(10)\\"',
            "params": {
                "io_parameters": {"uid_save": "uid0001", "uid_retrieve": "uid0001"}
            },
            "volumes": [f"{RESULTS_DIR}:/app/work/results"],
        },
    ],
}

In that use case the training process kicked off 2 jobs, one for training the ML model and one for partial inference. This is to avoid segmenting the entire dataset every time a new ML model is trained. This may be helpful for us if we decide to split the dimension reduction algorithms into "train/inference" processes in the future. For an initial test, I think it's sufficient to test the dimension reduction algorithms by launching a single job.

[runboj]

Try to launch pca job through prefect, nothing was saved in the output dir.