More consistent way of passing model parameters

[dylanmcreynolds]

As we worked to link the new prefect worker to the model, we came across an issue where there are a large number of parameters sent into the model. These are a little hard to track, and they are a mixture of positional parameters and a json tree.

What if we did this?

• Like today, the prefect flow is defined with only several input parameters, but one of them, model_params is defined as a dict
• The worker picks up this dictionary of model parameter and writes them to a python tempfile. It can write it as either yaml or json (I think I'd prefer yaml)
• When the worker launches the podman container, it reads the path of the tempfile and volume mounts it to the container in a known location, that is the same for all MLE containers, something like /config/model_params.yml

@Wiebke @taxe10 @TibbersHao is this flexible enough?

[TibbersHao]

Adding notes from today's group meeting discussion:

For DLSIA training side, the plan to accommodate parameter loading:

1. Refactor the parameter class into individual classes that will be model-specific, and each will inherit from a mother class which will hold general parameters that will be used in all scenarios. Benefits of this way:

• Easy to scale up with future models
• Getting rid of the nested structure that would cause a hard time for dash app to store and access.

2. both train.py and segment.py script will take only one argument, most likely a file path of the json or yml file that dash app generates for the parameters (in a dictionary) in order to execute the job.

Structure of the dictionary (version 1):

{
data_tiled_uri: 
data_tiled_api_key: 
mask_tiled_uri: 
mask_tiled_api_key: 
seg_tiled_uri: 
seg_tiled_api_key: 
uid: 
model_parameters: {
    model_name:
    num_classes: xxx
    ...
    }
}

This will be a discussion topic for Friday's hackathon.

[dylanmcreynolds]

Closing as I believe it's done. Please re-open if I missed something