create metadata store node

[mdo6180]

MLflow's experiment logging scheme:

Experiments:

• users can create an experiment
• experiment info includes:
  • artifact location: Location where all artifacts for the experiment are stored. If not provided, the remote server will select an appropriate default. (optional parameter from user)
  • name: Experiment name. (required parameter from user)
  • experiment ID: Unique identifier for the experiment. (generate upon creation of experiment)
  • tags: A collection of tags to set on the experiment. Maximum tag size and number of tags per request depends on the storage backend. All storage backends are guaranteed to support tag keys up to 250 bytes in size and tag values up to 5000 bytes in size. All storage backends are also guaranteed to support up to 20 tags per request.

Runs:

• there can be multiple runs per experiment
• run info includes:
  • experiment ID: ID of the associated experiment. (required parameter from user, or automatically determined by system)
  • user ID: Unique identifier for the user executing the run. (required parameter from user)
  • run name: Name of the run. (required parameter from user)
  • start time: Unix timestamp in milliseconds of when the run started. (generate upon creation of run)
  • end time: Unix timestamp in milliseconds of when the run ended. (leave null at the beginning, generate upon termination of run)
  • tags: Additional metadata for run.

Users:

• user info includes:
  • name: the user's name
  • username: a unique name created by the user
  • user ID: Unique identifier for the user (generated upon creation of a new user)

[mdo6180]

Anacostia Metadata Store logging scheme:

Runs:

• difference between Anacostia Metadata Store and MLflow Metadata Store: in MLflow, runs are associated with an experiment, whereas in Anacostia run.
• one run = one execution of the dag
• run info includes:
  • run ID: Unique identifier for the run (generated upon creation of run)
  • user ID: Unique identifier for the user executing the run. (required parameter from user)
  • run name: Name of the run. (required parameter from user)
  • start time: Unix timestamp in milliseconds of when the run started. (generate upon creation of run)
  • end time: Unix timestamp in milliseconds of when the run ended. (leave null at the beginning, generate upon termination of run)
  • tags: additional metadata for run.
  • params:
    • A key-value pair (string key, string value).
    • Examples include hyperparameters used for ML model training and constant dates and values used in an ETL pipeline.
    • A param can be logged only once for a run.
  • metrics:
    • A key-value pair (string key, float value) with an associated timestamp.
    • Examples include the various metrics that represent ML model accuracy.
    • A metric can be logged multiple times.

Users:

• user info includes:
  • name: the user's name (required parameter from user)
  • username: a unique name created by the user (required parameter from user)
  • user ID: Unique identifier for the user (generated upon creation of a new user)

artifacts:

• artifacts are defined by the resource node
• artifact info must include:
  • run ID: Unique identifier of the run that produced the artifact. (determined by anacostia)
  • type: e.g., model, csv file, http response, preprocessed data, etc. (determined by resource node)
  • location: URI where the artifact is located. (determined by resource node)
  • produced by: name of node that created the artifact. (determined by action node)
  • creation time: Unix timestamp in milliseconds of when the artifact was created. (generated upon creation of artifact)

[mdo6180]

example json entry:

{
    "users": [
        {
            "user_id": 1,
            "name": "Minh Quan Do",
            "username": "mdo6180" 
        },
        {
            "user_id": 13,
            "name": "Carlos Alcaraz",
            "username": "calcaraz" 
        },
    ],
    "runs": [
        {
            "run_id": 1,
            "user_id": 13,
            "start_time": 123456789,
            "end_time": 123456789,
            "tags": [
                "experiment": "pipeline v3"
            ]
        },
        {
            "run_id": 2,
            "user_id": 13,
            "start_time": 123456789,
            "end_time": 123456789,
            "tags": [
                "experiment": "pipeline v3"
            ],
            "metrics": [
                {
                    "key": "mae", 
                    "value": 2.5, 
                    "timestamp": 1552550804
                },
                {
                    "key": "rmse", 
                    "value": 2.7, 
                    "timestamp": 1552550804
                },
            ],
            "params": [
                {
                    "key": "model_class", 
                    "value": "LogisticRegression"
                },
            ]
        },
        {
            "run_id": 3,
            "user_id": 13,
            "start_time": 123456789,
            "end_time": None,
            "tags": [
                "experiment": "pipeline v3"
            ]
        }
    ]
}