Feature: mlflow tags

[tanlin2013]

Background

MLflow offers automatic tracking to the code version, but that requires one to execute the program through MLflow Project.

Should we

• [x] Implement the code versioning tags manually, or
• [ ] Use MLflow Project instead

Pros & Cons

1. MLflow Project can trigger runs in docker environment, but it's not clear in the documentation how one can elaborate features in docker-compose to here, e.g. network mode, docker volume, shm size, etc.
2. MLflow Project support multistep workflows, apart from current candidate Apache Airflow.
3. This means we have to install MLflow at system level.
4. An alternative way is to set MLflow Project environment as system environment, but actually it runs in docker already. But, though in this way we can track git version automatically, docker version is still required to be retrieved manually.

Reference

1. https://mlflow.org/docs/latest/tracking.html#concepts
2. https://mlflow.org/docs/latest/projects.html#building-multistep-workflows

[tanlin2013]

EDITED

Turns out the error still exist, so threading is not an issue. Though at this point is't not harm to

• get git commit hash in prior of run
• joint multiple decorators into one

---

In https://github.com/tanlin2013/mbl/commit/a0a37807182d05023b40b7858a51484359befe00, we have the subprocess to talk with system. But it appears like the threading issue would cause the run been execute twice? Since the background mechanism of ray is not clear, it's probably better to get those tags in prior of trial runs. Then pass them into the experiment runs.

Affecting Parts

def mlflow_auto_tags(func: Callable):
    @wraps(func)
    def wrapper(*args, **kwargs):
        tags = {
            "docker.image.id": "hostname",
            "git.commit": f"git --git-dir {os.getenv('GITDIR')} "
            f"rev-parse --short HEAD",
        }
        mlflow.set_tags(
            {
                k: subprocess.check_output(v.split()).decode("ascii").strip()
                for k, v in tags.items()
            }
        )
        func(*args, **kwargs)

    return wrapper

Error Logs

Traceback (most recent call last):
  File "/home/mbl/mbl/workflow/grid_search.py", line 89, in wrapper
    func(*args, **kwargs)
  File "/home/mbl/mbl/workflow/grid_search.py", line 121, in experiment
    mlflow.log_params(config)
  File "/usr/local/lib/python3.9/site-packages/mlflow/tracking/fluent.py", line 639, in log_params
    MlflowClient().log_batch(run_id=run_id, metrics=[], params=params_arr, tags=[])
  File "/usr/local/lib/python3.9/site-packages/mlflow/tracking/client.py", line 918, in log_batch
    self._tracking_client.log_batch(run_id, metrics, params, tags)
  File "/usr/local/lib/python3.9/site-packages/mlflow/tracking/_tracking_service/client.py", line 292, in log_batch
    self.store.log_batch(run_id=run_id, metrics=metrics, params=params, tags=tags)
  File "/usr/local/lib/python3.9/site-packages/mlflow/store/tracking/rest_store.py", line 309, in log_batch
    self._call_endpoint(LogBatch, req_body)
  File "/usr/local/lib/python3.9/site-packages/mlflow/store/tracking/rest_store.py", line 56, in _call_endpoint
    return call_endpoint(self.get_host_creds(), endpoint, method, json_body, response_proto)
  File "/usr/local/lib/python3.9/site-packages/mlflow/utils/rest_utils.py", line 256, in call_endpoint
    response = verify_rest_response(response, endpoint)
  File "/usr/local/lib/python3.9/site-packages/mlflow/utils/rest_utils.py", line 185, in verify_rest_response
    raise RestException(json.loads(response.text))
mlflow.exceptions.RestException: INVALID_PARAMETER_VALUE: Changing param values is not allowed. Param with key='n' was already logged with value='20' for run ID='d404ec5e9ad341feaade870dfca5af34'. Attempted logging new value '8'.

[tanlin2013]

After hours of try-and-error and a lots of panics, it's finally resolved with ray==1.12.1. The reason is not clear, may have to report this behavior to ray team.