[infra] Add MLFlow for tracking pruning/optimization experiments

[nickvazz]

How to track things:

https://www.mlflow.org/docs/latest/tracking.html#id74

import mlflow

mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("/my-experiment")
with mlflow.start_run():
    mlflow.log_param("a", 1)
    mlflow.log_metric("b", 2)

in command prompt

• filepath -> file:///path/to/location/to/save/things
• sqlite -> sqlite:///path/to/location/of/database.db

mlflow server --backend-store-uri  sqlite:///path/to/location/of/database.db --default-artifact-root /folder/to/store/all/artifacts/in/like/plots/or/jsons/or/whatever/else --host 0.0.0.0

mlflow server --backend-store-uri  sqlite:///sqlite/database.db --default-artifact-root /artifact --host 0.0.0.0

List of functions you might use:

• https://www.mlflow.org/docs/latest/python_api/mlflow.html#mlflow.start_run
• http://mlflow.org/docs/latest/python_api/mlflow.html#mlflow.log_metrics
• https://www.mlflow.org/docs/latest/python_api/mlflow.html#mlflow.log_params
• https://www.mlflow.org/docs/latest/python_api/mlflow.html#mlflow.log_artifact

Options for storage:

Local files

• https://www.mlflow.org/docs/latest/tracking.html#scenario-1-mlflow-on-localhost

Sqlite DB

(file-like database not requiring a server - limitation: is write locked so if writing often and quickly, can be slow)

• https://www.mlflow.org/docs/latest/tracking.html#scenario-2-mlflow-on-localhost-with-sqlite

Parameters you might track

Pruning

• pruning method
  • prune.l1_unstructured() -> "l1_unstructured"
  • prune.ln_structured() -> "ln_structured"
• pruning hyper params
  • prune.l1_unstructured(module, name="bias", amount=3) -> {"amount": 3}
  • prune.ln_structured(module, name="weight", amount=0.5, n=2, dim=0) ->{"amount" : 0.5, "n":2, "dim": 0}

    Time & Memory

[nickvazz]

https://docs.conda.io/projects/conda/en/master/user-guide/tasks/manage-environments.html#activating-an-environment

Activating environments is essential to making the software in the environments work well. Activation entails two primary functions: adding entries to PATH for the environment and running any activation scripts that the environment may contain. These activation scripts are how packages can set arbitrary environment variables that may be necessary for their operation. You can also use the config API to set environment variables.

When installing Anaconda, you have the option to “Add Anaconda to my PATH environment variable.” This is not recommended because the add to PATH option appends Anaconda to PATH. When the installer appends to PATH, it does not call the activation scripts.

On Windows, PATH is composed of two parts, the system PATH and the user PATH. The system PATH always comes first. When you install Anaconda for Just Me, we add it to the user PATH. When you install for All Users, we add it to the system PATH. In the former case, you can end up with system PATH values taking precedence over our entries. In the latter case, you do not. We do not recommend multi-user installs.

Activation prepends to PATH. This only takes effect when you have the environment active so it is local to a terminal session, not global.

[jhchang]

import mlflow

mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("/my-experiment")
with mlflow.start_run():
    mlflow.log_param("a", 1)
    mlflow.log_metric("b", 2)
    mlflow.log_artifact("C:/Users/Justin/Documents/SJSU/cs274/DFDC/mlflow_test.py")

[jhchang]

mlflow server --backend-store-uri  sqlite:///sqlite/database.db --default-artifact-root C:/Users/Justin/Documents/SJSU/cs274/DFDC/ --host 0.0.0.0

[jhchang]

[nickvazz]

from https://www.mlflow.org/docs/latest/tracking.html#artifact-stores

The artifact store is a location suitable for large data (such as an S3 bucket or shared NFS file system) and is where clients log their artifact output (for example, models). artifact_location is a property recorded on mlflow.entities.Experiment for default location to store artifacts for all runs in this experiment. Additional, artifact_uri is a property on mlflow.entities.RunInfo to indicate location where all artifacts for this run are stored.

In addition to local file paths, MLflow supports the following storage systems as artifact stores: Amazon S3, Azure Blob Storage, Google Cloud Storage, SFTP server, and NFS.

Use --default-artifact-root (defaults to local ./mlruns directory) to configure default location to server’s artifact store. This will be used as artifact location for newly-created experiments that do not specify one. Once you create an experiment, --default-artifact-root is no longer relevant to that experiment.

To allow the server and clients to access the artifact location, you should configure your cloud provider credentials as normal. For example, for S3, you can set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, use an IAM role, or configure a default profile in ~/.aws/credentials. See Set up AWS Credentials and Region for Development for more info.

Important

If you do not specify a --default-artifact-root or an artifact URI when creating the experiment (for example, mlflow experiments create --artifact-location s3://), the artifact root is a path inside the file store. Typically this is not an appropriate location, as the client and server probably refer to different physical locations (that is, the same path on different disks).

[jhchang]

(icpr2020) C:\Users\Justin\Documents\SJSU\cs274\DFDC>python mlflow_test.py
INFO: '/my-experiment4' does not exist. Creating a new experiment
Traceback (most recent call last):
  File "mlflow_test.py", line 8, in <module>
    mlflow.log_artifact(__file__)
  File "C:\Users\Justin\AppData\Roaming\Python\Python36\site-packages\mlflow\tracking\fluent.py", line 529, in log_artifact
    MlflowClient().log_artifact(run_id, local_path, artifact_path)
  File "C:\Users\Justin\AppData\Roaming\Python\Python36\site-packages\mlflow\tracking\client.py", line 875, in log_artifact
    self._tracking_client.log_artifact(run_id, local_path, artifact_path)
  File "C:\Users\Justin\AppData\Roaming\Python\Python36\site-packages\mlflow\tracking\_tracking_service\client.py", line 271, in log_artifact
    artifact_repo = self._get_artifact_repo(run_id)
  File "C:\Users\Justin\AppData\Roaming\Python\Python36\site-packages\mlflow\tracking\_tracking_service\client.py", line 262, in _get_artifact_repo
    return get_artifact_repository(artifact_uri)
  File "C:\Users\Justin\AppData\Roaming\Python\Python36\site-packages\mlflow\store\artifact\artifact_repository_registry.py", line 102, in get_artifact_repository
    return _artifact_repository_registry.get_artifact_repository(artifact_uri)
  File "C:\Users\Justin\AppData\Roaming\Python\Python36\site-packages\mlflow\store\artifact\artifact_repository_registry.py", line 68, in get_artifact_repository
    artifact_uri, list(self._registry.keys())
mlflow.exceptions.MlflowException: Could not find a registered artifact repository for: c:/Users/Justin/Documents/SJSU/cs274/DFDC/4/223add3a56324cd5b27317a7a97a0939/artifacts. Currently registered schemes are: ['', 'file', 's3', 'gs', 'wasbs', 'ftp', 'sftp', 'dbfs', 'hdfs', 'viewfs', 'runs', 'models']

[jhchang]

(icpr2020) C:\Users\Justin\Documents\SJSU\cs274\DFDC>mlflow server --backend-store-uri  sqlite:///sqlite/database.db --default-artifact-root file:///C:/Users/Justin/Documents/SJSU/cs274/DFDC --host 0.0.0.0

[jhchang]

create a file that sets up the cmd line interface

cli.py

import click

@click.command()
@click.option("--count", default=1, help="Number of greetings.")
@click.option("--name", prompt="Your name", help="The person to greet.")
def hello(count, name):
    """Simple program that greets NAME for a total of COUNT times."""
    for _ in range(count):
        click.echo(f"Hello, {name}!")

if __name__ == '__main__':
    hello()

install ml with :

pip install mlflow --user

Add this directory to the windows environment variables to be able to run mlflow from the command line:

Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script mlflow.exe is installed in 'C:\Users\Justin\AppData\Roaming\Python\Python36\Scripts' which is not on PATH.
arning, use --no-warn-script-location.

create a mlflow file that runs your experiements:

mlflow_test.py

import mlflow

mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("/my-experiment")
with mlflow.start_run():
    mlflow.log_param("a", 1)
    mlflow.log_metric("b", 2)
    mlflow.log_artifact(__file__)

run this command:

mlflow server --backend-store-uri  sqlite:///sqlite/database.db --default-artifact-root file:///C:/Users/Justin/Documents/SJSU/cs274/DFDC/artifact/ --host 0.0.0.0

then run the mlflow script:

python mlflow_test.py

type localhost:5000 into my browser to see the mlflow ui

[nickvazz]

results = [1,2,3,4]
column_names = ['a','b','c','d']

mlflow.log_params({k:v for k, v in zip(column_naames, results)})

[jhchang]

Get this error while trying to write to mlflow. Doesn't happen until the second time I try to write.

[jhchang]

I'll probably make my param entries code look something like this:

a = ['hi', 'hello', 'goodbye', 'hola']
b = ['hi', 'hola']

b_set = set(b)

result = []
for i in a:
    result.append(int(i in b_set))

print(result)