Tracker

Tracker is a command line utility (CLI) for creating machine learning (ML) projects (using a template project structure), creating experiments, and running and tracking trials. The goal of Tracker is to ease the task of creating reproducible research within ML.

The project is created by ML developers for ML developers.

What's the Payoff?

You know when you're improving and when you're regressing
You can go back in time with git hashes linked to each experiment and even trial.
You can analyse differences across runs to better understand results
You can collaborate with colleagues as they know the folder structure
You can share your results with colleagues
You can save and backup trained models
You can automatically optimise your model
You can make guarantees about model performances
You ensure the reproducibility of your work

Using Tracker

Creating a new project

Creating a new project is easy, simply run the following command and follow the instructions.

tracker create [OPTIONS] CONFIGURATION

Tracker will prompt you for a version control location (git/svn/ssh/...) where a repository will be created automatically, and all files generated by this command will be synced to the remote location.

Creating an experiment

By envoking the command below you'll get an experiment configuration file wherein all parameters for the trial should be placed.

tracker experiments create [OPTIONS] NAME

Running a trial within an experiment

When you're ready to run a trial, simply run the command below replacing the NAME with your actual experiment's name and a trial will commence.

tracker experiments run [OPTIONS] EXPERIMENT

Note that by default, Tracker will prompt you for a commit message and push your changes to the user-specified remote location, subsequently obtaining a commit hash which can be stored and linked to the conducted trial.

Help is all around

If you're struggling, don't worry about it. Tracker got your back; simply call for --help at any level of arguments:

$ tracker --help
Usage: tracker [OPTIONS] COMMAND [ARGS]...

  Tracker command line interface.

Options:
  --version       Show the version and exit.
  --verbose       Print debug information
  --logfile PATH  File to be used as a stream for DEBUG logging
  --help          Show this message and exit.

Commands:
  create  Creates a new project based on a CONFIGURATION FILE and pushes it...
  run     Running a new experiment
  [...]

$ tracker create --help
Usage: tracker create [OPTIONS] CONFIGURATION

  Creates a new project based on a CONFIGURATION FILE and pushes it to the
  chosen git repository

Options:
  --help  Show this message and exit.

Installing Tracker

Please note that Tracker is not released to pypi yet. See INSTALL.md for how to install from source.

# Install Tracker
pip3 install dti-tracker

# Enable bash autocompletion for Tracker on boot
echo ""  >> ~/.bashrc
echo "# Bash autocompletion for Tracker"  >> ~/.bashrc
echo 'eval "$(_TRACKER_COMPLETE=source tracker)"'  >> ~/.bashrc
source ~/.bashrc

Tests

You can check whether Tracker is installed correctly by running the included tests.

cd tracker/
make test

Levels of Abstraction

We acknowledge that using Tracker can seem a bit overwhelming to begin with, so here are a few definitions that will help you get started:

Projects: At the highest level of abstraction lies the project which is created by invoking the command: tracker create CONF_FILE.yaml. The project will contain all experiments and trials (definitions below).
Experiments: At the next level we have experiments which denotes specific parameter configurations (or ranges for the automatic hyper-parameter optimisation). An experiment physically consists of a configuration file similar to the one we use to create the project, but instead of containing template URLs, etc. the experiment configuration file contains the specific parameter configuration for the algorithm(s), environment(s), and so on. Experiments are created by invoking tracker experiments create NAME.
Trials: At the lowest level we have trials which denotes a single run of a specific experiment. When we're evaluating probabilistic models we need the ability to conduct multiple repetitions (trials) of the same experiment in order to account for the stochasticity of the methods deployed.

Support

Installation problems? Feature requests? General questions?

read through examples and API documentation
create github issues on the Tracker project

Contributions

We welcome contributions to Tracker by the community! Please refer to the Contribution Guide.

Acknowledgements

This project is developed by the DTI AI Research Team at Center for Robot Technology. Nicolai Anton Lynnerup originally developed the CLI. The CLI is heavily inspired by GuildAI, customised to our needs.

Citing Tracker

For Tracker, please cite Lynnerup et al. (2019).

Lynnerup, N. A., Hallam, J., Hasle, R. (2019). Tools for Reproducible Research in Machine Learning. In IROS 2019 Workshop on 'Reproducibility of Research Benchmarking and Verification of Autonomous Robotic Systems'.

Epilogue

This is the part where I say something thoughtful about how wonderful it is to create reproducible research. And what a wonderful person you are for choosing to follow good experimental practices.

But hey, you made it all the way here, so you already know this stuff.

So if there is something else that you still feel you don’t know, please explore the documentation and/or get in touch with me.

dti-research / tracker

readme