Add CLI for manipulating experiments

[andnp]

The preferred way to build an experiment definition is through the python scripting interface, e.g.:

part = DefinitionPart(name='qrc')

part.add_sweepable_property('alpha', (2**-i for i in range(3, 8)))
part.add_sweepable_property('beta', [0.5, 1.0, 2.0])

part.commit()

However, there are several features that are far more convenient to interact with via cmdline.

Tagging

Very useful for tracking deadlines or other major events.

exp tag qrc 2 "icml deadline"
exp tag --part=qrc --version=2 "icml deadline"

Resetting results

Currently, we will reuse results as much as possible (which is a good thing). However, sometimes we find bugs in the code that invalidate prior results. It isn't currently easy to selectively delete these. It also isn't currently possible to not reuse them. This puts the user in a bad position.

exp reset qrc 2
exp reset --part=qrc --version=2

Expected behavior: this would create a new version, v3, that has the exact same configurations as v2 except all new configuration ids.

# v2
alpha  id
0.01    0
0.001   1
0.1     2

# v3
alpha  id
0.01    3
0.001   4
0.1     5

Deleting results

For instance, in the case of a bug where results are fully meaningless or in case the results db becomes too large.

exp delete qrc 2 22
exp delete --part=qrc --version=2 --config-id=22
exp delete --part=qrc --version=2 --all

[panahiparham]

Tagging

One can tag on experiment name, part name, or part version. How do you propose we tackle this? The easiest use case would be tagging an entire experiment after a major deadline with most recent version of all experiment parts.

Suppose atari-online is an experiment made of two parts:

Part Name    Versions
QRC              V0, V1, V2, V3
DQN              V0, V1

Simply tagging the atari-online experiment should tag the latest table version

exp tag atari-online "icml deadline"

How do you propose we save a record of tags? Is this something to be stored in metadata database? I think we want each tag to be associated with at most one version of a part. With this restriction, tags and their corresponding experiment parts can be stored in something like the following:

Tags -> List of parts
aamas-deadline -> [dqn-v0]
icml-deadline -> [qrc-v3, dqn-v1]

Tagged experiments should be easily recoverable for rerun or analysis of results data.

[panahiparham]

I agree with proposal for resetting and deleting results. We just need to discuss how to incorporate experiment-name, experiment-part and version numbers, and come up with reasonable default behaviour for not providing any of them.