Come up with a location convention for output files

[jonbinney]

We keep training and test data in the "datasets" folder, and maybe a similar convention for output files would help? At the moment annotations made by the inference script go into the folder for the dataset they were created from, as do text descriptions. Visualization videos go into the object_tracker_0 directory. This works pretty well, but a few thoughts:

• my feeling (not based on any strong reasoning) is the datasets directory should be somewhat "curated". Annotations created by the inference script could go into it, but maybe after we've decided they are known to be good?
• multiple runs of inference (and text gen and visualization) create multiple output files which overwrite each other. I've found myself thinking "did i run visualization and create this video from the newest annotations, or is it from some older run"
• our scripts have (hyper)parameters which we'd like to be able to track

I've been mulling over something like the following:

• we create a "results" directory at the root of the repo,
• when we're going to run some scripts, we create a specific result directory inside of "results" with a unique name
• launch.json passes that results directory into our scripts as the "output" directory (or prepends it to the path of the command line argument that sets the path of each individual output file)

The idea is a bit similar to having "out of tree" builds vs. in-tree. Scripts would also eventually dump files into the output directory which include the (hyperparameters) they were run with.

@adamantivm @alejandromarcu thoughts?

[adamantivm]

I would be fine with this approach, but note that tools like MLFlow at least as far as I undrestand, are exactly for this purpose - to store results, with their corresponding artifacts, input parameters, and results so that they can be more easily organized, understood and looked at later.

I wouldn't be opposed to a PR and some guidance to store the results on our bucket using our own designed criteria, but would have a small preference for trying out one of these tools, maybe Weights & Biases instead - even if they take a bit more work at the beginning, they should help with understanding the state of the art.

I can volunteer for this as a next step if you all are interested in this path - we can discuss it on Monday

[alejandromarcu]

I agree we need a better organization, and I think it would be nice to try Weight and Biases or other tools, or even just see if there are standard ways to do it, so that we are not re-inventing the wheel

[jonbinney]

Sounds good!