Find what to log

[ngarneau]

Either in the command line, csv, or in Sacred

[TortillasAlfred]

For the ablation studies, I will need the information below in logs, after each* iteration of 3.3 :

• Objective function value
• Dictionary accuracy (what they report in their results)

At the end of each experiment :

• Total runtime of the exp
• Total runtime of each procedure section (3.2, 3.3 and 3.4)
• Total steps before convergence in step 3.3

* I say to collect at every step but I do not actually know how many iterations the code has to go through. If a single run of the method takes 50k steps for example, I will defintely be OK with those metrics only being collected every 100 steps or so. The idea is that I would enjoy having something between 100 or 1000 data points per experiment for possibly function plotting.

[TortillasAlfred]

Moreover, since standard deviation is a key point in reproducibility analysis, it would be best to be able to group all randomly seeded experiments for a given hyperparameter config under a same folder/file, something like :

project
└───{exp1_name}
└─────run{random_seed}
│    |    iter_stats.json
│    |    exp_stats.json
...