Modifications in `experiment_history` files after working through tutorials

[hayesall]

After working through the Jupyter notebooks, my local git directory listed a large number of files as having been modified:

modified:   tutorials/experiment_history/21-05-22_17.05.05/outputs_in_csv/chor-005.csv
modified:   tutorials/experiment_history/21-05-22_17.05.05/outputs_in_csv/chor-006.csv
modified:   tutorials/experiment_history/21-05-22_17.05.05/outputs_in_mat/chor001_cpitch.mat
modified:   tutorials/experiment_history/21-05-22_17.05.05/outputs_in_mat/chor001_melody_name.mat
modified:   tutorials/experiment_history/21-05-22_17.05.05/outputs_in_mat/chor001_onset.mat
modified:   tutorials/experiment_history/21-05-22_17.05.05/outputs_in_mat/chor002_cpitch.mat
modified:   tutorials/experiment_history/21-05-22_17.05.05/outputs_in_mat/chor002_melody_name.mat
modified:   tutorials/experiment_history/21-05-22_17.05.05/outputs_in_mat/chor002_onset.mat
modified:   tutorials/experiment_history/21-05-22_17.05.05/outputs_in_mat/entropy.mat
modified:   tutorials/experiment_history/21-05-22_17.05.05/outputs_in_mat/information_content.mat
modified:   tutorials/experiment_history/21-05-22_17.05.05/outputs_in_mat/melody_name.mat
modified:   tutorials/experiment_history/21-05-22_17.05.05/plots/entropy_plots/chor-006.png
modified:   tutorials/experiment_history/21-05-22_17.05.05/plots/pianoroll_groundtruth_surprisal/chor-001.png
modified:   tutorials/experiment_history/21-05-22_17.05.05/plots/pianoroll_groundtruth_surprisal/chor-002.png
modified:   tutorials/experiment_history/21-05-22_17.05.05/plots/pianoroll_pitch_prediction_groundtruth/chor-003.png
modified:   tutorials/experiment_history/21-05-22_17.05.05/plots/pianoroll_pitch_prediction_groundtruth/chor-010.png
modified:   tutorials/experiment_history/21-05-22_17.05.05/plots/selected_surprisal_entropy/chor-003.png
modified:   tutorials/experiment_history/21-05-22_17.05.05/plots/simple_plot_information.content/chor-001.png
modified:   tutorials/experiment_history/21-05-22_17.05.05/plots/simple_plot_information.content/chor-002.png
modified:   tutorials/experiment_history/21-05-22_17.05.05/plots/surprisals_plots/chor-005.png

It looks like they are not being reproduced exactly.

Recommendation: Clarify whether these should be reproduced exactly, update copies on remote branch, or add some of these files to a .gitignore

Reproducing

cd tutorials/
head -n 2 experiment_history/21-05-22_17.05.05/outputs_in_csv/chor-005.csv

(Run through the notebooks)

The previous version showed:

dataset.id      ,melody.id,note.id,melody.name   ,vertint12,articulation,comma,voice,ornament,dyn,phrase,bioi,deltast,accidental,mpitch,cpitch,barlength,pulses,tempo   ,mode,keysig,dur ,onset,cpitch.order.ltm.cpitch,cpitch.order.stm.cpitch,cpitch.weight.ltm,cpitch.weight.stm,cpitch.weight.ltm.cpitch,cpitch.weight.stm.cpitch,cpitch.probability,cpitch.information.content,cpitch.entropy,cpitch.55    ,cpitch.57    ,cpitch.58    ,cpitch.59    ,cpitch.60    ,cpitch.62    ,cpitch.63    ,cpitch.64    ,cpitch.65    ,cpitch.66    ,cpitch.67    ,cpitch.68    ,cpitch.69  ,cpitch.70    ,cpitch.71  ,cpitch.72  ,cpitch.73    ,cpitch.74  ,cpitch.75    ,cpitch.76  ,cpitch.77    ,cpitch.78    ,cpitch.79    ,cpitch.81    ,cpitch.82    ,cpitch.83    ,cpitch.84    ,cpitch.85    ,cpitch.86    ,cpitch.88    ,onset.order.ltm.onset,onset.order.stm.onset,onset.weight.ltm,onset.weight.stm,onset.weight.ltm.onset,onset.weight.stm.onset,onset.probability,onset.information.content,onset.entropy,onset.0    ,onset.3     ,onset.6     ,onset.9     ,onset.12   ,onset.18    ,onset.24   ,onset.36    ,onset.48    ,onset.72    ,onset.120   ,probability ,information.content,entropy  ,information.gain
66052122170523.0,5.0      ,1.0    ,"""chor-005""",NA       ,0.0         ,0.0  ,1.0  ,0.0     ,NA ,0.0   ,0.0 ,0.0    ,NA        ,NA    ,72.0  ,96.0     ,4.0   ,600000.0,NA  ,NA    ,24.0,0.0  ,0.0                    ,0.0                    ,0.55405396       ,0.445946         ,1.0                     ,1.0                     ,0.041816834       ,4.5797725                 ,4.8619256     ,0.02490338   ,0.02490338   ,0.025781138  ,0.02490338   ,0.028281277  ,0.029853348  ,0.025781138  ,0.034215137  ,0.028281277  ,0.027468303  ,0.043547735  ,0.023999978  ,0.04467151 ,0.027468303  ,0.044112504,0.041816834,0.026635475  ,0.043547735,0.025781138  ,0.052472092,0.03489898   ,0.036888853  ,0.03879583   ,0.049981304  ,0.029075868  ,0.04738632   ,0.025781138  ,0.029075868  ,0.030614864  ,0.029075868  ,0.0                  ,0.0                  ,0.648152        ,0.351848        ,0.99999994            ,1.0                   ,0.13584055       ,2.880014                 ,3.3046489    ,0.13584055 ,0.03393319  ,0.03393319  ,0.03393319  ,0.10838893 ,0.069162674 ,0.10838893 ,0.089950934 ,0.13060634  ,0.13060634  ,0.12525566  ,0.005680422 ,7.459786           ,8.166577 ,NA

... and after re-running, they show:

dataset.id,melody.id,note.id,melody.name,vertint12,articulation,comma,voice,ornament,dyn,phrase,bioi,deltast,accidental,mpitch,cpitch,barlength,pulses,tempo,mode,keysig,dur,onset,cpitch.order.ltm.cpitch,cpitch.order.stm.cpitch,cpitch.weight.ltm,cpitch.weight.stm,cpitch.weight.ltm.cpitch,cpitch.weight.stm.cpitch,cpitch.probability,cpitch.information.content,cpitch.entropy,cpitch.55,cpitch.57,cpitch.58,cpitch.59,cpitch.60,cpitch.62,cpitch.63,cpitch.64,cpitch.65,cpitch.66,cpitch.67,cpitch.68,cpitch.69,cpitch.70,cpitch.71,cpitch.72,cpitch.73,cpitch.74,cpitch.75,cpitch.76,cpitch.77,cpitch.78,cpitch.79,cpitch.81,cpitch.82,cpitch.83,cpitch.84,cpitch.85,cpitch.86,cpitch.88,onset.order.ltm.onset,onset.order.stm.onset,onset.weight.ltm,onset.weight.stm,onset.weight.ltm.onset,onset.weight.stm.onset,onset.probability,onset.information.content,onset.entropy,onset.0,onset.3,onset.6,onset.9,onset.12,onset.18,onset.24,onset.36,onset.48,onset.72,onset.120,probability,information.content,entropy,information.gain
66052122170523.0,5.0,1.0,"""chor-005""",NA,0.0,0.0,1.0,0.0,NA,0.0,0.0,0.0,NA,NA,72.0,96.0,4.0,600000.0,NA,NA,24.0,0.0,0.0,0.0,0.55405396,0.445946,1.0,1.0,0.041816834,4.5797725,4.8619256,0.02490338,0.02490338,0.025781138,0.02490338,0.028281277,0.029853348,0.025781138,0.034215137,0.028281277,0.027468303,0.043547735,0.023999978,0.04467151,0.027468303,0.044112504,0.041816834,0.026635475,0.043547735,0.025781138,0.052472092,0.03489898,0.036888853,0.03879583,0.049981304,0.029075868,0.04738632,0.025781138,0.029075868,0.030614864,0.029075868,0.0,0.0,0.648152,0.351848,0.99999994,1.0,0.13584055,2.880014,3.3046489,0.13584055,0.03393319,0.03393319,0.03393319,0.10838893,0.069162674,0.10838893,0.089950934,0.13060634,0.13060634,0.12525566,0.005680422,7.459786,8.166577,NA

[xinyiguan]

For the modified files (plots and csv's), the contents are the same. The metadata (time created) has been changed as the previous ones are overwritten. I will add these output files to .gitignore.

For the data in the CSV files, the results (or the content within each file) are the same. It's just the formatting being different. Internally I'm using pandas' to_csv for exporting data to csv file, and there is a minor update after the tutorial has been written.

[hayesall]

Is there a backwards-compatibility reason for keeping old versions checked into git?

Or maybe they're used in a sphinx build?

(My bad if I'm missing something, this might be a non-issue!)

[xinyiguan]

Actually no. The output files generated from the tutorials are there just for the demonstration of the export functionality for the users. I just re-ran the 2b_data_preprocessing_exporting.ipynb to generate new output files so that the formatting is consistent. I think it's useful to keep the experiment_history folder there for reference (i.e., showing the users what is expected to happen). Does it make sense?

[hayesall]

Okay, this makes sense! My bad, I somehow spaced that this came from a notebook in the first place.