Record epoch times

[CarolinaFurtado]

It works:

[rak5216]

simple enough, nice job! one thing we should consider yet is whether we can take the output spreadsheet here and easily append it to the standard metrics output. would be good to have everything in one sheet, and easiest way would be to combine the spreadsheets probably. given that both are csv, i imagine something like this maybe:

https://stackoverflow.com/questions/55639115/combining-csvs-with-different-columns-using-pandas-with-key-column

[rak5216]

also, we def now want to plot epoch time too and add to mosaic

[CarolinaFurtado]

@rak5216 adding to the mosaic is not straight forward, because it is in a different file and also, does not have train and validation separately. Would having individual plots be enough?

[rak5216]

you could just make it the first plot in the mosaic with an if statement in the loop, no? that would be ideal

[CarolinaFurtado]

@rak5216 I was able to add it to metrics! super easy, in the end...

now working on the plots should be easy

[CarolinaFurtado]

individual plots (i'll add seconds in the y axis)

Mosaic plot

Metrics.csv

@rak5216

[CarolinaFurtado]

@rak5216 the new callback edits the logs by adding a new entry to the dictionary everytime a epoch ends. The callbacks in

    results = compiled_model.fit(
        train_generator,
        steps_per_epoch=len(train_generator),
        epochs=epochs,
        validation_data=validation_generator,
        validation_steps=len(validation_generator),
        callbacks=[model_checkpoint_callback, tensorboard_callback, time_callback, csv_logger_callback]
    )

are called sequentially. That is why csv_looger_callback is called last. This way it already accounts for the new entry created by time_callback. Pretty simple in the end... But I only found one link that explained this well.