search

bdzyubak / tensorflow-sandbox

A repository for studying applications of Deep Learning across fields, and demonstrating samples of my code and project managment
0 stars 0 forks source link

Epoch timing callback failing to save to excel #29

Closed bdzyubak closed 1 year ago

bdzyubak commented 1 year ago

When using train_fresh=False (default), the epoch timing callback cannot save history to a csv log causing a crash. It needs to append, not insert the epoch time column.

bdzyubak commented 1 year ago

The issue is twofold. The existing file I was using did not have 'epoch_time' as it was from a version of training before the callback. This will now throw a warning saying that a train_fresh=True is required to generate a valid training history for comparison, and epoch time won't be written. The issue of appending to an existing column was patched by integrating with CSVLogger more organically. Now, the custom callback will not open and write to a CSV on its own. It will simply update the callback logs which are used by CSVLogger. This appends to old files and introduces the initial condition of epoch 0 (training start parameters) organically. 3ec13ef067ed15ccb332f09f91b7f2b63dd87d38