neuroailab / tfutils

Utilities for working with tensorflow
MIT License
25 stars 8 forks source link

Grid file system saving of train results #49

Closed nhaber closed 7 years ago

nhaber commented 7 years ago

Right now, if one specifies something to be saved to gfs, only the results passed into save get included. Hence, training targets meant to be saved to gfs get lost except at save_metrics_freq. Case in mind: we want to store data ids seen at training time. Adding this as a training target, these get appended until save_metrics_freq, and then only the most recent result is popped into save_to_gfs.

TODO: At saving time, anything that should be saved to gfs is popped from the list of training results stored up until then.