Closed damro closed 7 years ago
On #8, need to think about how to deal with the case where filenames provided do not line up w corresponding data -- unclear whether there is a check.
-check that all filenames lists are the same length -check that corresponding elements have the same number of records
How is this issue progressing?
I'm thinking of implementing a benchmark test comparing TFRecords / HDF5 reading from the filename queue, varying the number of reading and preprocessing threads and whether data is fed into a placeholder or directly. Seems like it would be generally useful to see how fast each combination is.
We're nearing completion on this. It turned out to have some nontrivial technical challenges. I should have something by later today or tomorrow.
1. dict of dtypes and shapes as input or via convention out of saved metadata in a pkl file (or whatever); specified one has priority