Tfrecord is Tensorflow's own binary storage format for data files (including sequence data).
With this format, a better performance of up to 50% can be reached since the files require less space on the disk, less time to copy and can be read more efficiently. This is especially useful for memory intensive data and batch processing.
The data is stored as a sequence of binary strings. To convert any data type, follow these major steps (already implemented in src/...):
convert all entries into either tf.train.BytesList, tf.train.FloatList, or tf.train.Int64List (and strings into bytes), then into tf.train.Feature (for each feature), create dictionary out of them
store each sample in tf.train.Example or tf.train.SequenceExample
serialize it
use a tf.python_io.TFRecordWriter to write it to disk
Find more information in the Tensorflow documentation.
Tfrecord is Tensorflow's own binary storage format for data files (including sequence data). With this format, a better performance of up to 50% can be reached since the files require less space on the disk, less time to copy and can be read more efficiently. This is especially useful for memory intensive data and batch processing.
The data is stored as a sequence of binary strings. To convert any data type, follow these major steps (already implemented in src/...):
Find more information in the Tensorflow documentation.