Parallel tfrecord reading and parsing.

zhengying-liu / autodl_starting_kit_stable

Starting kit for AutoCV/AutoDL challenge (https://autodl.chalearn.org)

40 stars 23 forks source link

Parallel tfrecord reading and parsing. #7

Closed JulienSiems closed 5 years ago

JulienSiems commented 5 years ago

Allow for parallel reading and parsing of the tfrecord files. The level of parallelism of tfrecord parsing is dynamically tuned by the tf.data pipeline. The number of parallel readers has a reasonable default, but can be changed by an additional argument in the constructor of AutoDLDataset Follows the recommendations given in the tf.data performance guideline.

zhengying-liu commented 5 years ago

Already tested and pushed here: https://github.com/zhengying-liu/autodl/commit/e8baf075551050e276c36839694d6563372fdd77

Will merge to master branch next Monday (July 1st).