Closed johntmyers closed 3 years ago
Misc perf improvements:
Added suggestions from TensorFlow's documentation when creating the TF Dataset and shuffling it
Modified char-by-char tokenizer to read training data line-by-line vs all into memory
Updated default behavior for buffering synthetic records to buffer to disk vs memory
Misc perf improvements:
Added suggestions from TensorFlow's documentation when creating the TF Dataset and shuffling it
Modified char-by-char tokenizer to read training data line-by-line vs all into memory
Updated default behavior for buffering synthetic records to buffer to disk vs memory