Open AnnKatrinBecker opened 4 years ago
Any update on this? I'm getting a similar problem import a 38GB dataset using ImportExampleGen.
For large scale datasets it is recommended to use DataflowRunner (on GCP) or FlinkRunner or SparkRunner if it is a on-premises execution.
Ubuntu 18.04 tensorflow 2.2.0 tfx 0.21.4
I am generating a vocabulary from my own dataset containing 28GB TFRecords with short description strings (up to 20 words) and integer labels from 1-100.
Generating the vocabulary without labels works fine. But as soon as the labels argument is provided to tft.vocabulary memory usage increases dramatically >100GB until the processes gets killed due to running out of memory.
`def preprocessing_fn(inputs): label = inputs['label'] desc = inputs['description'] desc = tf.strings.lower(desc.values)
remove all numbers and punctuations
`def main():
Brings data into the pipeline
pipe = pipeline.Pipeline( pipeline_name='test', pipeline_root='pipelines/test', components=[ example_gen, schema_importer, transform, ], metadata_connection_config=metadata.sqlite_metadata_connection_config( 'test/metadata.db'), enable_cache=True, beam_pipeline_args=['--direct_num_workers=0'])