ManifoldRG / MultiNet

Apache License 2.0
18 stars 1 forks source link

Wrapper script to manage RAM util when translating control dataset shards #105

Closed pranavguru closed 3 months ago

pranavguru commented 3 months ago

Translation of control dataset shards to TFDS is a very RAM intensive process, especially due to tf.data.Dataset.save(). This wrapper script checks RAM util when running the translation script and restarts the program everytime the RAM util goes above 80%.