tensorflow / model-remediation

Model Remediation is a library that provides solutions for machine learning practitioners working to create and train models in a way that reduces or eliminates user harm resulting from underlying performance biases.
https://www.tensorflow.org/responsible_ai/model_remediation?hl=en
Apache License 2.0
42 stars 19 forks source link

TF Dataset to TF Examples List in FDW Utils: Implementation is slow and consumes excessive RAM #31

Open CLSchmitz opened 1 year ago

CLSchmitz commented 1 year ago

The current implementation of TF Dataset to TF Examples List uses a lot of RAM and runs very slowly on large datasets.

In this colab, the CelebA dataset, as processed by the fairness indicators tutorial, can't be processed on a default hosted runtime. After 18-20 minutes, the runtime crashes because RAM requirements are exceeded.

Because of all the preprocessing required, it's not easy to create a smaller reproducible example.