Open madsbk opened 1 month ago
For context, I have seen this error the most while writing partitioned datasets. Don't know how it impacts there.
dask_df.to_parquetpartition_on=["xyz"])
And if https://github.com/rapidsai/kvikio/pull/410 helps in that case too.
Should help with issues mentioned in the comments here:
https://github.com/NVIDIA/NeMo-Curator/pull/157#discussion_r1687215245
In order to avoid
ulimit
issues, it would be useful to have an option that limits the number of open files. Maybe open files lazily?cc. @VibhuJawa