Make sure query-planning is disabled for now

NVIDIA / NeMo-Curator

Scalable toolkit for data curation

Apache License 2.0

327 stars 32 forks source link

Make sure query-planning is disabled for now #97

Closed rjzamora closed 3 weeks ago

rjzamora commented 3 weeks ago

As far as I can tell, the existing logic used to disable query-planning isn't really doing anything. As soon as dask.dataframe is imported, changing the "dataframe.query-planning" configuration will no longer have an effect.

rjzamora commented 3 weeks ago

Is this good to merge, or do you want to update?

Current version of the PR should be fine, but happy to make revisions if requested :)