[FEA] Support dask query planning

[ayushdg]

Is your feature request related to a problem? Please describe. Currently many functionalities/tests do not work when dask query planning is enabled (Default dask behavior).

This is an issue to track the gaps for query planning to work with Curator

• [ ] FuzzyDuplicates
  • [x] Melt (now supported in 24.06)
  • [ ] Custom blockwise/hlg logic in shuffle/merge steps
• [ ] TestShuffle
  • Specifically around dataset.df[self.rand_col] = dataset.df.map_partitions(self._add_rand_col) TypeError: Expected Pandas-like Index, Series, DataFrame, or scalar, got numpy.ndarray

[rjzamora]

@ayushdg - I'd like to start working on this. Do you want to support both query-planning "on" and "off" moving forward? It may be a bit hard to do this without a bunch of compatibly code.

Also, note that melt should now be supported with cudf-2406.