alteryx / featuretools

An open source python library for automated feature engineering
https://www.featuretools.com
BSD 3-Clause "New" or "Revised" License
7.28k stars 878 forks source link

Remove Dask and Spark DataFrame Support #2705

Closed thehomebrewnerd closed 6 months ago

thehomebrewnerd commented 7 months ago

Closes #2704

Removes support for creating EntitySets from Dask and Spark dataframes.

codecov[bot] commented 7 months ago

Codecov Report

Attention: Patch coverage is 99.85141% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 99.38%. Comparing base (21d0bf0) to head (dcee907).

Files Patch % Lines
...s/computational_backends/feature_set_calculator.py 87.50% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2705 +/- ## =========================================== + Coverage 86.97% 99.38% +12.41% =========================================== Files 404 397 -7 Lines 24230 22292 -1938 =========================================== + Hits 21073 22156 +1083 + Misses 3157 136 -3021 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

tamargrey commented 6 months ago

A couple places that probably still need to be updated:

thehomebrewnerd commented 6 months ago
  • The Parallel Computation by Partitioning Data section in the performance.ipynb file has some references to spark - Not sure if we want to just remove all references entirely or leave some subset of it (maybe just reference that an old version of featuretools can be used if you want to use it with Spark), but we should definitely remove the link to the Feature Engineering on Spark Notebook, which will no longer exist.

@tamargrey I think those references are still valid since they relate to manually partitioning data and not doing it via pyspark dataframes in an EntitySet. The Feature Engineering on Spark Notebook will still exist here: https://github.com/alteryx/predict-customer-churn/blob/main/churn/4.%20Feature%20Engineering%20on%20Spark.ipynb. The linked article in that section is gone though, so I'll remove that reference.