search

feature-engine / feature_engine

Feature engineering package with sklearn like functionality
https://feature-engine.trainindata.com/
BSD 3-Clause "New" or "Revised" License
1.8k stars 304 forks source link

implement partial_fit #691

Open solegalli opened 9 months ago

solegalli commented 9 months ago

not sure it is possible, and it is certainly a major undertaking but:

Is it possible to use something like partial_fit for feature engineering?

So that we can allow feature engineering to deal with incremental training for big dataset.

Transformers should be progressively fit on different chunks of data, and somehow retain that information and be updated with the second chunk. Need to look into how partial fit works in sklearn