It would be ideal for distributed deployment with a shared filesystem to upload the parquet df with the folds assignment before cv to the shared filesystem and write all feature extraction operations of the cv as lazy operations that can be done starting from a scan parquet on the shared drive.
furlat
commented
4 months ago
It would be ideal for distributed deployment with a shared filesystem to upload the parquet df with the folds assignment before cv to the shared filesystem and write all feature extraction operations of the cv as lazy operations that can be done starting from a scan parquet on the shared drive.