The sequential table transformer (#802) is great if later transformations depend on prior ones. Often, however, columns are transformed independently, which gives us opportunity to parallelize.
Desired solution
New class ParallelTableTransformer.
Superclass: InvertibleTableTransformer
Constructor parameters:
transformers: list[TableTransformer].
Raise if multiple transformers are applied to the same column.
fit:
Fit the individual transformers on the given table in parallel. Keep in mind that multithreading is usually not helpful for CPU-bound tasks in Python due to the Global Interpreter lock. Multiprocessing is likely more efficient. Profile your code!
transform:
Transform the table with the individual transformers in parallel (see fit). Gather the results of the transformers and merge them to a single table.
inverse_transform:
Raise an error if any transformer is not invertible
Inverse-transform the table with the individual transformers in parallel (see fit). Gather the results of the transformers and merge them to a single table.
Is your feature request related to a problem?
The sequential table transformer (#802) is great if later transformations depend on prior ones. Often, however, columns are transformed independently, which gives us opportunity to parallelize.
Desired solution
New class
ParallelTableTransformer
.InvertibleTableTransformer
transformers: list[TableTransformer]
.fit
:transform
:fit
). Gather the results of the transformers and merge them to a single table.inverse_transform
:fit
). Gather the results of the transformers and merge them to a single table.Possible alternatives (optional)
No response
Screenshots (optional)
No response
Additional Context (optional)
No response