search

JuliaML / TableTransforms.jl

Transforms and pipelines with tabular data in Julia
https://juliaml.github.io/TableTransforms.jl/stable
MIT License
102 stars 15 forks source link

Use `Distributed.pmap` in `ParallelTableTransform` and `ColwiseFeatureTransform` #288

Closed eliascarv closed 3 weeks ago

eliascarv commented 3 weeks ago

closes #286

codecov-commenter commented 3 weeks ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 94.93%. Comparing base (56e0add) to head (4893c3c).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #288 +/- ## ========================================== - Coverage 94.95% 94.93% -0.02% ========================================== Files 48 48 Lines 1388 1383 -5 ========================================== - Hits 1318 1313 -5 Misses 70 70 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

juliohm commented 3 weeks ago

We need to benchmark the change, compare it with the previous code. Make sure that pmap is using the CachePool if needed. I know it is the default in latest Julia versions, please double check if Julia v1.9 has it as the default.

juliohm commented 3 weeks ago

Additionally, we use tcollect in ColwiseTransform, and should probably use a pmap too. If that is indeed the case, please update the code, drop Transducers.jl from the list of deps, and update the documentation accordingly.