chenyangkang / stemflow

A Python Package for Adaptive Spatio-Temporal Exploratory Model (AdaSTEM)
https://chenyangkang.github.io/stemflow/
MIT License
17 stars 1 forks source link

[Feature] Make parallel computing available #41

Closed chenyangkang closed 9 months ago

chenyangkang commented 9 months ago

Is your feature request related to a problem? Please describe. Add parallel modeling for:

  1. split
  2. fit
  3. predict
  4. assign importance to points

Describe the solution you'd like Add joblib as parallel backend.

Describe alternatives you've considered multiprocessing with shared memory

Additional context Add any other context or screenshots about the feature request here.

chenyangkang commented 9 months ago

I tried shared_memory module in multiprocessing. Doesn't work well for overhead problem and show bugs when running in script. Falling back to joblib. The memory will be come an issue for big data parallel.