search

insongkim / PanelMatch

111 stars 34 forks source link

Parallelize PanelMatch #134

Closed hollypuppy closed 4 months ago

hollypuppy commented 4 months ago

Hi,

I am implementing PanelMatch on a large scale of data, which kills my kernel as it requires extensive computation. I was wondering if there is any parallelization version of PanelMatch that I can parallelize the matching process.

I would appreciate your comment on this. Thanks!

adamrauh commented 4 months ago

Hi @hollypuppy ,

Parallelization is something that is on our to-do list. However, some parts will be easier to parallelize than others. For instance, parallelizing the matching process is likely to be challenging. Each unit needs to be compared with many other units, so it tends to be pretty memory-intensive, and that would be exacerbated with simple approaches to parallelization. It would likely require reworking substantial portions of the code to make it work well.

It would be great to solve this, so if you have any ideas, let us know.

-Adam

hollypuppy commented 4 months ago

Yes, I see. Thank you very much for your help! And looking forward to your future work.

Best, Holly