search

ModelOriented / kernelshap

Different SHAP algorithms
https://modeloriented.github.io/kernelshap/
GNU General Public License v2.0
38 stars 7 forks source link

Parallel processing #17

Closed dswatson closed 2 years ago

dswatson commented 2 years ago

Thanks for this awesome package!

I was wondering if it might be possible to have an option for parallel processing? I'm thinking specifically of the big for loop:

for (i in seq_len(n)) res[[i]] <- kernelshap_one(...)

which could easily be replaced with foreach:

res <- foreach(i = seq_len(n)) %dopar% kernelshap_one(...)

The downside is, you'd lose the progress bar. (There are some methods for doing parallel progress bars in R, but they all kind of suck in my experience.) The upside is, you can speed up the slowest part of the task by a factor of n_cores. Perhaps a logical parallel argument could allow users to toggle between the two?

mayer79 commented 2 years ago

Excellent idea - this is the right place to parallelize Kernel SHAP and I am definitively open to add such functionality in the future.

However, I have not yet been successful to make things run with foreach. I checked how "missForest" and other packages are using it, but I do get different types of error messages. An example: "object 'fit' not found", where fit is part of the predict function. Furthermore, we should keep an eye on the random seed.

Support would be highly appreciated. We don't need to hurry as the project is still young.

dswatson commented 2 years ago

I've got a local version that seems to run without issue. I'll make a pull request.

mayer79 commented 2 years ago

Implemented in https://github.com/mayer79/kernelshap/pull/20 :-)