I am trying to parallelize multiple LIME explain_instance calls using Pool() from the python multiprocessing library.
It works great for all sklearn models.
However, when I use any non-sklearn models (Keras , XGboost), the explain_instance call simply hangs with no result.
Is there any reason why this might be happening?
Any help is greatly appreciated.
The code looks like this
Worker method
def explain_instance_worker(num):
global explainer
global train_X
global clf_fn
exp = explainer.explain_instance(train_X[num], clf_fn, top_labels=1,
num_samples=10)
pool = Pool(processes = cpu_count())
pool.map(explain_instance_worker, range(4))
Hi,
I am trying to parallelize multiple LIME explain_instance calls using Pool() from the python multiprocessing library. It works great for all sklearn models. However, when I use any non-sklearn models (Keras , XGboost), the explain_instance call simply hangs with no result.
Is there any reason why this might be happening? Any help is greatly appreciated. The code looks like this
Worker method
def explain_instance_worker(num): global explainer global train_X global clf_fn exp = explainer.explain_instance(train_X[num], clf_fn, top_labels=1, num_samples=10)
pool = Pool(processes = cpu_count()) pool.map(explain_instance_worker, range(4))