search

marcotcr / lime

Lime: Explaining the predictions of any machine learning classifier
BSD 2-Clause "Simplified" License
11.5k stars 1.79k forks source link

Process hangs while using multiprocessing for LIME explanations for Keras models #561

Closed thisissreya closed 3 years ago

thisissreya commented 3 years ago

Hi,

I am trying to parallelize multiple LIME explain_instance calls using Pool() from the python multiprocessing library. It works great for all sklearn models. However, when I use any non-sklearn models (Keras , XGboost), the explain_instance call simply hangs with no result.

Is there any reason why this might be happening? Any help is greatly appreciated. The code looks like this

Worker method

def explain_instance_worker(num): global explainer global train_X global clf_fn exp = explainer.explain_instance(train_X[num], clf_fn, top_labels=1, num_samples=10)

pool = Pool(processes = cpu_count()) pool.map(explain_instance_worker, range(4))

marcotcr commented 3 years ago

explain_instance calls clf_fn, and I'm guessing those models don't let you call that function in parallel.