PierrickPochelu
commented
4 months ago According to the error message I see that you want to explain 8 million variables. The matrix raising the memory error consumes O(N^2) with N the number of input variables. It is not doable. You can use the attribute max_shap_value_inputs to sample and approximate Shapley, but max_shap_value_inputs should be superior to N and 8Millions (features) will still take a long time. In conclusion, I fear you need to reformulate the problem you want to solve before using this tool by reducing the number of input variables.
You can try the code below with different combinations of nb_features, nb_events, and max_shap_value_inputs and see when it fits your problem and your computer.
import numpy as np
np.random.seed(42)
import pandas as pd
def gen_xy(nb_features=10, nb_events=10, x_type=np.float32, date_type=np.float32):
X_train=np.random.rand(nb_events, nb_features).astype(x_type)
np_time=np.random.rand(nb_events)
noise=np.clip(0,1, np.random.rand(nb_events)*0.01)
np_is_living=X_train[:,0] < np_time+noise # <--- dumb y
y_train=np.empty(nb_events, dtype=[('event', bool), ('time', date_type)])
y_train['event']=np_is_living
y_train['time']=np_time
X_train=pd.DataFrame(X_train,columns=['f'+str(i) for i in range(1,nb_features+1)])
return X_train, y_train
X_train, y_train = gen_xy(nb_events=1800)
X_test, y_test = gen_xy(nb_events=1800)
from sksurv.ensemble import RandomSurvivalForest
rsf = RandomSurvivalForest(
n_estimators=10, max_depth=25, min_samples_split=10, min_samples_leaf=15, n_jobs=-1, random_state=42
)
rsf.fit(X_train, y_train)
from survshap import SurvivalModelExplainer, PredictSurvSHAP, ModelSurvSHAP
rsf_exp = SurvivalModelExplainer(rsf, X_test, y_test)
exp1_survshap_global_rsf = ModelSurvSHAP(random_state=42, max_shap_value_inputs=20)
exp1_survshap_global_rsf.fit(rsf_exp)
kaalen
Issue Description
I used a Random Survival Forest with 10 estimators and a max depth of 25 on approximately 1800 data samples. The full dataset otherwise contains approximately 200,000 data samples, but I intentionally only used a very small sample when I encountered this error. When attempting to fit a ModelSurvSHAP on this very small dummy random survival forest I encounter the following error:
MemoryError: Unable to allocate 512. TiB for an array with shape (8388608, 8388608) and data type float64I'm using survshap version 0.4.2.
Minimal Reproducible Code Sample
Error Trace: