Platinum analysis

[jyaacoub]

1. Build dataset
   1. [x] Copy all aflow platinum conf files from #narval to #h4h ✅ 2024-04-29
   2. [x] Init new platinum dataset with new confs ✅ 2024-04-29

      create_datasets(cfg.DATA_OPT.platinum, 
              cfg.PRO_FEAT_OPT.nomsa, cfg.PRO_EDGE_OPT.aflow, 
              ligand_features=cfg.LIG_FEAT_OPT.gvp,
              ligand_edges=cfg.LIG_EDGE_OPT.binary,
              k_folds=None, train_split=0, val_split=0)       

2. Run inference
   1. [x] Find trained weights for GVPL-aflow model on pdbbind (all 5 for each split to use as an ensemble) ✅ 2024-04-29
      • Model key is GVPLM_PDBbind1D_nomsaF_aflowE_128B_0.00022659LR_0.02414D_2000E_gvpLF_binaryLE
      • They are all on #h4h
   2. [x] Save predictions of each model to a csv file ✅ 2024-04-29

      
      loaders = Loader.load_DataLoaders(cfg.DATA_OPT.platinum,
                             cfg.PRO_FEAT_OPT.nomsa, cfg.PRO_EDGE_OPT.aflow,
                             ligand_feature=cfg.LIG_FEAT_OPT.gvp, ligand_edge=cfg.LIG_EDGE_OPT.binary,
                             datasets=['test'])

%%

model = Loader.init_model(cfg.MODEL_OPT.GVPL, cfg.PRO_FEAT_OPT.nomsa, cfg.PRO_EDGE_OPT.aflow, dropout=0.02414, output_dim=256)

%%

cp_dir = "/cluster/home/t122995uhn/projects/MutDTA/results/model_checkpoints/ours" MODEL_KEY = lambda fold: f"GVPLM_PDBbind{fold}D_nomsaF_aflowE_128B_0.00022659LR_0.02414D_2000E_gvpLF_binaryLE" cp = lambda fold: f"{cp_dir}/{MODEL_KEY(fold)}.model"

out_dir = f'{cfg.MEDIA_SAVE_DIR}/test_set_pred/' os.makedirs(out_dir, exist_ok=True)

for i in range(5): model.safe_load_state_dict(torch.load(cp(i), map_location=device)) model.to(device) model.eval()

loss, pred, actual = test(model, loaders['test'], device, verbose=True)

# saving as csv with columns code, pred, actual
# get codes from test loader
codes, pid = [b['code'][0] for b in loaders['test']], [b['prot_id'][0] for b in loaders['test']]
df = pd.DataFrame({'prot_id': pid, 'pred': pred, 'actual': actual}, index=codes)
df.index.name = 'code'
df.to_csv(f'{out_dir}/{MODEL_KEY(i)}_PLATINUM.csv')

4. Analysis  ^3fe12b
    1. [x] Check for overlap in pdbIDs and remove them 
        - provide a plot of with and without performance?
    2. how can I analyze the performance of the model in its ability to detect dangerous mutations?
        1. [x] Normal predictive performance analysis with cindex and MSE scores
            - Statistical t-test to determine how different the predicted and experimental distributions are from each other
        2. [x] Mutation impact analysis: 
            - calculate $\Delta pk_a$ (change in binding affinity for each protein-ligand pair). Then correlate the predicted $\Delta pk_a$ and the experimental $\Delta pk_a$ 
                - higher correlation indicates that the **model effectively captures the impact of mutations**
            - Split up scores depending on number of mutations for even more analysis (maybe it struggles with larger # of mutations)
        3. [x] Identifying "significant" mutations
            - Based on the distribution of $\Delta pk_a$ scores classify all above **[1 and 2] standard deviations to be "significant"**?
            - Then Build a confusion matrix to analyze the true positive rate and true negative rate of the model in identifying these mutations.

[jyaacoub]

There is huge overlap between the PDBbind training data and the platinum dataset unfortunately...

Removing all exact instances of pdbids leaves us with 975 rows (967 if we drop nan):

However, if we consider both mutated and wildtype proteins as the same protein then we are left with 480 rows.

code

```python import pandas as pd df = pd.read_csv("results/model_media/test_set_pred/GVPLM_PDBbind0D_nomsaF_aflowE_128B_0.00022659LR_0.02414D_2000E_gvpLF_binaryLE_PLATINUM.csv", index_col=0) # training set codes: data_p = lambda set: f'/cluster/home/t122995uhn/projects/data/PDBbindDataset/nomsa_aflow_gvp_binary/{set}0/cleaned_XY.csv' df_t = pd.Index.append(pd.read_csv(data_p('train'), index_col=0).index, pd.read_csv(data_p('val'), index_col=0).index) df_t = df_t.str.upper() df['pdb'] = df['prot_id'].str.split('_').str[0] #%% remove training codes from df # dont remove wt prots: # df = df[~(df['pdb'].isin(df_t) & df.index.str.contains('_mt'))] print(df) # remove all df = df[~(df['pdb'].isin(df_t))] print(df) # %% treat mutated and wt proteins as the same wt_df = df[df.index.str.contains("_wt")] mt_df = df[df.index.str.contains("_mt")] missing_wt = delta_pkds = 0 for m in mt_df.index: i_wt = m.split('_')[0] + '_wt' if i_wt not in wt_df.index: missing_wt += 1 else: delta_pkds += 1 print("missing wt:", missing_wt) print("delta_pkds:", delta_pkds) ```

[jyaacoub]

1. predictive performance

raw:

	with overlap	without overlap	p-val
cindex	0.674 $\pm$ 0.010	0.641 $\pm$ 0.011	0.0624
pcorr	0.530 $\pm$ 0.023	0.415 $\pm$ 0.028	0.0139
scorr	0.503 $\pm$ 0.030	0.411 $\pm$ 0.039	0.0973
mse	3.106 $\pm$ 0.121	3.467 $\pm$ 0.112	0.0598
mae	1.380 $\pm$ 0.026	1.467 $\pm$ 0.014	0.0198
rmse	1.761 $\pm$ 0.035	1.861 $\pm$ 0.030	0.062

distribution


z-normalized:

	with overlap	without overlap	p-val
cindex	0.674 $\pm$ 0.010	0.641 $\pm$ 0.011	0.0624
pcorr	0.530 $\pm$ 0.023	0.415 $\pm$ 0.028	0.0139
scorr	0.503 $\pm$ 0.030	0.411 $\pm$ 0.039	0.0973
mse	0.939 $\pm$ 0.046	0.947 $\pm$ 0.034	0.9008
mae	0.758 $\pm$ 0.020	0.754 $\pm$ 0.009	0.8436
rmse	0.968 $\pm$ 0.024	0.972 $\pm$ 0.018	0.8879

distribution


code

```python def predictive_performance( MODEL = lambda i: f"results/model_media/test_set_pred/GVPLM_PDBbind{i}D_nomsaF_aflowE_128B_0.00022659LR_0.02414D_2000E_gvpLF_binaryLE_PLATINUM.csv", TRAIN_DATA_P = lambda set: f'{cfg.DATA_ROOT}/PDBbindDataset/nomsa_aflow_gvp_binary/{set}0/cleaned_XY.csv', NORMALIZE = True, n_models=5, compare_overlap=False, verbose=True, plot=False, ): df_t = pd.Index.append(pd.read_csv(TRAIN_DATA_P('train'), index_col=0).index, pd.read_csv(TRAIN_DATA_P('val'), index_col=0).index) df_t = df_t.str.upper() results_with_overlap = [] results_without_overlap = [] for i in range(n_models): df = pd.read_csv(MODEL(i), index_col=0).dropna() df['pdb'] = df['prot_id'].str.split('_').str[0] if NORMALIZE: mean_df = df[['actual','pred']].mean(axis=0, numeric_only=True) std_df = df[['actual','pred']].std(axis=0, numeric_only=True) df[['actual','pred']] = (df[['actual','pred']] - mean_df) / std_df # z-normalization if i==0: print(df) # with overlap cindex, p_corr, s_corr, mse, mae, rmse = get_metrics(df['actual'], df['pred']) results_with_overlap.append([cindex, p_corr[0], s_corr[0], mse, mae, rmse]) # without overlap df_no_overlap = df[~(df['pdb'].isin(df_t))] cindex, p_corr, s_corr, mse, mae, rmse = get_metrics(df_no_overlap['actual'], df_no_overlap['pred']) results_without_overlap.append([cindex, p_corr[0], s_corr[0], mse, mae, rmse]) if i==0 and plot: n_plots = int(compare_overlap)+1 fig = plt.figure(figsize=(14,5*n_plots)) axes = fig.subplots(n_plots,1) ax = axes[0] if compare_overlap else axes sns.histplot(df_no_overlap['actual'], kde=True, ax=ax, alpha=0.5, label='True pkd') sns.histplot(df_no_overlap['pred'], kde=True, ax=ax, alpha=0.5, label='Predicted pkd', color='orange') ax.set_title(f"{'Normalized 'if NORMALIZE else ''} pkd distribution") ax.legend() if compare_overlap: sns.histplot(df_no_overlap['actual'], kde=True, ax=axes[1], alpha=0.5, label='True pkd') sns.histplot(df_no_overlap['pred'], kde=True, ax=axes[1], alpha=0.5, label='Predicted pkd', color='orange') axes[1].set_title(f"{'Normalized 'if NORMALIZE else ''} pkd distribution (no overlap)") axes[1].legend() if compare_overlap: return generate_markdown([results_with_overlap, results_without_overlap], names=['with overlap', 'without overlap'], cindex=True,verbose=verbose) return generate_markdown([results_without_overlap], names=['mean $\pm$ se'], cindex=True, verbose=verbose) ```

[jyaacoub]

2. Mutation impact analysis

Same thing but looking at $\Delta pkd$ this time

2.1 delta pkd predictive performance

raw $\Delta pkd$

Model-0 Distribution


metric	With Overlap	Without Overlap	Significance
pcorr	0.176 $\pm$ 0.026	0.037 $\pm$ 0.079	*
scorr	0.099 $\pm$ 0.019	0.046 $\pm$ 0.060
mse	1.505 $\pm$ 0.009	1.303 $\pm$ 0.006	*
mae	0.905 $\pm$ 0.003	0.847 $\pm$ 0.002	*
rmse	1.227 $\pm$ 0.004	1.141 $\pm$ 0.003	*

Z-normalized

Model-0 Distribution


	With Overlap	Without Overlap	Significance
pcorr	0.176 $\pm$ 0.026	0.037 $\pm$ 0.079	*
scorr	0.099 $\pm$ 0.019	0.046 $\pm$ 0.060
mse	1.649 $\pm$ 0.053	1.927 $\pm$ 0.158	*
mae	0.899 $\pm$ 0.029	1.014 $\pm$ 0.023	*
rmse	1.284 $\pm$ 0.021	1.387 $\pm$ 0.057	*

Code

```python from src.analysis.figures import tbl_dpkd_metrics_overlap, tbl_dpkd_metrics_n_mut MODEL = lambda i: f"results/model_media/test_set_pred/GVPLM_PDBbind{i}D_nomsaF_aflowE_128B_0.00022659LR_0.02414D_2000E_gvpLF_binaryLE_PLATINUM.csv" TRAIN_DATA_P = lambda set: f'/cluster/home/t122995uhn/projects/data/PDBbindDataset/nomsa_aflow_gvp_binary/{set}0/cleaned_XY.csv' NORMALIZE = True # %% print('OVERLAP') mkdo = tbl_dpkd_metrics_overlap(MODEL, TRAIN_DATA_P, NORMALIZE, plot=False) print('NUM MUTATIONS:') mkdnm = tbl_dpkd_metrics_n_mut(MODEL, NORMALIZE, plot=False) ```

2.2. Stratify by mutation count

2 classes "single mutation" vs "2+ mutations"

histogram


	1 mutations	2+ mutations	Sig
pcorr	0.076 $\pm$ 0.019	0.336 $\pm$ 0.047	*
scorr	0.053 $\pm$ 0.015	0.252 $\pm$ 0.043	*
mse	1.848 $\pm$ 0.038	1.328 $\pm$ 0.095	*
mae	0.961 $\pm$ 0.031	0.833 $\pm$ 0.028	*
rmse	1.359 $\pm$ 0.014	1.152 $\pm$ 0.041	*

3 classes "single", "2", "3+"

histogram


	1 mutations	2 mutations	3+ mutations
pcorr	0.076 $\pm$ 0.019	0.207 $\pm$ 0.055	0.509 $\pm$ 0.070
scorr	0.053 $\pm$ 0.015	0.131 $\pm$ 0.038	0.496 $\pm$ 0.078
mse	1.848 $\pm$ 0.038	1.586 $\pm$ 0.110	0.982 $\pm$ 0.140
mae	0.961 $\pm$ 0.031	0.964 $\pm$ 0.016	0.732 $\pm$ 0.022
rmse	1.359 $\pm$ 0.014	1.259 $\pm$ 0.043	0.989 $\pm$ 0.070

CODE

```python from src.analysis.figures import tbl_dpkd_metrics_overlap, tbl_dpkd_metrics_n_mut MODEL = lambda i: f"results/model_media/test_set_pred/GVPLM_PDBbind{i}D_nomsaF_aflowE_128B_0.00022659LR_0.02414D_2000E_gvpLF_binaryLE_PLATINUM.csv" TRAIN_DATA_P = lambda set: f'/cluster/home/t122995uhn/projects/data/PDBbindDataset/nomsa_aflow_gvp_binary/{set}0/cleaned_XY.csv' NORMALIZE = True print('NUM MUTATIONS:') mkdnm = tbl_dpkd_metrics_n_mut(MODEL, NORMALIZE, conditions=[1,2], plot=True) ```

2.3. Stratify by location of mutation

2.3.1. binding pocket vs not in binding pocket

histogram


	mutation in pocket	mutation NOT in pocket	Sig
pcorr	0.180 $\pm$ 0.029	0.110 $\pm$ 0.056	*
scorr	0.108 $\pm$ 0.022	0.050 $\pm$ 0.102
mse	1.640 $\pm$ 0.057	1.781 $\pm$ 0.113	*
mae	0.904 $\pm$ 0.021	0.981 $\pm$ 0.018	*
rmse	1.280 $\pm$ 0.022	1.334 $\pm$ 0.042	*

code

```python # %% import matplotlib.pyplot as plt import seaborn as sns from src.analysis.figures import get_dpkd NORMALIZE = True dfr = pd.read_csv(f"{cfg.DATA_ROOT}/PlatinumDataset/raw/platinum_flat_file.csv", index_col=0) df = pd.read_csv("/cluster/home/t122995uhn/projects/data/PlatinumDataset/nomsa_binary_original_binary/full/cleaned_XY.csv", index_col=0).dropna() df = get_in_binding(df, dfr=dfr) fig = plt.figure(figsize=(14,5)) ax = fig.subplots(1,1) # must include 0 in both cases since they are the wildtype reference true_dpkd1 = get_dpkd(df.query('(pocket == 0) | (pocket == 2)'), 'pkd', NORMALIZE) sns.histplot(true_dpkd1, kde=True, ax=ax, alpha=0.6, color='orange', label='not in pocket', stat='proportion') true_dpkd1 = get_dpkd(df.query('(pocket == 0) | (pocket == 1)'), 'pkd', NORMALIZE) sns.histplot(true_dpkd1, kde=True, ax=ax, alpha=0.6, color=None, label='in pocket', stat='proportion') ax.set_title(f"{'Normalized 'if NORMALIZE else ''}TRUE Δpkd distribution") ax.set_xlabel('Δpkd') ax.legend() ```

2.3.2. distance to ligand

• We will treat this as a classification problem (near is <4A)

	counts
wt	981
near lig	577
not near lig	372

	mutation near lig (<4A)	mutation not near lig (>4A)	p-val
pcorr	0.164 $\pm$ 0.012	0.198 $\pm$ 0.017	0.1405
scorr	0.104 $\pm$ 0.009	0.079 $\pm$ 0.015	0.1844
mse	1.672 $\pm$ 0.023	1.604 $\pm$ 0.034	0.1405
mae	0.912 $\pm$ 0.010	0.939 $\pm$ 0.005	0.0408
rmse	1.293 $\pm$ 0.009	1.266 $\pm$ 0.013	0.1381

Distribution


Code

```python from src.analysis.figures import tbl_dpkd_metrics_overlap, tbl_dpkd_metrics_n_mut, tbl_dpkd_metrics_in_binding, predictive_performance, tbl_stratified_dpkd_metrics from src.analysis.metrics import get_metrics from src import config as cfg import pandas as pd #%% MODEL = lambda i: f"results/model_media/test_set_pred/GVPLM_PDBbind{i}D_nomsaF_aflowE_128B_0.00022659LR_0.02414D_2000E_gvpLF_binaryLE_PLATINUM.csv" RAW_PLT_CSV=f"{cfg.DATA_ROOT}/PlatinumDataset/raw/platinum_flat_file.csv" NORMALIZE = True n_models=5 verbose=True plot=True dfr = pd.read_csv(RAW_PLT_CSV, index_col=0) dfp = pd.read_csv(MODEL(0), index_col=0) #%% import seaborn as sns sns.histplot(dfr['mut.distance_to_lig']) #%% # add in_binding info to df thres = 4 def get_in_binding(df, dfr): """ df is the predicted csv with index as _wt (or *_mt) where raw_idx corresponds to an index in dfr which contains the raw data for platinum including ('mut.in_binding_site') - 0: wildtype rows - 1: close (<8 Ang) - 2: Far (>8 Ang) """ near_lig = dfr[dfr['mut.distance_to_lig'] < thres].index pclass = [] for code in df.index: if '_wt' in code: pclass.append(0) elif int(code.split('_')[0]) in near_lig: pclass.append(1) else: pclass.append(2) df['near_lig'] = pclass return df conditions = ['(near_lig == 0) | (near_lig == 1)', '(near_lig == 0) | (near_lig == 2)'] names = [f'mutation near lig (<{thres}A)', f'mutation not near lig (>{thres}A)'] df = get_in_binding(dfp, dfr) if verbose: cnts = df.near_lig.value_counts() cnts.index = ['wt', 'near lig', 'not near lig'] cnts.name = "counts" print(cnts.to_markdown(), end="\n\n") #%% tbl_stratified_dpkd_metrics(MODEL, NORMALIZE, n_models=n_models, df_transform=get_in_binding, conditions=conditions, names=names, verbose=verbose, plot=plot, dfr=dfr) ```

[jyaacoub]

3. Significant Mutation impact analysis

With overlap best threshold is 0.1*STD:

Figures (ROC curve and sample confusion matrix)

 

Without overlap best threshold is 0.3*STD

Figures (ROC curve and sample confusion matrix)

 

code

```python #%% from src.analysis.figures import get_dpkd, fig_sig_mutations_conf_matrix, generate_roc_curve from src.analysis.metrics import get_metrics import numpy as np import pandas as pd from scipy.stats import ttest_ind MODEL = lambda i: f"results/model_media/test_set_pred/GVPLM_PDBbind{i}D_nomsaF_aflowE_128B_0.00022659LR_0.02414D_2000E_gvpLF_binaryLE_PLATINUM.csv" data_p = lambda set: f'/cluster/home/t122995uhn/projects/data/PDBbindDataset/nomsa_aflow_gvp_binary/{set}0/cleaned_XY.csv' df_t = pd.Index.append(pd.read_csv(data_p('train'), index_col=0).index, pd.read_csv(data_p('val'), index_col=0).index) df_t = df_t.str.upper() results_with_overlap = [] results_without_overlap = [] i=0 df = pd.read_csv(MODEL(i), index_col=0).dropna() df['pdb'] = df['prot_id'].str.split('_').str[0] df_no = df[~(df['pdb'].isin(df_t))] #%% true_dpkd = get_dpkd(df, pkd_col='actual') pred_dpkd = get_dpkd(df, pkd_col='pred') true_dpkd_no = get_dpkd(df_no, pkd_col='actual') pred_dpkd_no = get_dpkd(df_no, pkd_col='pred') # %% # ROC _, _, _, best_threshold = generate_roc_curve(true_dpkd, pred_dpkd, thres_range=(0,5), step=0.1) _ = fig_sig_mutations_conf_matrix(true_dpkd, pred_dpkd, std=round(best_threshold, 3)) # %% _, _, _, best_threshold = generate_roc_curve(true_dpkd_no, pred_dpkd_no, thres_range=(0,5), step=0.1) _ = fig_sig_mutations_conf_matrix(true_dpkd_no, pred_dpkd_no, std=round(best_threshold, 3)) ```

[jyaacoub]

Pretrained Davis results

outline

• [x] build dataset from aflow confirmations
• [x] Run hyperparameter tuning
• [x] Train davis
• [x] Run platinum evaluation

results:

Code

```python # %% import torch, os import pandas as pd from src import cfg from src import TUNED_MODEL_CONFIGS from src.utils.loader import Loader from src.train_test.training import test from src.analysis.figures import predictive_performance, tbl_stratified_dpkd_metrics, tbl_dpkd_metrics_overlap, tbl_dpkd_metrics_in_binding device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") INFERENCE = False VERBOSE = True out_dir = f'{cfg.MEDIA_SAVE_DIR}/test_set_pred/' os.makedirs(out_dir, exist_ok=True) cp_dir = cfg.CHECKPOINT_SAVE_DIR RAW_PLT_CSV=f"{cfg.DATA_ROOT}/PlatinumDataset/raw/platinum_flat_file.csv" #%% load up model: for KEY, CONFIG in TUNED_MODEL_CONFIGS.items(): MODEL_KEY = lambda fold: Loader.get_model_key(CONFIG['model'], CONFIG['dataset'], CONFIG['feature_opt'], CONFIG['edge_opt'], CONFIG['batch_size'], CONFIG['lr'], CONFIG['architecture_kwargs']['dropout'], n_epochs=2000, fold=fold, ligand_feature=CONFIG['lig_feat_opt'], ligand_edge=CONFIG['lig_edge_opt']) print('\n\n'+ '## ' + KEY) OUT_PLT = lambda i: f'{out_dir}/{MODEL_KEY(i)}_PLATINUM.csv' db_p = f"{CONFIG['feature_opt']}_{CONFIG['edge_opt']}_{CONFIG['lig_feat_opt']}_{CONFIG['lig_edge_opt']}" if CONFIG['dataset'] in ['kiba', 'davis']: db_p = f"DavisKibaDataset/{CONFIG['dataset']}/{db_p}" else: db_p = f"{CONFIG['dataset']}Dataset/{db_p}" train_p = lambda set: f"{cfg.DATA_ROOT}/{db_p}/{set}0/cleaned_XY.csv" if not os.path.exists(OUT_PLT(0)) or INFERENCE: print('running inference!') cp = lambda fold: f"{cp_dir}/{MODEL_KEY(fold)}.model" model = Loader.init_model(model=CONFIG["model"], pro_feature=CONFIG["feature_opt"], pro_edge=CONFIG["edge_opt"],**CONFIG['architecture_kwargs']) # load up platinum test db loaders = Loader.load_DataLoaders(cfg.DATA_OPT.platinum, pro_feature = CONFIG['feature_opt'], edge_opt = CONFIG['edge_opt'], ligand_feature = CONFIG['lig_feat_opt'], ligand_edge = CONFIG['lig_edge_opt'], datasets=['test']) for i in range(5): model.safe_load_state_dict(torch.load(cp(i), map_location=device)) model.to(device) model.eval() loss, pred, actual = test(model, loaders['test'], device, verbose=True) # saving as csv with columns code, pred, actual # get codes from test loader codes, pid = [b['code'][0] for b in loaders['test']], [b['prot_id'][0] for b in loaders['test']] df = pd.DataFrame({'prot_id': pid, 'pred': pred, 'actual': actual}, index=codes) df.index.name = 'code' df.to_csv(OUT_PLT(i)) # run platinum eval: print('\n### 1. predictive performance') mkdown = predictive_performance(OUT_PLT, train_p, verbose=VERBOSE, plot=False) print('\n### 2 Mutation impact analysis') print('\n#### 2.1 $\Delta pkd$ predictive performance') mkdn = tbl_dpkd_metrics_overlap(OUT_PLT, train_p, verbose=VERBOSE, plot=False) print('\n#### 2.2 Stratified by location of mutation (binding pocket vs not in binding pocket)') m = tbl_dpkd_metrics_in_binding(OUT_PLT, RAW_PLT_CSV, verbose=VERBOSE, plot=False) ```

davis_gvpl_aflow

1. predictive performance

	mean predictive performance
cindex	0.472 $\pm$ 0.026
pcorr	-0.062 $\pm$ 0.050
scorr	-0.082 $\pm$ 0.078
mse	2.123 $\pm$ 0.099
mae	1.122 $\pm$ 0.049
rmse	1.455 $\pm$ 0.034

2 Mutation impact analysis

2.1 $\Delta pkd$ predictive performance

	with overlap	without overlap	p-val
pcorr	0.005 $\pm$ 0.012	0.005 $\pm$ 0.012	1
scorr	0.003 $\pm$ 0.025	0.003 $\pm$ 0.025	1
mse	1.990 $\pm$ 0.025	1.990 $\pm$ 0.025	1
mae	0.980 $\pm$ 0.013	0.980 $\pm$ 0.013	1
rmse	1.411 $\pm$ 0.009	1.411 $\pm$ 0.009	1

2.2 Stratified by location of mutation (binding pocket vs not in binding pocket)

	counts
wt	981
pckt	708
not pckt	241

	mutation in pocket	mutation NOT in pocket	p-val
pcorr	0.004 $\pm$ 0.015	0.025 $\pm$ 0.032	0.5777
scorr	-0.009 $\pm$ 0.026	0.067 $\pm$ 0.071	0.3433
mse	1.992 $\pm$ 0.030	1.950 $\pm$ 0.065	0.5777
mae	1.000 $\pm$ 0.009	0.981 $\pm$ 0.021	0.4374
rmse	1.411 $\pm$ 0.011	1.396 $\pm$ 0.023	0.5578

davis_gvpl

1. predictive performance

	mean predictive performance
cindex	0.469 $\pm$ 0.007
pcorr	-0.058 $\pm$ 0.020
scorr	-0.095 $\pm$ 0.021
mse	2.114 $\pm$ 0.040
mae	1.148 $\pm$ 0.017
rmse	1.454 $\pm$ 0.014

2 Mutation impact analysis

2.1 $\Delta pkd$ predictive performance

	with overlap	without overlap	p-val
pcorr	0.008 $\pm$ 0.014	0.008 $\pm$ 0.014	1
scorr	0.023 $\pm$ 0.011	0.023 $\pm$ 0.011	1
mse	1.983 $\pm$ 0.029	1.983 $\pm$ 0.029	1
mae	0.974 $\pm$ 0.011	0.974 $\pm$ 0.011	1
rmse	1.408 $\pm$ 0.010	1.408 $\pm$ 0.010	1

2.2 Stratified by location of mutation (binding pocket vs not in binding pocket)

	counts
wt	981
pckt	725
not pckt	256

	mutation in pocket	mutation NOT in pocket	p-val
pcorr	0.012 $\pm$ 0.012	-0.008 $\pm$ 0.052	0.7244
scorr	0.014 $\pm$ 0.019	0.025 $\pm$ 0.024	0.7289
mse	1.977 $\pm$ 0.024	2.016 $\pm$ 0.105	0.7244
mae	0.985 $\pm$ 0.014	0.979 $\pm$ 0.023	0.8043
rmse	1.406 $\pm$ 0.009	1.418 $\pm$ 0.037	0.7599

davis_aflow

1. predictive performance

	mean predictive performance
cindex	0.446 $\pm$ 0.018
pcorr	-0.127 $\pm$ 0.059
scorr	-0.172 $\pm$ 0.051
mse	2.253 $\pm$ 0.117
mae	1.236 $\pm$ 0.032
rmse	1.499 $\pm$ 0.039

2 Mutation impact analysis

2.1 $\Delta pkd$ predictive performance

	with overlap	without overlap	p-val
pcorr	-0.007 $\pm$ 0.010	-0.007 $\pm$ 0.010	1
scorr	-0.019 $\pm$ 0.015	-0.019 $\pm$ 0.015	1
mse	2.015 $\pm$ 0.019	2.015 $\pm$ 0.019	1
mae	0.922 $\pm$ 0.024	0.922 $\pm$ 0.024	1
rmse	1.419 $\pm$ 0.007	1.419 $\pm$ 0.007	1

2.2 Stratified by location of mutation (binding pocket vs not in binding pocket)

	counts
wt	981
pckt	708
not pckt	241

	mutation in pocket	mutation NOT in pocket	p-val
pcorr	-0.013 $\pm$ 0.009	0.031 $\pm$ 0.024	0.117
scorr	-0.039 $\pm$ 0.018	0.063 $\pm$ 0.030	0.0208
mse	2.027 $\pm$ 0.018	1.937 $\pm$ 0.048	0.117
mae	0.922 $\pm$ 0.023	0.951 $\pm$ 0.027	0.4358
rmse	1.424 $\pm$ 0.006	1.391 $\pm$ 0.017	0.1167

PDBbind_gvpl_aflow

1. predictive performance

	mean predictive performance
cindex	0.641 $\pm$ 0.011
pcorr	0.415 $\pm$ 0.028
scorr	0.411 $\pm$ 0.039
mse	0.947 $\pm$ 0.034
mae	0.754 $\pm$ 0.009
rmse	0.972 $\pm$ 0.018

2 Mutation impact analysis

2.1 $\Delta pkd$ predictive performance

	with overlap	without overlap	p-val
pcorr	0.176 $\pm$ 0.012	0.037 $\pm$ 0.035	0.0058
scorr	0.099 $\pm$ 0.009	0.046 $\pm$ 0.027	0.0974
mse	1.649 $\pm$ 0.024	1.927 $\pm$ 0.071	0.0058
mae	0.899 $\pm$ 0.013	1.014 $\pm$ 0.010	0.0001
rmse	1.284 $\pm$ 0.009	1.387 $\pm$ 0.025	0.005

2.2 Stratified by location of mutation (binding pocket vs not in binding pocket)

	counts
wt	981
pckt	708
not pckt	241

	mutation in pocket	mutation NOT in pocket	p-val
pcorr	0.180 $\pm$ 0.013	0.110 $\pm$ 0.025	0.0374
scorr	0.108 $\pm$ 0.010	0.050 $\pm$ 0.046	0.2533
mse	1.640 $\pm$ 0.026	1.781 $\pm$ 0.050	0.0374
mae	0.904 $\pm$ 0.010	0.981 $\pm$ 0.008	0.0003
rmse	1.280 $\pm$ 0.010	1.334 $\pm$ 0.019	0.0365