Output changes when changing batchsize in test_DRD.py

[KiranTen63]

Hello! I very much enjoyed reading about this work, and I was able to successfully install and run all of the example code that you have provided. I modified the test_DRD.py code to print the dataframe that stores the prediction output. Here is the modification to the code, starting from line 108 in test_DRD.py:

` batch_drug_name.index = ikey_list[jargs['batchsize']:(j+1)args['batchsize']] one_protein= pd.concat([one_protein,batch_drug_name]) screened_pfam[protein]= one_protein

print(screened_pfam) df_all = pd.DataFrame() for protein in dark_proteins:`

Running this gives me the following output in addition to the original output:

{'P35462 (DRD3_HUMAN)': score3 W1 0.389289 W2 0.434220 W3 0.619866 W4 0.578805 W5 0.475355 .. ... W61 0.440514 W62 0.389829 W63 0.602567 W64 0.599470 W65 0.465283

I then tried changing the batchsize variable defined on line 43 from 5 to 13: batchsize=13 The result now comes as:

{'P35462 (DRD3_HUMAN)': score3 W1 0.352451 W2 0.398314 W3 0.594502 W4 0.547320 W5 0.424260 .. ... W61 0.622131 W62 0.509086 W63 0.423841 W64 0.578107 W65 0.514104

I did not expect the predicted value to change as a function of the batch size. Is this expected? I assumed that since the model is already trained and is only being loaded with fixed weights that the output should not change depending on batch size.

Thank you for your time,

Kiran

[BeatriceCai]

Hi @KiranTen63,

Thanks for your interest. And thanks for spotted the issue.

This is a test file for evaluating robustness by allowing dropout at inference time, which leads to changes of prediction scores. This equivalent to testing multiple trained models in an ensemble manner. Because dropout at inference time means only part of the embeddings are used, just like during training time. In this stress test, we still observed performance significantly better than baseline models. You may see from the above picture (PR-AUC for one protein against PLD+SIGN). Sorry we didn't make it clear in Readme.

We just uploaded another test_DRD_inference.py file which turned dropout off at inference time.

[BeatriceCai]

Hi @KiranTen63,

Our paper put emphasize on proposing new OOD algorithm. We didn't carry out extensive tuning or cherry picking the best one set of model parameters. If you want to use the model for a specific application, we encourage you to follow the described training procedure and carry out model tuning in a swiping style for the best performance.

[KiranTen63]

Hi @BeatriceCai ,

Thank you for the quick and informative response! I just repeated the same test with the new test_DRD_inference.py, and I am still getting the same issue as the original post. From a quick look at the "models*_test.py" files that you are now calling in the inference script, I still see that there are some torch Dropout layers being called here. Are these the layers that I would need to turn off in order to solve this issue.

Thank you as well for the suggestion to reproduce the full training and optimize the model tuning, I will definitely look into this further. However, my issue is not with optimizing the model for a specific task. Rather, I just want to understand how to make the results reproducible even with varying batch sizes. If a constant batch size is required, where can I set the seed value for the dropout layers? I am still not fully familiar with the code, so I apologize if I missed it somewhere, but do you have anything like the manual_seed function described below in the code?

https://stackoverflow.com/questions/52730405/does-pytorch-seed-affect-dropout-layers

Thank you again for your time!

Kiran

[BeatriceCai]

@KiranTen63 would you pls clone all new scripts and try again? The scores are consistent even when batch size changes at my end.

[KiranTen63]

I did get an import error at first saying " No module named 'utils_MAML_test' ", but after I renamed these module calls to "utils_MAML" and ran the code, the issue was fixed! Thank you for your help!!!

[BeatriceCai]

@KiranTen63 Thank you!