Result reproduction

[amhummer]

Hi,

Thank you very much for providing the code for your model!

I am trying to reproduce your results for this LSTM method for calculating antibody humanness, as presented in Wollacott et al., 2019.

Could you confirm whether the model on your GitHub is identical to the one used in the Wollacott paper and, if so, how to evaluate test data identically to how you did in the paper? If it is not the same model, could you please advise me how to tune the model on GitHub to obtain the same results reported in the paper and/or provide me with access to the pre-trained model that achieved the results reported in the paper?

I have tried to run the model on the human_test.txt data provided using two approaches based on your documentation (full code included below). With one approach (python ablstm.py eval … model_tmp.npy …), I obtain similar but not identical test set results to what is reported in the result_human_test.txt file. With the other approach (from ablstm import ModelLSTM…), I obtain much higher/poorer scores than expected or reported (~3, instead of ~0-1).

Any guidance on how to reproduce the model and results from the paper would be greatly appreciated. Thank you very much in advance for your time and help!

Best wishes, Alissa Hummer

My code to run the model on the provided test data:

(1) In the command line: python3 ablstm.py eval data/sample/human_test.txt saved_models/tmp/model_tmp.npy human_test-output.txt

(2) In a Python script:

from ablstm import ModelLSTM
model = ModelLSTM(embedding_dim=64, hidden_dim=64, device='cpu', gapped=True, fixed_len=True)

tst_fn = './data/sample/human_test.txt'
tst_scores = model.eval(fn=tst_fn, batch_size=512)

Also, I am unable to run the saved_models/tmp/lstm_0.589547.npy model, as I encounter the following error:

File "/data/pegasus/hummer/Hu-mAb-revisions/LSTM-test/peds2019/model.py", line 131, in load
  self.gapped = param_dict['gapped']
KeyError: 'gapped'

[xf3227]

Hi @amhummer, it has been days since the last commit. Sorry that I probably need some time to get familiar with my code again. I will respond by tonight.

[xf3227]

Hi @amhummer, thank you for opening the thread, but sorry that we may not be able to find the exact pre-trained model that 'result_human_test.txt' corresponds to. This work was collaborated by two institutions. We shared the common code base, but verified results independently to ensure the report being reliable. Since we have large population of samples for testing, a result was valid if different runs (with randomness) were statistically consistent.

model_tmp.npy was put in the repo for tutorial purpose while lstm_0.589547.npy was actually uploaded by mistake. I will soon add it to .gitignore and put a note in README. Sorry for any caused confusion. By the way, the error you encountered was caused by the inconsistency between the latest code and the pre-trained model which was based on an earlier code version and didn't have an attribute named gapped.

Our approach intended to offer as many flexibilities as possible for different needs. Researchers may prefer aligning their sequences by different schemes while some others want to apply algorithms on raw sequences directly. That's another reason why we didn't have it in mind to provide pre-trained models.

Hope this reply could be helpful. Be a bit off the topic, would you please tell me why you try reproducing the exact result?

[SenyorDrew]

Hi Alissa, I should have a copy of that pre-trained model that I used for the paper if you'd like to have it. Based on the comment above it sounds like there's a chance it may not be compatible with the current version of the code, but you can always try it and see. Let me know if you'd like me to send along that file.

Cheers, Andrew

[amhummer]

Hi @xf3227 and @SenyorDrew,

Many thanks for your quick and helpful responses.

Andrew – It would be great if you could share that file with me, thank you very much! How would work best for you to send it (e.g., via GitHub or something else)?

@xf3227 – I am benchmarking humanisation methods on a new dataset and as one of the best performing and most-recently published tools, I wanted to ensure your LSTM model is represented. I just wanted to check that I am achieving the recorded level of performance on your test data and as reported in your paper before trying it on my dataset.

Many thanks, Alissa

[xf3227]

Hi @amhummer Alissa, you are welcomed!

Greetings! @SenyorDrew Andrew, great to have you helping Alissa out! According to the error that Alissa reported, the previously saved model might be inconsistent with the latest code. There could potentially be more errors considering that the code structure was reworked for publishing on github. @amhummer when you have Andrew's model and encounter any code related issues, just let me know, I will guide you through to make it work.

[SenyorDrew]

Hi Alissa, I've attached a tarred gzipped file. You should be able to extract it using: tar -xzf peds_data.gz

You'll get a directory called "share" that contains:

• human_testing.txt: this is the 10,000 sequences that we used for testing in the paper. Note that this is different than what's in this git repository. The version on git is much larger (~76K), but I think it's a dataset that we used for testing early in development, but which wasn't as well processed as the final datasets. For the paper we ensured that all sequences used for training and testing were "higher quality", as described in the manuscript.
• mouse_testing.txt: the 10,000 mouse sequences that we used for testing in the paper.
• human_vh_aho.npy: this is the trained model that I used for "gapped" sequences. The gapped sequences are generated with the help of a really handy tool called ANARCI (which I know you are very familiar with :) ). Note that this model wasn't made for ungapped sequences.
• human_vh.npy: trained model that I use for ungapped sequences. I included this in case you want to score ungapped (unaligned) sequences
• lstm_scores.pdf: I put together a jupyter notebook to show that scores generated by the attached gapped model match those of the paper. note that I use some utilities and classes to facilitate the analysis (which I unfortunately can't attach) - however, these are just wrappers to simplify the calls to the models.
• paper_scores.tsv: This is a tsv file of LSTM scores for the 10,000 human and 10,000 mouse test sets. Hopefully this makes it a bit easier for you to see if you have everything working well. Each line has both the input sequence and the humanness score (see the jupyter notebook on how this was generated).

Hope this helps!

Cheers, Andrew

peds_data.gz

[amhummer]

Hi Andrew, This is very helpful – thank you so much for taking the time! Best, Alissa