search

sidhomj / DeepTCR

Deep Learning Methods for Parsing T-Cell Receptor Sequencing (TCRSeq) Data
https://sidhomj.github.io/DeepTCR/
MIT License
110 stars 40 forks source link

run DeepTCR using more than one GPUs #47

Closed qidi-yang closed 2 years ago

qidi-yang commented 3 years ago

Hi,

I have a few questions regarding using GPU.

  1. I was wondering if I can run DeepTCR using multiple GPUs. I noticed that I am allowed to select which GPU I want to put the graph and train on if I have a multi-GPU environment. But does that mean I can only specify one GPU?
  2. I saw "I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero" in my log. Some people say this is just a warning instead of an error and can be simply ignored. Have you encountered the same issue before? Any insight will be greatly appreciated.

Also, I was wondering if you could provide some insights on training an imbalanced dataset (binary classification) for this algorithm. Would you suggest using a balanced training dataset or including as much data as possible?

Thanks for your time and help!

sidhomj commented 2 years ago

Hi!

  1. In regards to multiple GPU training, I have not yet implemented this yet. In the coming year, it's one of my goals to refactor DeepTCR to use tensorflow's multi-GPU capabilities. Stay tuned! For now, single GPU training is all that is available!
  2. I have not personally seen this error (to my knowledge), but I believe it can be ignored. One way to check is to run the tutorials with known antigen specificities and just make sure the results replicate.
  3. Finally, in regards to unbalanced datasets, depending on how imbalanced they are, one can sometimes train the models without adjusting the weights of the losses. However, DeepTCR has two options to deal with this explicitly. You can use the keyword weight_by_class and set it to True for repertoire/sequence classifier. This computes a weight as a function of the frequency of the class. The second way to do this is to provide the weights of each class explicitly with the key word class_weights. See the docs for how to use.

https://sidhomj.github.io/DeepTCR/api/#DeepTCR.DeepTCR.DeepTCR_WF.Monte_Carlo_CrossVal

qidi-yang commented 2 years ago

Thanks a lot for the detailed reply! Those are very helpful! A quick additional question -- in terms of hyperparameter tuning, did you do a grid search, or is there any automated hyperparameter tuning package you would recommend?

sidhomj commented 2 years ago

I haven't done extensive grid searches in the past. In general, these models tend to fit pretty well with minimum adjustment. However, to optimize performance, thought usually has to be given to the biological problem and let that guide the parameters. In general though, running these models "out of the box" should usually tell you if there is a signal or not. I really designed DeepTCR not to be a predictive tool but more use predictive power to reveal biological insight. Maximizing the AUC a few points here or there I don't think is usually worthwhile endeavor because it usually doesn't change the biological conclusion/insight. However, you should feel free to look at the code in this repository as well as in the DeepTCR COVID repository (https://github.com/sidhomj/DeepTCR_COVID19) to get a sense for how models can be trained.