Implementation in pytorch

[garytann]

Good day

I am a final year student looking to implement this model training code using a pytorch framework instead of tensorflow. Do you think that is possible? After going through your paper and implementation I am not able to replicate the same results using the same parameters and datasets. The reason for implementing it in pytorch is to be used with a federated framework. I am also exploring to use your implementation in tensorflow federated learning framework.

Can I get some suggestion please?

Thank you & Best regards Gary

[NinaWie]

Hi Gary, thanks for your interest in the project; I think it would be great to provide a pytorch implementation! It is for sure possible to implement the same training in pytorch. Our best performing model was a pre-trained VGG16 network with a trainable head, which you could easiliy create in pytorch, for example see here: https://pytorch.org/hub/pytorch_vision_vgg/. Could you explain more specifically where you run into problems? Did you already implement it in Pytorch, and the results are different? I'm not sure how we can help you. Best Nina

[garytann]

Hi Nina

Thank you replying. Yes, so I have followed the way the POCUS dataset was being processed and applying a stratified k-fold method for splitting the datasets. For the pre-trained model I am also using a vgg16 net.

I already have a running training code implemented in PyTorch, you can have a look here: https://github.com/garytann/covid19-federated-learning/blob/develop/train_covid19.py

However, I am getting very poor training loss and accuracy. I would take a look at the resources that you provided, but at the same time could you see if there's anyway to help to improve the training

Thank you Gary

[jannisborn]

Hi @garytann, Thanks for the interest. Unfortunately, "poor training loss and accuracy" is not an issue of this repo but more a challenge in your research project. I'd recommend to either use our tensorflow/keras implementation of VGG (we verified that it works well on this dataset) or, if you want to stick to pytorch, use the wrapper provided in torchvision: https://pytorch.org/vision/main/models/generated/torchvision.models.vgg16.html

[garytann]

hi @jannisborn thanks for the reply. Can I just check with you, I have tried running your train_covid19 script with the hyper parameters provided in your publish paper, but it seems like I am not able to achieve the results that you've mentioned in the paper. The current settings are

Learning Rate: 0.01 with regularization
Batch size: 8
Epochs: 40
Model: vgg16
Training set len: 2541
Testing set len: 628

My results is as shown:

covid: precision(0.61), recall (0.57), f1-score(0.59)
pneumonia: precision(0.81), recall(0.80), f1-score(0.80)
regular: precision(0.64), recall(0.69), f1-score(0.67)

Could it be that I did not pre-process the datasets properly or could it be that I did not gather all the datasets? Please advice

Best Regards Gary

[jannisborn]

Yes, both could be the case and it's impossible to see from far what went wrong on your side.

[garytann]

Hi @jannisborn could you let me know what kind of information will you require on my end, so that you could see what went wrong? I did not made any amendments to the training script and only edited the hyper-parameters as mentioned. For the preprocessing I followed https://github.com/jannisborn/covid19_ultrasound/blob/master/pocovidnet/README.md to:

1. Download the datasets (although during this process I was not able to download some of the videos, some of the links are not reachable anymore)
2. Convert the videos to frames
3. Perform the cross-validation on the datasets

and run the script with the provided template

python scripts/train_covid19.py --data_dir ../data/cross_validation/ --fold 0 --epochs 40

[jannisborn]

If you're training on a different dataset (bc some videos are not available) then you can't really compare the performances anymore.

I would start by quantifying the deviation to the original dataset.

[jannisborn]

If you're training on a different dataset (bc some videos are not available) then you can't really compare the performances anymore.

I would start by quantifying the deviation to the original dataset.

[garytann]

@jannisborn thank you, can I just verify with you, is the dataset_metadata.csv, the dataset that you used to obtain the results? Is it the most up to date already? Also, could suggest some alternatives if I am unable to acquire some of the datasets?