bvanaken
commented
2 years ago Hello and thanks for your interest in our work!
We have indeed experimented with calculating the class weights to balance the loss (with the balance_classes parameter in our repo which uses the FARM method that you linked). We used this as one parameter in our hyperparameter search and found that it does not always improve results, but sometimes.
For the BERT-based models (without CORe-pretraining) we found that the best set of parameters was actually not using the balancing. The parameters for our reported results are:
'balance_classes': False, 'grad_acc': 10, 'dropout': 0.2, 'lr': 1e-05, 'warmup_steps': 50
Maybe you can try to run your setup with these parameters and see if it produces better results. We have not experimented with different splits (e.g. 50-50) as you did, but it would be interesting if this could further improve results in general.
Hope this helps, otherwise don't hesitate to come back for further questions!
Betty
NtaylorOX
Hello - this is not a technical/code issue, rather a request for information.
Amazing codebase by the way!
I have successfuly rebuilt the datasets for each of the clinical-outcomes - although my particular interest right now is the mortality prediction. As a baseline I am trying to replicate the results here - although using my own code for generating the BERT model followed by classifier, I have attempted a mixture of downsamping the dataset to have a 50-50 split of classes, or use all of the imbalanced class data and calculating class_weights for cross entropy, a implementation I belive your code uses (from the farm repo: https://farm.deepset.ai/_modules/farm/data_handler/data_silo.html#DataSilo.calculate_class_weights).
I am still finding the BERT based models are tending to predict the majority class - and my evaluation metrics are far from those reported in your paper (~0.7 vs 0.84).
Did your expierments use the class_weights with the full training dataset?
Any information would be much appreciated.