songlab-cal / tape-neurips2019

Tasks Assessing Protein Embeddings (TAPE), a set of five biologically relevant semi-supervised learning tasks spread across different domains of protein biology. (DEPRECATED)
https://arxiv.org/abs/1906.08230
MIT License
118 stars 34 forks source link

Evaluation on contact prediction with binary metrics #23

Open nbuton opened 3 years ago

nbuton commented 3 years ago

Hello, In the script tape/analysis/contact_prediction/evaluate_contact_prediction_metrics.py at the line 62 there is metric(label, prediction), but in the true label we will also have* -1 when the position is not correctly determined in the dataset(from valid_mask column), so the default parameter average='binary' will not work. So Is there something I have missed ?

*label possible value in my understanding : 1 : contact, 0: not in contact, -1 : not enough informations

The exact error : ValueError: Target is multiclass but average='binary'. Please choose another average setting, one of [None, 'micro', 'macro', 'weighted'] at line metrics.append(metric(label, prediction)).