AUC Metric y_score dimension

[denesbartha]

Hi,

In all metrics defined in pytorch_tabnet/metrics.py variable y_score is passed to the appropriate sklearn metrics function except for AUC. For that case roc_auc_score(y_true, y_score[:, 1]) is being called instead of roc_auc_score(y_true, y_score).

Because of this I ran into an error during training:

ValueError: Expected 2D array, got 1D array instead:
array=[ 343.7495     333.73627    344.73627    520.656      345.73627
...
].

Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

Once I corrected the line in the class AUC to return roc_auc_score(y_true, y_score) the training succeeded.

My question is: is it intentional? If so how should I shape my data for training? (I use TabNetRegressor for multi-label classification, so my variable y_test is 2 dimensional.)

[Optimox]

Hello @denesbartha,

The reason behind this is that AUC is usually computed for binary classification, so people tend to use TabNetClassifier. As TabNetClassifier is generic and can take any number of classes, it ends up being 2 classes for binary classification task, so that is why AUC is computed this way.

If you have a multi task classification problem (i.e multiple classification task and not a single multi class classification task) you can use TabNetMultiTaskClassifier. If you have a multi task regression problem then TabNetRegressor is fine.

When you have non generic configurations like yours you need to customize the metrics https://github.com/dreamquark-ai/tabnet#custom-evaluation-metrics as it's impossible to imagine every setup in advance and add it to the default library's metrics.

[denesbartha]

Hi @Optimox,

Thanks for your quick response and explanation. Actually I would like to do multi-label classification and not multi task classification. In case of multi-label classification a single task is needed where more than one label is allowed as correct prediction.

I will define my own evaluation metric as you proposed.

[Optimox]

It might be just a matter of syntax, but what you described is what I call multi task classification :

• multi class classification : which animal is in the picture ["cat", "dog", "rabbit"] you have 3 classes represented as 0 , 1, 2 and must predict one of them
• multi task classification: which of these animals are in the picture ["cat", "dog", "rabbit"], you then have 3 binary tasks.

[denesbartha]

Oh I get it, it is indeed what I shall try out then, thanks. :)