eval_metric is missing in Pretrainer

[sooeun67]

Describe the bug eval_metric is not available in TabNetPretrainer.fit even though it is described as one of parameters -- seems like you forgot eval_metric to be included in parameters

What is the current behavior? got an error TypeError: fit() got an unexpected keyword argument 'eval_metric'

If the current behavior is a bug, please provide the steps to reproduce.

Expected behavior

Screenshots

Other relevant information: poetry version:
python version: Operating System: Additional tools:

Additional context

[Optimox]

What kind of evaluation metric would you like to use during pretraining?

[sooeun67]

Hey, thanks for the prompt reply! I was hoping to have default metrics (such as logloss, auc for classifier) available during pretraining as well as custom metric as documented

[Optimox]

The pretrainer is an unsupervised training, there is no target to predict, the model is trying to predict missing inputs it received. So both logloss or AUC does not make any sense.

Custom metric could be used but I think it's would be a minor improvement. What kind of unsupervised metric would you use ?

[sooeun67]

ah my bad -- you're right. I am not sure if using one of the traditional metrics of unsupervised learning, such as kmeans clustering, rand index, etc, is a proper way to evaluate pretraining either.

What about having a module that compares/visualizes reconstructed x with original x ? My intention was to confirm the performance of pretrainer.

[Optimox]

It's an interesting topic, at the moment you can expect to have a 'working' reconstruction as long as the pretraining loss is bellow 1.

Feel free to propose some other methods in a PR to discus more concretely