classification_report metric not working with Multilabel classification

[shahid-cloud]

Describe the bug I have successfully implemented my own multilabel classification notebook with 30 labels per your blog: https://towardsdatascience.com/multi-label-classification-using-bert-roberta-xlnet-xlm-and-distilbert-with-simple-transformers-b3e0cda12ce5

Now, I want to be able to get a sklearn classification_report as per the instructions here https://simpletransformers.ai/docs/tips-and-tricks/

However, when I run

`model = MultiLabelClassificationModel('roberta', 'roberta-large', num_labels=30, args=model_args)

Train the model

model.train_model(train_df=train_df, eval_df=eval_df, classification_report=sklearn.metrics.classification_report)`

I get the following error:

`

ValueError Traceback (most recent call last)

in 75 76 # Train the model ---> 77 model.train_model(train_df=train_df, eval_df=eval_df, classification_report=sklearn.metrics.classification_report) 78 79 ~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/simpletransformers/classification/multi_label_classification_model.py in train_model(self, train_df, multi_label, eval_df, output_dir, show_running_loss, args, verbose, **kwargs) 223 verbose=True, 224 args=args, --> 225 **kwargs, 226 ) 227 ~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/simpletransformers/classification/classification_model.py in train_model(self, train_df, multi_label, output_dir, show_running_loss, args, eval_df, verbose, **kwargs) 468 eval_df=eval_df, 469 verbose=verbose, --> 470 **kwargs, 471 ) 472 ~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/simpletransformers/classification/classification_model.py in train(self, train_dataloader, output_dir, multi_label, show_running_loss, eval_df, verbose, **kwargs) 860 silent=args.evaluate_during_training_silent, 861 wandb_log=False, --> 862 **kwargs, 863 ) 864 ~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/simpletransformers/classification/multi_label_classification_model.py in eval_model(self, eval_df, multi_label, output_dir, verbose, silent, **kwargs) 228 def eval_model(self, eval_df, multi_label=True, output_dir=None, verbose=False, silent=False, **kwargs): 229 return super().eval_model( --> 230 eval_df, output_dir=output_dir, multi_label=multi_label, verbose=verbose, silent=silent, **kwargs 231 ) 232 ~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/simpletransformers/classification/classification_model.py in eval_model(self, eval_df, multi_label, output_dir, verbose, silent, wandb_log, **kwargs) 960 961 result, model_outputs, wrong_preds = self.evaluate( --> 962 eval_df, output_dir, multi_label=multi_label, verbose=verbose, silent=silent, wandb_log=wandb_log, **kwargs 963 ) 964 self.results.update(result) ~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/simpletransformers/classification/multi_label_classification_model.py in evaluate(self, eval_df, output_dir, multi_label, prefix, verbose, silent, **kwargs) 233 def evaluate(self, eval_df, output_dir, multi_label=True, prefix="", verbose=True, silent=False, **kwargs): 234 return super().evaluate( --> 235 eval_df, output_dir, multi_label=multi_label, prefix=prefix, verbose=verbose, silent=silent, **kwargs 236 ) 237 ~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/simpletransformers/classification/classification_model.py in evaluate(self, eval_df, output_dir, multi_label, prefix, verbose, silent, wandb_log, **kwargs) 1128 preds = np.argmax(preds, axis=1) 1129 -> 1130 result, wrong = self.compute_metrics(preds, model_outputs, out_label_ids, eval_examples, **kwargs) 1131 result["eval_loss"] = eval_loss 1132 results.update(result) ~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/simpletransformers/classification/multi_label_classification_model.py in compute_metrics(self, preds, labels, eval_examples, multi_label, **kwargs) 244 245 def compute_metrics(self, preds, labels, eval_examples, multi_label=True, **kwargs): --> 246 return super().compute_metrics(preds, labels, eval_examples, multi_label=multi_label, **kwargs) 247 248 def predict(self, to_predict, multi_label=True): ~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/simpletransformers/classification/classification_model.py in compute_metrics(self, preds, model_outputs, labels, eval_examples, multi_label, **kwargs) 1310 extra_metrics = {} 1311 for metric, func in kwargs.items(): -> 1312 extra_metrics[metric] = func(labels, preds) 1313 1314 if multi_label: ~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/sklearn/metrics/_classification.py in classification_report(y_true, y_pred, labels, target_names, sample_weight, digits, output_dict, zero_division) 1969 """ 1970 -> 1971 y_type, y_true, y_pred = _check_targets(y_true, y_pred) 1972 1973 labels_given = True ~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/sklearn/metrics/_classification.py in _check_targets(y_true, y_pred) 88 if len(y_type) > 1: 89 raise ValueError("Classification metrics can't handle a mix of {0} " ---> 90 "and {1} targets".format(type_true, type_pred)) 91 92 # We can't have more than one value on y_type => The set is no more needed ValueError: Classification metrics can't handle a mix of multilabel-indicator and continuous-multioutput targets ` Can you please help with how I can get the classification report to work?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.