"comet-ml not installed" error in Trainer (despite comet-ml being installed)

[przecze]

System Info

- `transformers` version: 4.19.4
- Platform: Linux-4.19.0-17-amd64-x86_64-with-glibc2.31
- Python version: 3.9.6
- Huggingface_hub version: 0.4.0
- PyTorch version (GPU?): 1.11.0+cu102 (False)
- Tensorflow version (GPU?): 2.4.1 (False)
- Flax version (CPU?/GPU?/TPU?): 0.4.0 (cpu)
- Jax version: 0.3.4
- JaxLib version: 0.3.2
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

Who can help?

@sg

Information

• [ ] The official example scripts
• [X] My own modified scripts

Tasks

• [X] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [ ] My own task or dataset (give details below)

Reproduction

1. Install comet-ml (in my case comet-ml==3.31.3)
2. Create TrainingArguments with `report-to='comet_ml'
3. Try to instantiate Trainer

This can be reproduced by adding report_to='comet_ml' to training arguments in this notebook: https://github.com/NielsRogge/Transformers-Tutorials/blob/master/BERT/Fine_tuning_BERT_(and_friends)_for_multi_label_text_classification.ipynb

Following error happens when creating the Trainer:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_5296/3132099784.py in <module>
----> 1 trainer = Trainer(
      2     model,
      3     args,
      4     train_dataset=encoded_dataset["train"],
      5     eval_dataset=encoded_dataset["validation"],

/opt/conda/lib/python3.9/site-packages/transformers/trainer.py in __init__(self, model, args, data_collator, train_dataset, eval_dataset, tokenizer, model_init, compute_metrics, callbacks, optimizers, preprocess_logits_for_metrics)
    444         default_callbacks = DEFAULT_CALLBACKS + get_reporting_integration_callbacks(self.args.report_to)
    445         callbacks = default_callbacks if callbacks is None else default_callbacks + callbacks
--> 446         self.callback_handler = CallbackHandler(
    447             callbacks, self.model, self.tokenizer, self.optimizer, self.lr_scheduler
    448         )

/opt/conda/lib/python3.9/site-packages/transformers/trainer_callback.py in __init__(self, callbacks, model, tokenizer, optimizer, lr_scheduler)
    288         self.callbacks = []
    289         for cb in callbacks:
--> 290             self.add_callback(cb)
    291         self.model = model
    292         self.tokenizer = tokenizer

/opt/conda/lib/python3.9/site-packages/transformers/trainer_callback.py in add_callback(self, callback)
    305 
    306     def add_callback(self, callback):
--> 307         cb = callback() if isinstance(callback, type) else callback
    308         cb_class = callback if isinstance(callback, type) else callback.__class__
    309         if cb_class in [c.__class__ for c in self.callbacks]:

/opt/conda/lib/python3.9/site-packages/transformers/integrations.py in __init__(self)
    667     def __init__(self):
    668         if not _has_comet:
--> 669             raise RuntimeError("CometCallback requires comet-ml to be installed. Run `pip install comet-ml`.")
    670         self._initialized = False
    671         self._log_assets = False

RuntimeError: CometCallback requires comet-ml to be installed. Run `pip install comet-ml`.

Expected behavior

A Trainer is successfully created with cometml callback enabled.

[LysandreJik]

cc @sgugger

[sgugger]

As the error message indicates, you need to have cometml installed to use it report_to="comet_ml"

RuntimeError: CometCallback requires comet-ml to be installed. Run `pip install comet-ml`.

It also tells you exactly which command to run to fix this: pip install comet-ml.

[przecze]

Hey, The issue here is that error appears despite cometml being installed (with pip).

EDIT: Edited the issue title to make it more clear.

On Mon, Jul 4, 2022, 14:33 Sylvain Gugger @.***> wrote:

As the error message indicates, you need to have cometml installed to use it report_to="comet_ml"

RuntimeError: CometCallback requires comet-ml to be installed. Run pip install comet-ml.

It also tells you exactly which command to run to fix this: pip install comet-ml.

— Reply to this email directly, view it on GitHub https://github.com/huggingface/transformers/issues/17691#issuecomment-1173767326, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF7MPQSGKFHH4UZWW3JTEWLVSLKYRANCNFSM5YURU4KQ . You are receiving this because you authored the thread.Message ID: @.***>

[sgugger]

Did you properly initialize it with your API key then?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[mani-rai]

@sgugger How to do it? In this doc, there's no mentioning about API key in comet callback. I tried set up COMET_API_KEY, COMET_MODE, COMET_PROJECT_NAME inside function that runs on spawn, but no luck so far. Also downgraded comet-ml till 3.1.17.

os.environ["COMET_API_KEY"] = "<api-key>" os.environ["COMET_MODE"] = "ONLINE" os.environ["COMET_PROJECT_NAME"] = "<project-name>"

[sgugger]

Maybe open an issue with them? We did not write this integration with comet-ml and we don't maintain it. It was written by the Comet team :-)

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[frmccann97]

This is still an issue

[maximejkb]

This is still an issue, please re-open or address it -- none of the suggested methods of integrating with Comet ML are working for me -- neither the report_to="comet_ml" approach or the manual compute_metrics approach from this tutorial (https://www.comet.com/docs/v2/integrations/ml-frameworks/huggingface/).

[ngctnnnn]

This is still an issue right now, please kindly consider reopen and resolve this problem ! Thank you.

[guyshur]

Still happens to me with comet_ml installed with both conda and pip.

[amyeroberts]

@ngctnnnn @guyshur @frmccann97 @maximejkb As mentioned above, we didn't add and don't maintain the comet-ml integration. Have you raised an issue in their relevant repo?