🐛 Bug

Env: AWS SageMaker Jupyter Notebook

Comet Code: !comet-score -s {src_file} -t Candidate_file -r {ref_file}

Error along with output: Seed set to 1 Fetching 5 files: 100%|████████████████████████| 5/5 [00:00<00:00, 73071.50it/s] Lightning automatically upgraded your loaded checkpoint from v1.8.3.post1 to v2.1.0. To apply the upgrade to your files permanently, run python -m pytorch_lightning.utilities.upgrade_checkpoint ../.cache/huggingface/hub/models--Unbabel--wmt22-comet-da/snapshots/371e9839ca4e213dde891b066cf3080f75ec7e72/checkpoints/model.ckpt Encoder model frozen. /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/pytorch_lightning/core/saving.py:177: Found keys that are not in the model state dict but in the checkpoint: ['encoder.model.embeddings.position_ids'] /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/torch/cuda/init.py:611: UserWarning: Can't initialize NVML warnings.warn("Can't initialize NVML") GPU available: False, used: False TPU available: False, using: 0 TPU cores IPU available: False, using: 0 IPUs HPU available: False, using: 0 HPUs Predicting DataLoader 0: 0%| | 0/94 [00:00<?, ?it/s]Traceback (most recent call last): File "/home/ec2-user/anaconda3/envs/python3/bin/comet-score", line 8, in sys.exit(score_command()) File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/comet/cli/score.py", line 225, in score_command outputs = model.predict( File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/comet/models/base.py", line 627, in predict predictions = trainer.predict( File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 865, in predict return call._call_and_handle_interrupt( File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt return trainer_fn(*args, kwargs) File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 904, in _predict_impl results = self._run(model, ckpt_path=ckpt_path) File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 990, in _run results = self._run_stage() File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1031, in _run_stage return self.predict_loop.run() File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/pytorch_lightning/loops/utilities.py", line 181, in _decorator return loop_run(self, *args, kwargs) File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/pytorch_lightning/loops/prediction_loop.py", line 122, in run self._predict_step(batch, batch_idx, dataloader_idx, dataloader_iter) File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/pytorch_lightning/loops/prediction_loop.py", line 250, in _predict_step predictions = call._call_strategy_hook(trainer, "predict_step", step_args) File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 309, in _call_strategy_hook output = fn(args, kwargs) File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 429, in predict_step return self.lightning_module.predict_step(*args, kwargs) File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/comet/models/base.py", line 430, in predict_step model_outputs = Prediction(scores=self(batch).score) File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(args, kwargs) File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/comet/models/regression/regression_metric.py", line 273, in forward return self.estimate(src_sentemb, mt_sentemb, ref_sentemb) File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/comet/models/regression/regression_metric.py", line 245, in estimate return Prediction(score=self.estimator(embedded_sequences).view(-1)) File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/transformers/utils/generic.py", line 327, in init raise TypeError( TypeError: comet.models.utils.Prediction is not a dataclasss. This is a subclass of ModelOutput and so must use the @dataclass decorator.

It was working earlier for many weeks but with last week its failing with above error. Even out of the box, Comet sample code is failing with the same error.

To Reproduce

Code form: https://github.com/Unbabel/COMET

echo -e "10 到 15 分钟可以送到吗\nPode ser entregue dentro de 10 a 15 minutos?" >> src.txt echo -e "Can I receive my food in 10 to 15 minutes?\nCan it be delivered in 10 to 15 minutes?" >> hyp1.txt echo -e "Can it be delivered within 10 to 15 minutes?\nCan you send it for 10 to 15 minutes?" >> hyp2.txt echo -e "Can it be delivered between 10 to 15 minutes?\nCan it be delivered between 10 to 15 minutes?" >> ref.txt

comet-score -s src.txt -t hyp1.txt -r ref.txt

The code fail & give the error as TypeError: comet.models.utils.Prediction is not a dataclasss. This is a subclass of ModelOutput and so must use the @dataclass decorator.

Expected behaviour

Predicting DataLoader 0: 100%|████████████████| 188/188 [08:53<00:00, 2.84s/it] Candidate_file Segment 0 score: 0.8637 Candidate_file Segment 1 score: 0.5772

This is how the output should come.

Screenshots

Environment

OS: AWS Sagemaker Jupyter Notebook on ml.m5.xlarge (4 CPU, 16 GB Memory) Packaging

!pip install --upgrade pip -q

!pip install unbabel-comet Version pytorch-lightning<3.0.0,>=2.0.0->unbabel-comet) (1.3.1)

Additional context

The code was working fine for many weeks but last week, the code failed everywhere including the Google Colab notebook.

Unbabel / COMET

Comet Evaluation Metric Error with Command line Utility #170