Multi-gpu inference returns AttributeError

[arturnn]

🐛 Bug

When trying to run scoring with COMET models via comet-score (although the issue is the same with comet-mbr when pre-scoring with qe models using multiple gpus) I get the following error. It works fine with --gpus 0 (running on cpu) or --gpus 1. I tested it with wmt22-comet-da and wmt21-comet-mqm models.

Traceback (most recent call last):
  File "/opt/conda/envs/test-3.11/bin/comet-score", line 8, in <module>
    sys.exit(score_command())
             ^^^^^^^^^^^^^^^
  File "/opt/conda/envs/test-3.11/lib/python3.11/site-packages/comet/cli/score.py", line 192, in score_command
    outputs = model.predict(
              ^^^^^^^^^^^^^^
  File "/opt/conda/envs/test-3.11/lib/python3.11/site-packages/comet/models/base.py", line 643, in predict
    predictions = pred_writer.gather_all_predictions()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/test-3.11/lib/python3.11/site-packages/comet/models/predict_writer.py", line 99, in gather_all_predictions
    [
  File "/opt/conda/envs/test-3.11/lib/python3.11/site-packages/comet/models/predict_writer.py", line 100, in <listcomp>
    flatten_predictions(torch.load(os.path.join(self.output_dir, f))[0])
  File "/opt/conda/envs/test-3.11/lib/python3.11/site-packages/comet/models/predict_writer.py", line 89, in flatten_predictions
    scores=torch.cat([pred.scores for pred in predictions], dim=0)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/test-3.11/lib/python3.11/site-packages/comet/models/predict_writer.py", line 89, in <listcomp>
    scores=torch.cat([pred.scores for pred in predictions], dim=0)
                      ^^^^^^^^^^^
AttributeError: 'str' object has no attribute 'scores'

To Reproduce

Run comet-score command with --gpus > 1 using the latest version of the package.

Expected behaviour

Segment and system scores should be returned, as is the case with single gpu inference.

Environment

OS: Debian 11 (bullseye) Packaging: tried both conda (with python 3.11.6) and standard virtual environment with installation via pip (python 3.9.2) Version: latest master & current PyPI (2.2.0)

[arturnn]

FYI: I submitted a PR with proposed fixes in #178

[ricardorei]

Thanks @arturnn I will look into the PR asap!

[ricardorei]

@arturnn haven't had time yet, but its not forgotten.

[OrianeN]

Hi ! I just got a very similar error ('dict' object has no attribute 'scores', also highlighting pred.scores in the trace), but in my case there is no GPU on the machine I used.

The first time I got the error I specified gpus=0 (I thought it was the correct way to make it run on CPU), but then I removed the parameter and got the same error as well.

%env TOKENIZERS_PARALLELISM=false

issue_data = [
    {
        "src": "10 到 15 分钟可以送到吗",
        "mt": "Can I receive my food in 10 to 15 minutes?",
        "ref": "Can it be delivered between 10 to 15 minutes?"
    },
    {
        "src": "Pode ser entregue dentro de 10 a 15 minutos?",
        "mt": "Can you send it for 10 to 15 minutes?",
        "ref": "Can it be delivered between 10 to 15 minutes?"
    }
]
comet_model = load_from_checkpoint(download_model("Unbabel/wmt22-comet-da"))
comet_output = comet_model.predict(issue_data, batch_size=8)
print(model_output)
print(model_output.system_scores)

Ouput:

[...]

Predicting DataLoader 0: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  3.09it/s]

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[56], line 15
      1 get_ipython().run_line_magic('env', 'TOKENIZERS_PARALLELISM=false')
      3 issue_data = [
      4     {
      5         "src": "10 到 15 分钟可以送到吗",
   (...)
     13     }
     14 ]
---> 15 comet_output = comet_model.predict(issue_data, batch_size=8)
     16 # comet_output = comet_model.predict(test_dataset_comet, batch_size=8)
     17 print(model_output)

File ~/.../venv-mt-models/lib/python3.10/site-packages/comet/models/base.py:652, in CometModel.predict(self, samples, batch_size, gpus, devices, mc_dropout, progress_bar, accelerator, num_workers, length_batching)
    648 elif gpus > 1 and not trainer.is_global_zero:
    649     # If we are not in the GLOBAL RANK we will return None
    650     exit()
--> 652 scores = torch.cat([pred.scores for pred in predictions], dim=0).tolist()
    653 if "metadata" in predictions[0]:
    654     metadata = flatten_metadata([pred.metadata for pred in predictions])

File ~/.../venv-mt-models/lib/python3.10/site-packages/comet/models/base.py:652, in <listcomp>(.0)
    648 elif gpus > 1 and not trainer.is_global_zero:
    649     # If we are not in the GLOBAL RANK we will return None
    650     exit()
--> 652 scores = torch.cat([pred.scores for pred in predictions], dim=0).tolist()
    653 if "metadata" in predictions[0]:
    654     metadata = flatten_metadata([pred.metadata for pred in predictions])

AttributeError: 'dict' object has no attribute 'scores'

I'm using unbabel-comet version 2.2.0 and Python 3.10, on Ubuntu 22.04

[Novemberl1]

I ran into the same problem when using test cases on huggingface.

AttributeError: 'dict' object has no attribute 'scores'

[sitfoxfly]

Facing the same issue: AttributeError: 'dict' object has no attribute 'scores' when running prediction on 1 GPU with the example from https://huggingface.co/Unbabel/wmt23-cometkiwi-da-xl. Looking forward to a fix!

[ricardorei]

I just publish a new release that should solve this.