Open ohhan777 opened 1 month ago
Hi, @ohhan777, thanks for reporting the issue!
Could you try adding gather for metrics function in the script before using nested_to_cpu
?
https://huggingface.co/docs/accelerate/v0.33.0/en/package_reference/accelerator#accelerate.Accelerator.gather_for_metrics
smth like
outputs = accelerator.gather_for_metrics(outputs)
labels = accelerator.gather_for_metrics(batch["labels"])
Let me know if that helps, thanks!
I made the suggested modifications, but the same issue still persists.
The simplest solution seems to be processing the data on the GPUs without moving it to the CPU. I resolved the issue by commenting out the nested_to_cpu()
function and making a one-line modification in the convert_bbox_yolo_to_pascal()
function as follows:
boxes = boxes * torch.tensor([[width, height, width, height]]).to(boxes.device)
System Info
transformers
version: 4.45.0.dev0Who can help?
No response
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
When running
./examples/pytorch/object_detection/run_object_detection_no_trainer.py
in the transformers source code, it works well on a single GPU, but when running on multiple GPUs, an error occurs atmetric.compute()
inevaluation_loop()
function.Run:
accelerate launch run_object_detection_no_trainer.py --ignore_mismatched_sizes
Expected behavior