search

anosorae / IRRA

Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrieval (CVPR 2023)
MIT License
205 stars 27 forks source link

A small multi GPU training bug #5

Closed Aotle closed 1 year ago

Aotle commented 1 year ago

It's on lines 98-100 of IRRA/processor/processor.py

if args.distributed: 
     top1 = evaluator.eval(model.module.eval())
else:
     top1 = evaluator.eval(model.eval())
anosorae commented 1 year ago

Thank you for pointing out this issue.

Aotle commented 1 year ago

lines 108 of IRRA/processor/processor.py

if get_rank() == 0:
    logger.info(f"best R1: {best_top1} at epoch {arguments['epoch']}")
anosorae commented 1 year ago

We have only used a single GPU in our experiments and have not verified the validity of the multi-GPU training code, so we cannot guarantee its reliability if you want to train with DDP.

Anyway, you are welcome to keep submitting bugs about multi GPU training and we will fix it as soon as possible.

Pefect96 commented 1 year ago

lines 108 of IRRA/processor/processor.py

if get_rank() == 0:
    logger.info(f"best R1: {best_top1} at epoch {arguments['epoch']}")

I want to know if the multi GPU training is work?