search

baaivision / EVA

EVA Series: Visual Representation Fantasies from BAAI
MIT License
2.31k stars 167 forks source link

Cannot find Evaluation Metrics on LVIS #123

Closed FrancoisPorcher closed 1 year ago

FrancoisPorcher commented 1 year ago

Hi,

I launched the evaluation on LVIS with the following command:

python tools/lazyconfig_train_net.py \ --num-gpus 8 --num-machines 1 --machine-rank 0 \ --config-file projects/ViTDet/configs/eva2_o365_to_lvis/eva2_o365_to_lvis_cascade_mask_rcnn_vitdet_l_8attn_1536_lrd0p8.py \ --eval-only \ train.model_ema.use_ema_weights_for_eval_only=True \ model.roi_heads.use_soft_nms=True \ model.roi_heads.class_wise=True \ model.roi_heads.method=linear \ model.roi_heads.iou_threshold=0.6 \ model.roi_heads.maskness_thresh=0.5 \ dataloader.evaluator.max_dets_per_image=1000 \ train.init_checkpoint=/private/home/francoisporcher/EVA/EVA-02/checkpoints/eva02_L_lvis_sys_o365.pth

image

And it runs on the whole data loader, but I cannot find the metrics in the logs (AP, etc..). Are they saved elsewhere? Thanks!

FrancoisPorcher commented 1 year ago

On closer inspections; the evaluation on LVIS does not work:

image

It's the line 210, results = evaluator.evaluate() which returns a None. Any idea?

FrancoisPorcher commented 1 year ago

Edit: It works, it's just that there is a comm.gather(), and when you launch this on several GPUS, there is a master GPU and the other ones are empty. The script is super long to run for evaluation, I am sure there are ways to make it faster (for example the evaluator is adding sequentially predictions to the "._predictions", which fragments the memory