[Help Wanted] Calculating mAP each epoch during Training

[AlexanderJGomez]

Hello, is there a recommended way to also calculate mAP during training, so for example after each epoch?

I've tried adding a new function to coco_eval.py mimicking what is done in the main function looking like the following:

def train_eval(model):
    SET_NAME = params['val_set']
    VAL_GT = f'datasets/{params["project_name"]}/annotations/instances_{SET_NAME}.json'
    VAL_IMGS = f'datasets/{params["project_name"]}/{SET_NAME}/'
    MAX_IMAGES = 10000
    coco_gt = COCO(VAL_GT)
    image_ids = coco_gt.getImgIds()[:MAX_IMAGES]

    model.eval()

    evaluate_coco(VAL_IMGS, SET_NAME, image_ids, coco_gt, model)

    _eval(coco_gt, image_ids, f'{SET_NAME}_bbox_results.json')

and changed L78 in coco_eval.py from

features, regression, classification, anchors = model(x)

to

with torch.no_grad():
            features, regression, classification, anchors = model(x)

Lastly, I've changed the train loop in train.py to the following:

if epoch % opt.val_interval == 0:
    model.eval()
    train_eval(model)
    loss_regression_ls = []
    loss_classification_ls = []

Unfortunately this gives me the following error:

File "train.py", line 353, in train(opt) File "train.py", line 288, in train train_eval(model) File "coco_eval.py", line 149, in train_eval evaluate_coco(VAL_IMGS, SET_NAME, image_ids, coco_gt, model) File "coco_eval.py", line 80, in evaluate_coco features, regression, classification, anchors = model(x) File ".../lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in call result = self.forward(*input, **kwargs) TypeError: forward() missing 1 required positional argument: 'annotations'

Does anyone know why I am having issues here? and is there a better way to caculate mAP while training?

[AlexanderJGomez]

For anyone who is interested, the only change needed to make this work is

if epoch % opt.val_interval == 0:
    model.eval()
    train_eval(model.model)
    loss_regression_ls = []
    loss_classification_ls = []

and this is because in the training loop, we are using the ModelWithLoss wrapper and it stores the EfficientDetBackbone in the model attribute

[aliencaocao]

Hi, tried your method, but it somehow break the argument parsing for train.py if I import anything from coco_eval (I need to import the train_eval function from coco_eval.py into train.py). This is probably because coco_eval.py also uses argparser. Any idea how to fix that?

train.py: error: unrecognized arguments: --batch_size 8 --lr 1e-3 --head_only True --num_epochs 10 --es_patience 2 --debug True --load_weights weights\efficientdet-d1.pth

[rizwanriaz-se]

@aliencaocao Have you found the solution? I am getting the same error as well :(