What is the mAP accuracy on openimage dataset?

[chengdazhi]

Hi, I see that in openimages2 branch, you train for 14 epochs. What are the results?

[bharatsingh430]

we didn't calculate, but the model is available. No one working on this repo can decipher the tensorflow syntax yet, which is needed to evaluate the model :( Openimages is a new dataset and finding the right set of hyper-parameters is going to take some time (may be years, like coco). You can try with 7 epochs as well and may be that also gets the same result (or better).

[bharatsingh430]

I think we trained 5,10,12 and not 14 if I remember correctly for this model, (14 was for a model which we tried to train from scratch, but the imagenet init with 5,10,12 turned out to be better). We did use an internal eval script to get the mAP scores (with that the number was around 45% for 600 classes and it was on 35,000 images instead of all the 40,000)...that should give you a rough idea, scales for testing were same as training (768,1024) and (512,512)

[chengdazhi]

Hi, thanks for your information, it's very helpful. Could you tell me why your model is in tensorflow format?

Another question is that I see in your openimage config file that you used a pretrained model called CRCNN-1epoch, could you tell me what is that?

Also, the learning rate for openimage(0.075) is much higher than the one for coco(0.015), is this finetuned already?

[chengdazhi]

Besides, is the mAP you mentioned calculated on 0.5 IoU threshold or averaged over multiple thresholds like in coco?

[bharatsingh430]

its at 0.5, the model is in mxnet. Eval script needs tensorflow code. I think the learning rate here is higher as there are 2 step downs, and the step factor is 0.2 instead of 0.1. The naming could be because the code might have crashed and we did a resume in the middle. We didn't clean the openimages repo, we will fix those things soon after we finish other things like end to end training with negatives and mask inference.

[chengdazhi]

Thanks! Really cleared things up!

One more thing, can you by any chance share the proposal file used to train faster rcnn on openimage?

[bharatsingh430]

you can use this model to get the proposals as well but we didn't use negative mining to train on openimages as our goal was to only use it as a pre-training dataset and a 2-3 % boost wouldn't have affected numbers on coco

[chengdazhi]

Hi, the annotation file you are using seems to be different from the one I can get from https://storage.googleapis.com/openimages/web/download.html

Specifically, your annotation .csv file seems to contain absolute coordinates and image height/width, but in the official annotation file (at least right now) each box only has relative position in each image (from 0.0 to 1.0). And it doesn't contain image height and width. Could you perhaps share the annotation file you are using? Many thanks!

[chengdazhi]

More specifically, in your annotation file you have a is_crowd tag for each box, but in original annotations, there are five tags including isOccluded, isTruncated, isGroupOf, isDepiction and isInside. How do you define is_crowd?

[bharatsingh430]

using isGroupOf

[chengdazhi]

Thanks! BTW, in the symbol file in openimages branch, the MultiProposalTarget has four outputs, and isn't compatible with the one in mxnet, which has five with an additional label_weight output. Could you perhaps share the right symbol file? Thanks!

[bharatsingh430]

did you switch the mxnet branch?

[chengdazhi]

Thanks for your reply, I am using the SNIPER-mxnet specified by git submodule file. It's the 89799b5 commit under SNIPER-mxnet-open branch. The problem is that the MultiProposalTarget operator has five outputs, but in the symbol file it only accepts four, excluding the label_weight output.

[bharatsingh430]

I updated it

[chengdazhi]

Thanks a lot!

[stalkermustang]

@bharatsingh430 pls help me. I correctly writing train file for model? example line: 000002b66c9c498e,zzzz342,0.0125,0.14843800000000001,0.19531199999999999,0.5875,0,1024,1024 where : 000002b66c9c498e - img name(path) zzzz342 - class(should i replace letters?) 0.0125,0.14843800000000001,0.19531199999999999,0.5875 - coordinates( from 0 to 1, not int from 0 to 415) and other i understand.