Closed dolevr closed 4 years ago
As https://detectron2.readthedocs.io/tutorials/datasets.html#metadata-for-datasets said, evaluator_type
is used by builtin datasets and you should specify the evaluator to be used in your training script.
There is currently no "simple" evaluator. So if your dataset is not in a standard format, we currently cannot evaluate it. It would be very nice to have one.
Thanks, will try to implement one if time permits.
There is currently no "simple" evaluator... It would be very nice to have one.
@ppwwyyxx Can you check out maskrcnn-benchmark:#1104 and maskrcnn-benchmark:#1096?
I would be happy to implement something similar here.
And yeah, thanks for the supercool repo :)
https://github.com/facebookresearch/maskrcnn-benchmark/pull/1096 seems similar what we want to do here. You're welcome to contribute something alike here!
I think it's possible to make the existing COCOEvaluator
support a new dataset in its __init__
directly: if the dataset is not originally in coco format, create the COCO
object by getting the data from DatasetRegistry and converting the dataset dicts to COCO-format json. Then, add the json file path to the associated metadata so this conversion only happens once.
@ppwwyyxx, if I understood correctly the thing that is missing from a general "simple" evaluator is a DatasetRegistry 2 COCO-json converter, is it correct?
In the meantime I have started to modularize the mAP evaluation process that works with COCO-format json. My plan is to reproduce results from VOC, COCO and CityScapes with the same toolkit: you can find it here. So far the Precision and Recall curve computing is ready
As soon as I can validate the COCO scores with it, I'll make a PR
If you have converted all results to COCO-json, why not just use cocoapi to evaluate it?
there are multiple reasons for having a generic set of utility functions for evaluating the same metric:
Just wanted to mention that if you follow the Collab example in this repo and you implement your own get_balloon_dicts(), you have to cast your bbox array to Python ints instead of np.uint64 because there's a problem with json.dump's serialization of numpy ints(version 1.17.2)
What I did was generate the dict for my dataset and go through all the annotations and cast the bbox parameters to Python ints, and then I was able to dump the whole dictionary.
Make sense. I think we should convert them to float before dumping the json.
done by #175
@ppwwyyxx, if I understood correctly the thing that is missing from a general "simple" evaluator is a DatasetRegistry 2 COCO-json converter, is it correct?
In the meantime I have started to modularize the mAP evaluation process that works with COCO-format json. My plan is to reproduce results from VOC, COCO and CityScapes with the same toolkit: you can find it here. So far the Precision and Recall curve computing is ready
As soon as I can validate the COCO scores with it, I'll make a PR
I was hoping to use your evaluator to get a recall curve with my COCO dataset. Just noticed though that it says "AUC evaluation done, precision recall computation is wrong", so I was a bit hesitant to try to use it. Any updates here? Thanks!
@ppwwyyxx I am getting some good results from Detectron2 model for my instance segmentation as below shows; the AP for testing dataset is about 88%. Testing data is not seen by the training model, so the by running below command and I can get the summarized AP values. However, how can I obtain the AP for my training dataset? so I can see whether I overfit the model or not when I tune the model parameters. Is there a parameter in the trainer model to adjust to get the ouput when training the model, or can I just do similar inference on my training dataset as what I am doing to testing dataset to get the summarized AP ? Thanks!
evaluator = COCOEvaluator(test_instance, cfg, False, output_dir=cfg.OUTPUT_DIR) val_loader = build_detection_test_loader(cfg, test_instance) result = inference_on_dataset(trainer.model, val_loader, evaluator)
@bokelai1989 Did you ever figure this one out?
❓ Questions and Help
General questions about detectron2.
Thanks for all the great work! I have my own custom detection dataset(s) and a split to train/validation. I would like to run periodic evaluation during training.
I set:
If I understand correctly I need to set
MetadataCatalog.get(dataset_name).evaluator_type
but not sure what to use as evaluator. I have my ownget_json()
method since my data is not in any usual format. Is there a 'Simple detection evaluator'?