Getting the error : Results do not correspond to current coco set when running COCOEvaluator

If you do not know the root cause of the problem, please post according to this template:

Instructions To Reproduce the Issue:

Check https://stackoverflow.com/help/minimal-reproducible-example for how to ask good questions. Simplify the steps to reproduce the issue using suggestions from the above link, and provide them below:

Full runnable code or full changes you made:
```
CFUTrainer  - > DefaultTrainer
```

@classmethod def build_test_loader( cls, cfg: CfgNode, dataset_name: str, augmentations: Optional[List[Augmentation]] = None ) -> DataLoader: # type: ignore """ Create a test data loader.

    Returns
    -------
    iterable
    """
    return build_detection_test_loader(  # type: ignore
        cfg,
        dataset_name=dataset_name,
        mapper=CFUDatasetMapper(
            cfg,
            is_train=False,
            augmentations=augmentations if augmentations else cls.augmentations_test,
            recompute_boxes=True,  # to tighten the bboxes that got widened after transforms like rotation
        ),
    )

@classmethod
def build_evaluator(
    cls, cfg: CfgNode, dataset_name: str, output_folder: Optional[str] = None
) -> Union[DatasetEvaluators, DatasetEvaluator]:
    """
    Create evaluator(s) for a given dataset.

    This uses the special metadata "evaluator_type" associated with each builtin dataset.
    For your own dataset, you can simply create an evaluator manually in your
    script and do not have to worry about the hacky if-else logic here.
    """
    if output_folder is None:
        output_folder = os.path.join(cfg.OUTPUT_DIR, "inference")
    # Delete pre-existing evaluation folder to ensure a new one gets created and no old cache is being used
    if os.path.isdir(output_folder):
        logger.info(f"Overwriting existing: {output_folder}.")
        delete_folders([output_folder])

    # remove any dataset json_file metadata if any to ensure such cache gets recreated by the COCOEvaluator.
    # The cache issue is that as soon as the dataset gets extended, benchmarks using the old cache will be meaningless
    _metadata = MetadataCatalog.get(dataset_name)
    if hasattr(_metadata, "json_file"):
        del _metadata.json_file

    evaluator_list = [
        CFUEvaluator(),
        COCOEvaluator(dataset_name, cfg, True, output_folder),
    ]

    if len(evaluator_list) == 0:
        raise NotImplementedError(f"No Evaluator for the dataset {dataset_name}.")

    elif len(evaluator_list) == 1:
        return evaluator_list[0]

    return DatasetEvaluators(evaluator_list)

data_analysis.py:

def computekpis(cfg: CfgNode) -> DatasetEvaluatorResults: """Compute the KPIs for a given model and config.""" cfg = cfg.clone() # cfg can be modified by model model = CFUTrainer.buildmodel(cfg)

# a checkpointer is needed to load the weights into the model\n",
checkpointer = DetectionCheckpointer(model)
checkpointer.load(cfg_.MODEL.WEIGHTS)

res: DatasetEvaluatorResults = CFUTrainer.test(cfg_, model)
if cfg.TEST.AUG.ENABLED:
    res.update(CFUTrainer.test_with_TTA(cfg, model))
if comm.is_main_process():
    verify_results(cfg, res)
return res

2. What exact command you run:
detectron2/evaluation/evaluator.py
results = evaluator.evaluate()
3. __Full logs__ or other relevant observations:

2021-08-18 03:10:48 gpu-003 fvcore.common.checkpoint[26853] INFO [Checkpointer] Loading from ../models/02a08239-8220-48d0-85f2-c0c0425b4224/model_final.pth ... [08/18 03:10:48 d2.data.dataset_mapper]: [DatasetMapper] Augmentations used in inference: [RandomRotation(angle=[0.0, 360.0], interp=2)] [08/18 03:10:48 d2.data.datasets.coco]: Loaded 2 images in COCO format from /home/..../cfu_test/test.json [08/18 03:10:48 d2.data.common]: Serializing 2 elements to byte tensors and concatenating them all ... [08/18 03:10:48 d2.data.common]: Serialized dataset takes 0.00 MiB 2021-08-18 03:10:48 gpu-003 vxgioq_cfu_counting_ml.training.trainer[26853] INFO Overwriting existing: ../models/02a08239-8220-48d0-85f2-c0c0425b4224/inference. 2021-08-18 03:10:48 gpu-003 vxgioq_cfu_counting_ml.utils.conf[26853] INFO Deleting: ../models/02a08239-8220-48d0-85f2-c0c0425b4224/inference if exists... WARNING [08/18 03:10:48 d2.evaluation.coco_evaluation]: COCO Evaluator instantiated using config, this is deprecated behavior. Please pass in explicit arguments instead. [08/18 03:10:48 d2.evaluation.coco_evaluation]: 'coco_test' is not registered by `register_coco_instances`. Therefore trying to convert it to COCO format ... [08/18 03:10:48 d2.data.datasets.coco]: Converting annotations of dataset 'coco_test' to COCO format ...) [08/18 03:10:48 d2.data.datasets.coco]: Loaded 2 images in COCO format from /home/..../cfu_test/test.json [08/18 03:10:48 d2.data.datasets.coco]: Converting dataset dicts into COCO format [08/18 03:10:48 d2.data.datasets.coco]: Conversion finished, #images: 2, #annotations: 4 [08/18 03:10:48 d2.data.datasets.coco]: Caching COCO format annotations at '../models/02a08239-8220-48d0-85f2-c0c0425b4224/inference/coco_test_coco_format.json' ... [08/18 03:10:48 d2.evaluation.evaluator]: Start inference on 2 batches [08/18 03:10:49 d2.evaluation.evaluator]: Total inference time: 0:00:00.304959 (0.304959 s / iter per device, on 1 devices) [08/18 03:10:49 d2.evaluation.evaluator]: Total inference pure compute time: 0:00:00 (0.226768 s / iter per device, on 1 devices) [08/18 03:10:49 d2.evaluation.coco_evaluation]: Preparing results for COCO format ... [08/18 03:10:49 d2.evaluation.coco_evaluation]: Saving results to ../models/02a08239-8220-48d0-85f2-c0c0425b4224/inference/coco_instances_results.json [08/18 03:10:49 d2.evaluation.coco_evaluation]: Evaluating predictions with unofficial COCO API... Loading and preparing results...

AssertionError Traceback (most recent call last)

in 1 #cfg.TEST.AUG.ENABLED = False 2 cfg.DATASETS.TEST = ("coco_test", ) ----> 3 model_info = construct_model_info(cfg=cfg) ~/VXGIOQ-CFU-Counting-ML/notebooks/../src/vxgioq_cfu_counting_ml/training/data_analysis.py in construct_model_info(cfg) 79 iteration=cfg.SOLVER.MAX_ITER, 80 available=True, ---> 81 kpis=KPIs(**compute_kpis(cfg)), 82 ) 83 return model ~/VXGIOQ-CFU-Counting-ML/notebooks/../src/vxgioq_cfu_counting_ml/training/data_analysis.py in compute_kpis(cfg) 63 checkpointer.load(cfg_.MODEL.WEIGHTS) 64 ---> 65 res: DatasetEvaluatorResults = CFUTrainer.test(cfg_, model) 66 if cfg.TEST.AUG.ENABLED: 67 res.update(CFUTrainer.test_with_TTA(cfg, model)) /hpc/scratch/hdd1/..../conda_envs/cfu3_env/lib/python3.8/site-packages/detectron2/engine/defaults.py in test(cls, cfg, model, evaluators) 606 results[dataset_name] = {} 607 continue --> 608 results_i = inference_on_dataset(model, data_loader, evaluator) 609 results[dataset_name] = results_i 610 if comm.is_main_process(): /hpc/scratch/hdd1/..../conda_envs/cfu3_env/lib/python3.8/site-packages/detectron2/evaluation/evaluator.py in inference_on_dataset(model, data_loader, evaluator) 202 ) 203 --> 204 results = evaluator.evaluate() 205 # An evaluator may return None when not in main process. 206 # Replace it by an empty dict instead to make it easier for downstream code to handle /hpc/scratch/hdd1/..../conda_envs/cfu3_env/lib/python3.8/site-packages/detectron2/evaluation/evaluator.py in evaluate(self) 91 results = OrderedDict() 92 for evaluator in self._evaluators: ---> 93 result = evaluator.evaluate() 94 if is_main_process() and result is not None: 95 for k, v in result.items(): /hpc/scratch/hdd1/..../conda_envs/cfu3_env/lib/python3.8/site-packages/detectron2/evaluation/coco_evaluation.py in evaluate(self, img_ids) 173 self._eval_box_proposals(predictions) 174 if "instances" in predictions[0]: --> 175 self._eval_predictions(predictions, img_ids=img_ids) 176 # Copy so the caller can do whatever with results 177 return copy.deepcopy(self._results) /hpc/scratch/hdd1/..../conda_envs/cfu3_env/lib/python3.8/site-packages/detectron2/evaluation/coco_evaluation.py in _eval_predictions(self, predictions, img_ids) 233 assert task in {"bbox", "segm", "keypoints"}, f"Got unknown task: {task}!" 234 coco_eval = ( --> 235 _evaluate_predictions_on_coco( 236 self._coco_api, 237 coco_results, /hpc/scratch/hdd1/..../conda_envs/cfu3_env/lib/python3.8/site-packages/detectron2/evaluation/coco_evaluation.py in _evaluate_predictions_on_coco(coco_gt, coco_results, iou_type, kpt_oks_sigmas, use_fast_impl, img_ids) 550 c.pop("bbox", None) 551 --> 552 coco_dt = coco_gt.loadRes(coco_results) 553 coco_eval = (COCOeval_opt if use_fast_impl else COCOeval)(coco_gt, coco_dt, iou_type) 554 if img_ids is not None: /hpc/scratch/hdd1/..../conda_envs/cfu3_env/lib/python3.8/site-packages/pycocotools/coco.py in loadRes(self, resFile) 324 assert type(anns) == list, 'results in not an array of objects' 325 annsImgIds = [ann['image_id'] for ann in anns] --> 326 assert set(annsImgIds) == (set(annsImgIds) & set(self.getImgIds())), \ 327 'Results do not correspond to current coco set' 328 if 'caption' in anns[0]: AssertionError: Results do not correspond to current coco set ``` ## Expected behavior: If there are no obvious crash in "full logs" provided above, please tell us the expected behavior. If you expect a model to converge / work better, we do not help with such issues, unless a model fails to reproduce the results in detectron2 model zoo, or proves existence of bugs. ## Environment: Paste the output of the following command: ``` wget -nc -nv https://github.com/facebookresearch/detectron2/raw/master/detectron2/utils/collect_env.py && python collect_env.py pip install pyyaml==5.1 pip install torch==1.9.0+cu102 torchvision==0.10.0+cu102 -f https://download.pytorch.org/whl/torch_stable.html pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.9/index.html Using python 3.8 wget -nc -q https://github.com/facebookresearch/detectron2/raw/master/detectron2/utils/collect_env.py && python collect_env.py --------------------- ---------------------------------------------------------------------------------------------------- sys.platform linux Python 3.8.11 (default, Aug 3 2021, 15:09:35) [GCC 7.5.0] numpy 1.21.1 detectron2 0.5 @/hpc/scratch/hdd1/..../conda_envs/cfu2_env/lib/python3.8/site-packages/detectron2 Compiler GCC 7.3 CUDA compiler CUDA 10.2 DETECTRON2_ENV_MODULE PyTorch 1.9.0+cu102 @/hpc/scratch/hdd1/..../conda_envs/cfu2_env/lib/python3.8/site-packages/torch PyTorch debug build False GPU available No: torch.cuda.is_available() == False Pillow 8.3.1 torchvision 0.10.0+cu102 @/hpc/scratch/hdd1/..../conda_envs/cfu2_env/lib/python3.8/site-packages/torchvision fvcore 0.1.5.post20210804 iopath 0.1.8 cv2 4.5.3 --------------------- ---------------------------------------------------------------------------------------------------- PyTorch built with: - GCC 7.3 - C++ Version: 201402 - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications - Intel(R) MKL-DNN v2.1.2 (Git Hash 98be7e8afa711dc9b66c8ff3504129cb82013cdb) - OpenMP 201511 (a.k.a. OpenMP 4.5) - NNPACK is enabled - CPU capability usage: AVX2 - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=10.2, CUDNN_VERSION=7.6.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.9.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, ``` [test.zip](https://github.com/facebookresearch/detectron2/files/7008464/test.zip) If your issue looks like an installation issue / environment issue, please first check common issues in https://detectron2.readthedocs.io/tutorials/install.html#common-installation-issues

facebookresearch / detectron2

Getting the error : Results do not correspond to current coco set when running COCOEvaluator #3380

Instructions To Reproduce the Issue: