Unable to run model Evaluation with COCOEvaluator

[HOD101s]

Unable to run model Evaluation with COCOEvaluator

Instructions To Reproduce the Issue:

1. Registered my Datasets and generated Metadata

   
   DatasetCatalog.register("lplates_train", lambda train_df=train_df: preDtron(train_df, classes))
   MetadataCatalog.get("lplates_train").set(thing_classes=['NumPlate'])

DatasetCatalog.register("lplates_test", lambda test_df=test_df: preDtron(test_df, classes)) MetadataCatalog.get("lplates_test").set(thing_classes=['NumPlate'])

lplate_metadata = MetadataCatalog.get("lplates_train") test_metadata = MetadataCatalog.get("lplates_test")

here preDtron is my preprocessing function:

def preDtron(df,classes):

dataset_dicts = []
for image_id, img_name in tqdm(enumerate(df.file_name.unique())):
    info = {}
    data = df[df.file_name == img_name]

    info["file_name"] = img_name
    info["image_id"] = image_id
    info["height"] = data.iloc[0].height
    info["width"] = data.iloc[0].width

    objs = []

    for _,row in data.iterrows():

        x_min = row["x_min"] 
        y_min = row["y_min"]
        x_max = row["x_max"]
        y_max = row["y_max"]

        segs = [
            (x_min, y_min), (x_max, y_min),
            (x_max, y_max), (x_min, y_max)
        ]

        obj = {
            "bbox": [x_min, y_min, x_max, y_max],
            "bbox_mode": BoxMode.XYXY_ABS,
            "segmentation": [list(itertools.chain.from_iterable(segs))],
            "category_id": 0
        }

        objs.append(obj)

    info["annotations"] = objs
    dataset_dicts.append(info)
return dataset_dicts

I am aware `classes` is an unused argument was used in previous versions.

Trained My Model Sucessfully and was able to make inferences on test data from it as well

cfg = get_cfg() cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")) cfg.DATASETS.TRAIN = ("lplates_train",) cfg.DATASETS.TEST = ("lplates_test",) cfg.DATALOADER.NUM_WORKERS = 4 cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml") # Let training initialize from model zoo cfg.SOLVER.IMS_PER_BATCH = 4 cfg.SOLVER.BASE_LR = 0.01
cfg.SOLVER.MAX_ITER = 600
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 512 cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1

os.makedirs(cfg.OUTPUT_DIR, exist_ok=True) trainer = DefaultTrainer(cfg) trainer.resume_or_load(resume=False) trainer.train()

2. Began Evaluation. Now here is where the problem starts

evaluator = COCOEvaluator("lplates_test", cfg, False, output_dir="./output2/") val_loader = build_detection_test_loader(cfg, "lplates_test") print(inference_on_dataset(trainer.model, val_loader, evaluator))

this gives me 
`JSONDecodeError: Expecting value: line 1 column 161 (char 160)`
on `test_metadata.json_file`
Also the json_file is initialised after running the eval code block.

So I went ahead to view this generated metadata jsonfile to see that it has not been entirely generated. This is the file:

lplates_test_coco_format.json

{"info": {"date_created": "2020-09-09 10:17:29.866472", "description": "Automatically generated COCO json file for Detectron2."}, "images": [{"id": 0, "width":

3. Error Logs

---

JSONDecodeError Traceback (most recent call last)

in () ----> 1 evaluator = COCOEvaluator("lplates_test", cfg, False, output_dir="./output1/") 2 val_loader = build_detection_test_loader(cfg, "lplates_test") 3 print(inference_on_dataset(trainer.model, val_loader, evaluator)) 5 frames /usr/local/lib/python3.6/dist-packages/detectron2/evaluation/coco_evaluation.py in __init__(self, dataset_name, cfg, distributed, output_dir) 79 json_file = PathManager.get_local_path(self._metadata.json_file) 80 with contextlib.redirect_stdout(io.StringIO()): ---> 81 self._coco_api = COCO(json_file) 82 83 self._kpt_oks_sigmas = cfg.TEST.KEYPOINT_OKS_SIGMAS /usr/local/lib/python3.6/dist-packages/pycocotools/coco.py in __init__(self, annotation_file) 82 print('loading annotations into memory...') 83 tic = time.time() ---> 84 dataset = json.load(open(annotation_file, 'r')) 85 assert type(dataset)==dict, 'annotation file format {} not supported'.format(type(dataset)) 86 print('Done (t={:0.2f}s)'.format(time.time()- tic)) /usr/lib/python3.6/json/__init__.py in load(fp, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw) 297 cls=cls, object_hook=object_hook, 298 parse_float=parse_float, parse_int=parse_int, --> 299 parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw) 300 301 /usr/lib/python3.6/json/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw) 352 parse_int is None and parse_float is None and 353 parse_constant is None and object_pairs_hook is None and not kw): --> 354 return _default_decoder.decode(s) 355 if cls is None: 356 cls = JSONDecoder /usr/lib/python3.6/json/decoder.py in decode(self, s, _w) 337 338 """ --> 339 obj, end = self.raw_decode(s, idx=_w(s, 0).end()) 340 end = _w(s, end).end() 341 if end != len(s): /usr/lib/python3.6/json/decoder.py in raw_decode(self, s, idx) 355 obj, end = self.scan_once(s, idx) 356 except StopIteration as err: --> 357 raise JSONDecodeError("Expecting value", s, err.value) from None 358 return obj, end JSONDecodeError: Expecting value: line 1 column 161 (char 160) ``` Colab Logs: Tried deleting the generated json. On running again system gives FileNotFound Error. Timestamp | Level | Message Sep 13, 2020, 2:50:25 PM | WARNING | delete /content/output1 Sep 13, 2020, 2:50:21 PM | WARNING | delete /content/output1/lplates_test_coco_format.json.lock Sep 13, 2020, 2:50:18 PM | WARNING | delete /content/output1/lplates_test_coco_format.json Sep 13, 2020, 2:50:15 PM | WARNING | 400 DELETE /api/contents/content/output1 (172.28.0.3) 3.67ms referer=https://colab.research.google.com/ Sep 13, 2020, 2:50:15 PM | WARNING | Directory /content/output1 not empty Sep 13, 2020, 2:50:15 PM | WARNING | 400 DELETE /api/contents/content/output1 (172.28.0.3): Directory /content/output1 not empty Sep 13, 2020, 2:50:15 PM | WARNING | delete /content/output1 Sep 13, 2020, 2:47:56 PM | INFO | Kernel restarted: e4dd008a-08b5-4358-a1f8-906ade73421a ## Expected behavior: Begin Evaluation of model against test DataSet `lplates_test`. I'm guessing theres some issue with how i'm initiliasing the Datasets. Any Help would be great. Thanks. ## Environment: ---------------------- --------------------------------------------------------------- sys.platform linux Python 3.6.9 (default, Jul 17 2020, 12:50:27) [GCC 8.4.0] numpy 1.18.5 detectron2 0.2.1 @/usr/local/lib/python3.6/dist-packages/detectron2 Compiler GCC 7.3 CUDA compiler CUDA 10.1 detectron2 arch flags sm_35, sm_37, sm_50, sm_52, sm_60, sm_61, sm_70, sm_75 DETECTRON2_ENV_MODULE PyTorch 1.6.0+cu101 @/usr/local/lib/python3.6/dist-packages/torch PyTorch debug build False GPU available True GPU 0 Tesla P100-PCIE-16GB CUDA_HOME /usr/local/cuda Pillow 7.2.0 torchvision 0.7.0+cu101 @/usr/local/lib/python3.6/dist-packages/torchvision torchvision arch flags sm_35, sm_50, sm_60, sm_70, sm_75 fvcore 0.1.2.post20200912 cv2 4.1.2 ---------------------- --------------------------------------------------------------- PyTorch built with: - GCC 7.3 - C++ Version: 201402 - Intel(R) Math Kernel Library Version 2019.0.5 Product Build 20190808 for Intel(R) 64 architecture applications - Intel(R) MKL-DNN v1.5.0 (Git Hash e2ac1fac44c5078ca927cb9b90e1b3066a0b2ed0) - OpenMP 201511 (a.k.a. OpenMP 4.5) - NNPACK is enabled - CPU capability usage: AVX2 - CUDA Runtime 10.1 - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75 - CuDNN 7.6.3 - Magma 2.5.2 - Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,

[ppwwyyxx]

If you need help to solve an unexpected issue you observed, please include details following the "Unexpected behaviors" issue template.

[HOD101s]

If you need help to solve an unexpected issue you observed, please include details following the "Unexpected behaviors" issue template.

Followed the same. Just Renamed the issue. Any Help would be great!

[ppwwyyxx]

Please provide full logs following the issue template

[HOD101s]

Please provide full logs following the issue template

Updated issue with logs

[ppwwyyxx]

By "full logs" I'm expecting to see logs from the beginning of the program, not just the error.

Anyway, can you try again after removing the generated json file?

[HOD101s]

On deleting the file and running again it gives file not found error

FileNotFoundError                         Traceback (most recent call last)
<ipython-input-16-10943452416d> in <module>()
----> 1 evaluator = COCOEvaluator("lplates_test", cfg, False, output_dir="./output2/")
      2 val_loader = build_detection_test_loader(cfg, "lplates_test")
      3 print(inference_on_dataset(trainer.model, val_loader, evaluator))

1 frames
/usr/local/lib/python3.6/dist-packages/detectron2/evaluation/coco_evaluation.py in __init__(self, dataset_name, cfg, distributed, output_dir)
     79         json_file = PathManager.get_local_path(self._metadata.json_file)
     80         with contextlib.redirect_stdout(io.StringIO()):
---> 81             self._coco_api = COCO(json_file)
     82 
     83         self._kpt_oks_sigmas = cfg.TEST.KEYPOINT_OKS_SIGMAS

/usr/local/lib/python3.6/dist-packages/pycocotools/coco.py in __init__(self, annotation_file)
     82             print('loading annotations into memory...')
     83             tic = time.time()
---> 84             dataset = json.load(open(annotation_file, 'r'))
     85             assert type(dataset)==dict, 'annotation file format {} not supported'.format(type(dataset))
     86             print('Done (t={:0.2f}s)'.format(time.time()- tic))

FileNotFoundError: [Errno 2] No such file or directory: './output2/lplates_test_coco_format.json'

Updated issue with Colab Logs

[ppwwyyxx]

By "logs" I mean all logs like this as you can find in the official colab tutorial. These are still missing. The tutorial has instructions how to setup the logger if you couldn't see the logs.

On deleting the file and running again it gives file not found error

Did you restart Colab runtime? Please always restart before you try something, otherwise the behavior of the code is affected by everything you did in the past.

[HOD101s]

Those logs aren't generated. I get the error before any of those logs. And yes I restart the runtime before running.

[ppwwyyxx]

The tutorial has instructions how to setup the logger if you couldn't see the logs.

[HOD101s]

The generatedfile stopped generation at "width" :

{"info": {"date_created": "2020-09-09 10:17:29.866472", "description": "Automatically generated COCO json file for Detectron2."}, "images": [{"id": 0, "width":

So I figured he issue would be with the data itself where found that width was stored as a numpy object which was creating the issue. Once I updated my preprocessing function to :

info["height"] = int(data.iloc[0].height)
info["width"] = int(data.iloc[0].width)

Now it works. Thank you.