This is more a request for help other than an issue report, as I see other people successfully getting their results. Sorry if this is not the right place to do it, please tell me if that is so.
I've experienced almost all warnings mentioned in README and in other reported issues, such as
Underflow encountered in ubyte_scalars (mafa2kitti.py:51)
Len of BBox:1 in Image:15... (quite a few of these)
Top left coordinate must be less than bottom right. (tlt-dataset-convert)
distinct prediction count while training ( Matching predictions to ground truth )
But the most important, and something that couldn't find on other issues, is that training mAP is not even close to 88%, instead, i get 71% for mask and 38% for no-mask, even for the unpruned model. I've changed default paths but tried to be consistent across all settings files and command arguments. (I'm confused because all steps are completed and seem to be ok, but for what I read, the result is not ok)
Any idea of what am I doing wrong?
My guess is that datasets are somehow twisted, but I'm sure directory structure is as required (data-tree.txt).
Any comments will help, thank you in advance!
I'll paste some logs below: I can attach spec files but I think they may be ok as it 'almost' works.
data2kitty.py results in :
Kaggle Dataset: Total Mask faces: 4154 and No-Mask faces:790
Total Mask Labelled:4154 and No-Mask Labelled:790
Directory Already Exists
Directory Already Exists
/workspace/apps/face-mask-detection/data_utils/mafa2kitti.py:51: RuntimeWarning: overflow encountered in ubyte_scalars
bbox = [_bbox_label[0], _bbox_label[1], _bbox_label[0]+_bbox_label[2], _bbox_label[1]+_bbox_label[3]]
MAFA Dataset: Total Mask faces: 1846 and No-Mask faces:232
Total Mask Labelled:6000 and No-Mask Labelled:1022
Directory Already Exists
Directory Already Exists
FDDB Dataset: Mask Labelled:0 and No-Mask Labelled:2845
Total Mask Labelled:6000 and No-Mask Labelled:3867
Directory Already Exists
Directory Already Exists
Len of BBox:3 in Image:15_Stock_Market_Stock_Market_15_292
Len of BBox:1 in Image:15_Stock_Market_Stock_Market_15_563
...
(Quite a lot of these)
...
WideFace: Total Mask Labelled:0 and No-Mask Labelled:2133
----------------------------
Final: Total Mask Labelled:6000
Total No-Mask Labelled:6000
----------------------------
tls-dataset-convert :
tlt-dataset-convert -d $SPECS_DIR/detectnet_v2_tfrecords_kitti_trainval.txt \
> -o $DATA_DOWNLOAD_DIR/tfrecords/kitti_trainval/kitti_trainval
2020-12-28 14:19:06.377614: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
Using TensorFlow backend.
2020-12-28 14:19:14,916 - iva.detectnet_v2.dataio.build_converter - INFO - Instantiating a kitti converter
2020-12-28 14:19:14,930 - iva.detectnet_v2.dataio.kitti_converter_lib - INFO - Num images in
Train: 3899 Val: 974
2020-12-28 14:19:14,930 - iva.detectnet_v2.dataio.kitti_converter_lib - INFO - Validation data in partition 0. Hence, while choosing the validationset during training choose validation_fold 0.
2020-12-28 14:19:14,933 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 0
WARNING:tensorflow:From /home/vpraveen/.cache/dazel/_dazel_vpraveen/715c8bafe7816f3bb6f309cd506049bb/execroot/ai_infra/bazel-out/k8-py3-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py:142: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.
2020-12-28 14:19:14,934 - tensorflow - WARNING - From /home/vpraveen/.cache/dazel/_dazel_vpraveen/715c8bafe7816f3bb6f309cd506049bb/execroot/ai_infra/bazel-out/k8-py3-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py:142: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use None for the system default.
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/165_mask.txt.
Coordinates: x1 = 360, x2 = 249, y1: 194, y2: 325
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /workspace/apps/face-mask-detection-output/train/labels/165_mask.txt.
Coordinates: x1 = 714, x2 = 584, y1: 108, y2: 257
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/83_mask.txt.
Coordinates: x1 = 813, x2 = 675, y1: 160, y2: 276
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /workspace/apps/face-mask-detection-output/train/labels/83_mask.txt.
Coordinates: x1 = 514, x2 = 459, y1: 191, y2: 253
Skipping this object
Top left coordinate must be less than bottom right.Error in object 2 of label_file /workspace/apps/face-mask-detection-output/train/labels/83_mask.txt.
Coordinates: x1 = 270, x2 = 141, y1: 183, y2: 292
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/396_mask.txt.
Coordinates: x1 = 635, x2 = 330, y1: 185, y2: 508
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/85_mask.txt.
Coordinates: x1 = 862, x2 = 645, y1: 82, y2: 292
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/230_mask.txt.
Coordinates: x1 = 357, x2 = 175, y1: 127, y2: 342
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/61_mask.txt.
Coordinates: x1 = 633, x2 = 441, y1: 183, y2: 338
Skipping this object
2020-12-28 14:19:15,091 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 1
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/493_mask.txt.
Coordinates: x1 = 725, x2 = 385, y1: 71, y2: 371
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/123_mask.txt.
Coordinates: x1 = 749, x2 = 516, y1: 169, y2: 411
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/109_mask.txt.
Coordinates: x1 = 818, x2 = 586, y1: 139, y2: 413
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/445_mask.txt.
Coordinates: x1 = 741, x2 = 451, y1: 103, y2: 392
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/136_mask.txt.
Coordinates: x1 = 751, x2 = 537, y1: 77, y2: 307
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/346_mask.txt.
Coordinates: x1 = 573, x2 = 343, y1: 117, y2: 447
Skipping this object
2020-12-28 14:19:15,198 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 2
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/142_mask.txt.
Coordinates: x1 = 586, x2 = 468, y1: 228, y2: 353
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /workspace/apps/face-mask-detection-output/train/labels/142_mask.txt.
Coordinates: x1 = 759, x2 = 655, y1: 134, y2: 255
Skipping this object
Top left coordinate must be less than bottom right.Error in object 2 of label_file /workspace/apps/face-mask-detection-output/train/labels/142_mask.txt.
Coordinates: x1 = 237, x2 = 127, y1: 282, y2: 416
Skipping this object
Top left coordinate must be less than bottom right.Error in object 3 of label_file /workspace/apps/face-mask-detection-output/train/labels/142_mask.txt.
Coordinates: x1 = 419, x2 = 323, y1: 119, y2: 239
Skipping this object
Top left coordinate must be less than bottom right.Error in object 4 of label_file /workspace/apps/face-mask-detection-output/train/labels/142_mask.txt.
Coordinates: x1 = 154, x2 = 51, y1: 157, y2: 271
Skipping this object
Top left coordinate must be less than bottom right.Error in object 5 of label_file /workspace/apps/face-mask-detection-output/train/labels/142_mask.txt.
Coordinates: x1 = 929, x2 = 827, y1: 278, y2: 399
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/66_mask.txt.
Coordinates: x1 = 465, x2 = 236, y1: 20, y2: 304
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/58_mask.txt.
Coordinates: x1 = 509, x2 = 443, y1: 136, y2: 205
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /workspace/apps/face-mask-detection-output/train/labels/58_mask.txt.
Coordinates: x1 = 247, x2 = 169, y1: 45, y2: 123
Skipping this object
Top left coordinate must be less than bottom right.Error in object 2 of label_file /workspace/apps/face-mask-detection-output/train/labels/58_mask.txt.
Coordinates: x1 = 814, x2 = 735, y1: 82, y2: 170
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/55_mask.txt.
Coordinates: x1 = 363, x2 = 348, y1: 68, y2: 82
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /workspace/apps/face-mask-detection-output/train/labels/55_mask.txt.
Coordinates: x1 = 566, x2 = 530, y1: 133, y2: 176
Skipping this object
Top left coordinate must be less than bottom right.Error in object 2 of label_file /workspace/apps/face-mask-detection-output/train/labels/55_mask.txt.
Coordinates: x1 = 462, x2 = 420, y1: 127, y2: 172
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/331_mask.txt.
Coordinates: x1 = 833, x2 = 545, y1: 162, y2: 472
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/141_mask.txt.
Coordinates: x1 = 565, x2 = 378, y1: 102, y2: 321
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/56_mask.txt.
Coordinates: x1 = 357, x2 = 306, y1: 241, y2: 295
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /workspace/apps/face-mask-detection-output/train/labels/56_mask.txt.
Coordinates: x1 = 162, x2 = 100, y1: 261, y2: 310
Skipping this object
Top left coordinate must be less than bottom right.Error in object 2 of label_file /workspace/apps/face-mask-detection-output/train/labels/56_mask.txt.
Coordinates: x1 = 487, x2 = 412, y1: 178, y2: 259
Skipping this object
Top left coordinate must be less than bottom right.Error in object 3 of label_file /workspace/apps/face-mask-detection-output/train/labels/56_mask.txt.
Coordinates: x1 = 97, x2 = 69, y1: 217, y2: 244
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/154_mask.txt.
Coordinates: x1 = 766, x2 = 573, y1: 84, y2: 323
Skipping this object
2020-12-28 14:19:15,310 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 3
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/93_mask.txt.
Coordinates: x1 = 534, x2 = 464, y1: 129, y2: 194
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /workspace/apps/face-mask-detection-output/train/labels/93_mask.txt.
Coordinates: x1 = 654, x2 = 612, y1: 199, y2: 255
Skipping this object
Top left coordinate must be less than bottom right.Error in object 2 of label_file /workspace/apps/face-mask-detection-output/train/labels/93_mask.txt.
Coordinates: x1 = 332, x2 = 264, y1: 158, y2: 235
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/57_mask.txt.
Coordinates: x1 = 604, x2 = 499, y1: 81, y2: 214
Skipping this object
2020-12-28 14:19:15,424 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 4
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/403_mask.txt.
Coordinates: x1 = 667, x2 = 506, y1: 181, y2: 341
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/76_mask.txt.
Coordinates: x1 = 543, x2 = 425, y1: 170, y2: 321
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /workspace/apps/face-mask-detection-output/train/labels/76_mask.txt.
Coordinates: x1 = 364, x2 = 202, y1: 95, y2: 250
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/8_mask.txt.
Coordinates: x1 = 743, x2 = 685, y1: 192, y2: 260
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /workspace/apps/face-mask-detection-output/train/labels/8_mask.txt.
Coordinates: x1 = 194, x2 = 128, y1: 124, y2: 192
Skipping this object
Top left coordinate must be less than bottom right.Error in object 2 of label_file /workspace/apps/face-mask-detection-output/train/labels/8_mask.txt.
Coordinates: x1 = 575, x2 = 511, y1: 161, y2: 225
Skipping this object
Top left coordinate must be less than bottom right.Error in object 3 of label_file /workspace/apps/face-mask-detection-output/train/labels/8_mask.txt.
Coordinates: x1 = 435, x2 = 380, y1: 125, y2: 177
Skipping this object
Top left coordinate must be less than bottom right.Error in object 4 of label_file /workspace/apps/face-mask-detection-output/train/labels/8_mask.txt.
Coordinates: x1 = 888, x2 = 830, y1: 154, y2: 218
Skipping this object
Top left coordinate must be less than bottom right.Error in object 5 of label_file /workspace/apps/face-mask-detection-output/train/labels/8_mask.txt.
Coordinates: x1 = 223, x2 = 197, y1: 165, y2: 198
Skipping this object
Top left coordinate must be less than bottom right.Error in object 6 of label_file /workspace/apps/face-mask-detection-output/train/labels/8_mask.txt.
Coordinates: x1 = 355, x2 = 289, y1: 124, y2: 182
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/72_mask.txt.
Coordinates: x1 = 152, x2 = 98, y1: 133, y2: 192
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /workspace/apps/face-mask-detection-output/train/labels/72_mask.txt.
Coordinates: x1 = 530, x2 = 468, y1: 131, y2: 205
Skipping this object
Top left coordinate must be less than bottom right.Error in object 2 of label_file /workspace/apps/face-mask-detection-output/train/labels/72_mask.txt.
Coordinates: x1 = 451, x2 = 408, y1: 145, y2: 200
Skipping this object
Top left coordinate must be less than bottom right.Error in object 3 of label_file /workspace/apps/face-mask-detection-output/train/labels/72_mask.txt.
Coordinates: x1 = 278, x2 = 224, y1: 170, y2: 231
Skipping this object
Top left coordinate must be less than bottom right.Error in object 4 of label_file /workspace/apps/face-mask-detection-output/train/labels/72_mask.txt.
Coordinates: x1 = 793, x2 = 721, y1: 129, y2: 214
Skipping this object
2020-12-28 14:19:15,539 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 5
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/68_mask.txt.
Coordinates: x1 = 848, x2 = 783, y1: 78, y2: 155
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /workspace/apps/face-mask-detection-output/train/labels/68_mask.txt.
Coordinates: x1 = 243, x2 = 159, y1: 85, y2: 178
Skipping this object
Top left coordinate must be less than bottom right.Error in object 2 of label_file /workspace/apps/face-mask-detection-output/train/labels/68_mask.txt.
Coordinates: x1 = 636, x2 = 560, y1: 38, y2: 133
Skipping this object
Top left coordinate must be less than bottom right.Error in object 3 of label_file /workspace/apps/face-mask-detection-output/train/labels/68_mask.txt.
Coordinates: x1 = 527, x2 = 462, y1: 93, y2: 162
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/420_mask.txt.
Coordinates: x1 = 639, x2 = 318, y1: 232, y2: 506
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/140_mask.txt.
Coordinates: x1 = 683, x2 = 508, y1: 138, y2: 377
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/325_mask.txt.
Coordinates: x1 = 729, x2 = 215, y1: 168, y2: 444
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/227_mask.txt.
Coordinates: x1 = 755, x2 = 535, y1: 139, y2: 408
Skipping this object
2020-12-28 14:19:15,658 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 6
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/114_mask.txt.
Coordinates: x1 = 739, x2 = 478, y1: 90, y2: 388
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/63_mask.txt.
Coordinates: x1 = 669, x2 = 440, y1: 131, y2: 354
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/418_mask.txt.
Coordinates: x1 = 726, x2 = 343, y1: 114, y2: 329
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/113_mask.txt.
Coordinates: x1 = 535, x2 = 384, y1: 164, y2: 355
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/335_mask.txt.
Coordinates: x1 = 716, x2 = 316, y1: 100, y2: 279
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/428_mask.txt.
Coordinates: x1 = 583, x2 = 185, y1: 128, y2: 514
Skipping this object
2020-12-28 14:19:15,773 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 7
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/249_mask.txt.
Coordinates: x1 = 697, x2 = 342, y1: 66, y2: 233
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/469_mask.txt.
Coordinates: x1 = 732, x2 = 226, y1: 116, y2: 504
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/7_mask.txt.
Coordinates: x1 = 773, x2 = 482, y1: 85, y2: 386
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/73_mask.txt.
Coordinates: x1 = 858, x2 = 640, y1: 136, y2: 354
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /workspace/apps/face-mask-detection-output/train/labels/73_mask.txt.
Coordinates: x1 = 418, x2 = 173, y1: 162, y2: 359
Skipping this object
2020-12-28 14:19:15,885 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 8
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/117_mask.txt.
Coordinates: x1 = 640, x2 = 435, y1: 108, y2: 341
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/350_mask.txt.
Coordinates: x1 = 732, x2 = 525, y1: 83, y2: 264
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/59_mask.txt.
Coordinates: x1 = 664, x2 = 532, y1: 167, y2: 328
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/74_mask.txt.
Coordinates: x1 = 574, x2 = 326, y1: 118, y2: 374
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/25_mask.txt.
Coordinates: x1 = 424, x2 = 218, y1: 219, y2: 420
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /workspace/apps/face-mask-detection-output/train/labels/25_mask.txt.
Coordinates: x1 = 778, x2 = 572, y1: 141, y2: 364
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/226_mask.txt.
Coordinates: x1 = 907, x2 = 422, y1: 139, y2: 544
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/103_mask.txt.
Coordinates: x1 = 423, x2 = 216, y1: 122, y2: 324
Skipping this object
2020-12-28 14:19:15,995 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 9
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/82_mask.txt.
Coordinates: x1 = 884, x2 = 828, y1: 229, y2: 302
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /workspace/apps/face-mask-detection-output/train/labels/82_mask.txt.
Coordinates: x1 = 466, x2 = 288, y1: 113, y2: 359
Skipping this object
Top left coordinate must be less than bottom right.Error in object 2 of label_file /workspace/apps/face-mask-detection-output/train/labels/82_mask.txt.
Coordinates: x1 = 676, x2 = 578, y1: 227, y2: 361
Skipping this object
Top left coordinate must be less than bottom right.Error in object 3 of label_file /workspace/apps/face-mask-detection-output/train/labels/82_mask.txt.
Coordinates: x1 = 574, x2 = 530, y1: 229, y2: 300
Skipping this object
Top left coordinate must be less than bottom right.Error in object 4 of label_file /workspace/apps/face-mask-detection-output/train/labels/82_mask.txt.
Coordinates: x1 = 90, x2 = 62, y1: 216, y2: 252
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/208_mask.txt.
Coordinates: x1 = 655, x2 = 119, y1: 71, y2: 544
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/5_mask.txt.
Coordinates: x1 = 408, x2 = 283, y1: 151, y2: 311
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /workspace/apps/face-mask-detection-output/train/labels/5_mask.txt.
Coordinates: x1 = 742, x2 = 667, y1: 194, y2: 271
Skipping this object
Top left coordinate must be less than bottom right.Error in object 2 of label_file /workspace/apps/face-mask-detection-output/train/labels/5_mask.txt.
Coordinates: x1 = 899, x2 = 791, y1: 273, y2: 389
Skipping this object
Top left coordinate must be less than bottom right.Error in object 3 of label_file /workspace/apps/face-mask-detection-output/train/labels/5_mask.txt.
Coordinates: x1 = 770, x2 = 704, y1: 112, y2: 180
Skipping this object
2020-12-28 14:19:16,110 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO -
Wrote the following numbers of objects:
b'mask': 3423
b'no-mask': 674
2020-12-28 14:19:16,110 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 0
2020-12-28 14:19:16,517 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 1
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/train_00005401.txt.
Coordinates: x1 = 340, x2 = 72, y1: 175, y2: 360
Skipping this object
2020-12-28 14:19:16,933 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 2
2020-12-28 14:19:17,349 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 3
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/train_00000572.txt.
Coordinates: x1 = 639, x2 = 57, y1: 117, y2: 307
Skipping this object
2020-12-28 14:19:17,765 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 4
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/train_00000526.txt.
Coordinates: x1 = 560, x2 = 31, y1: 161, y2: 397
Skipping this object
2020-12-28 14:19:18,180 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 5
2020-12-28 14:19:18,599 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 6
2020-12-28 14:19:19,018 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 7
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/train_00000304.txt.
Coordinates: x1 = 434, x2 = 60, y1: 96, y2: 464
Skipping this object
2020-12-28 14:19:19,426 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 8
2020-12-28 14:19:19,840 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 9
Top left coordinate must be less than bottom right.Error in object 0 of label_file /workspace/apps/face-mask-detection-output/train/labels/train_00000650.txt.
Coordinates: x1 = 412, x2 = 50, y1: 133, y2: 509
Skipping this object
2020-12-28 14:19:20,261 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO -
Wrote the following numbers of objects:
b'no-mask': 5210
b'mask': 1532
2020-12-28 14:19:20,262 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Cumulative object statistics
2020-12-28 14:19:20,262 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO -
Wrote the following numbers of objects:
b'mask': 4955
b'no-mask': 5884
2020-12-28 14:19:20,262 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Class map.
Label in GT: Label in tfrecords file
b'Mask': b'mask'
b'No-Mask': b'no-mask'
For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.
2020-12-28 14:19:20,262 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Tfrecords generation complete.
Finally, first and last output of mAP
tlt-train detectnet_v2 -e $SPECS_DIR/detectnet_v2_train_resnet18_kitti.txt \
-r $USER_EXPERIMENT_DIR/experiment_dir_unpruned \
-k $KEY \
-n resnet18_detector \
--gpus $NUM_GPUS
Matching predictions to ground truth, class 1/2.: 100%|█████████████████████████████████████████████████████████| 760765/760765 [00:56<00:00, 13584.99it/s]
Matching predictions to ground truth, class 2/2.: 100%|███████████████████████████████████████████████████████████| 67015/67015 [00:02<00:00, 26594.77it/s]
Epoch 10/120
=========================
Validation cost: 0.000500
Mean average_precision (in %): 0.7594
class name average precision (in %)
------------ --------------------------
mask 0.917293
no-mask 0.601492
Median Inference Time: 0.005945
Matching predictions to ground truth, class 1/2.: 100%|█████████████████████████████████████████████████████████████| 3692/3692 [00:00<00:00, 12346.46it/s]
Matching predictions to ground truth, class 2/2.: 100%|█████████████████████████████████████████████████████████████| 1392/1392 [00:00<00:00, 17775.72it/s]
Epoch 120/120
=========================
Validation cost: 0.000127
Mean average_precision (in %): 55.0598
class name average precision (in %)
------------ --------------------------
mask 71.1593
no-mask 38.9604
Median Inference Time: 0.005945
2020-12-27 01:41:39,730 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 72.149
Time taken to run iva.detectnet_v2.scripts.train:main: 2:02:09.356023.
Hi all!
This is more a request for help other than an issue report, as I see other people successfully getting their results. Sorry if this is not the right place to do it, please tell me if that is so.
I've experienced almost all warnings mentioned in README and in other reported issues, such as
But the most important, and something that couldn't find on other issues, is that training mAP is not even close to 88%, instead, i get 71% for mask and 38% for no-mask, even for the unpruned model. I've changed default paths but tried to be consistent across all settings files and command arguments. (I'm confused because all steps are completed and seem to be ok, but for what I read, the result is not ok)
Any idea of what am I doing wrong? My guess is that datasets are somehow twisted, but I'm sure directory structure is as required (data-tree.txt).
Any comments will help, thank you in advance!
I'll paste some logs below: I can attach spec files but I think they may be ok as it 'almost' works.
data2kitty.py results in :
tls-dataset-convert :
Finally, first and last output of mAP