data2kitti.py No such file or directory Linux (Ubuntu 18.04)

[hectormdom]

Hi I'm trying to convert the data set to KITTI format, and I figured this command would be correct:

python3 data2kitti.py --kaggle-dataset-path face-mask-detection-data/Kaggle_Medical_Mask_Dataset \
                         --mafa-dataset-path face-mask-detection-data/MAFA_Dataset \
                         --fddb-dataset-path face-mask-detection-data/FDDB_Dataset \
                         --widerface-dataset-path cd face-mask-detection-data/Wider_Face_Dataset \
                         --kitti-base-path face-mask-detection-data/KITTI_gitcheck \
                         --train

I modified the paths in the data2kitti.py file like so:

    # Datasets for Masked Faces
    kaggle_base_dir = r'face-mask-detection-data/Kaggle_Medical_Mask_Dataset' #args.kaggle_dataset_path # Use all data from Kaggle
    mafa_base_dir = r'face-mask-detection-data/MAFA_Dataset' #args.mafa_dataset_path # Use only about 4000 images from MAFA
    # Datasets for No-Masked Faces
    fddb_base_dir = r'face-mask-detection-data/FDDB_Dataset' #args.fddb_dataset_path # Use all data from FDDB
    widerface_base_dir = r'face-mask-detection-data/Wider_Face_Dataset' #args.widerface_dataset_path # Use only from selected sub-folders
    ''' Note: Kaggle, FDDB Data sets does not have validation data thus we use all data for training '''
    # Store Converted annotations in KITTI format
    kitti_base_dir = r'face-mask-detection-data/KITTI_gitcheck' #args.kitti_base_path

However when I try to run the command I get this output:

Directory Already Exists
Directory Already Exists
Kaggle Dataset: Total Mask faces: 0 and No-Mask faces:0
Total Mask Labelled:0 and No-Mask Labelled:0
Directory Already Exists
Directory Already Exists
Traceback (most recent call last):
  File "data2kitti.py", line 107, in <module>
    main()
  File "data2kitti.py", line 61, in main
    count_masks, count_no_masks = kitti_label.mat2data()
  File "/workspace/tlt-ds-face_mask_detect/data_utils/mafa2kitti.py", line 134, in mat2data
    _count_no_mask=_count_no_mask)
  File "/workspace/tlt-ds-face_mask_detect/data_utils/mafa2kitti.py", line 65, in extract_labels
    if not self.check_image_dims(image_name=train_image_name):
  File "/workspace/tlt-ds-face_mask_detect/data_utils/mafa2kitti.py", line 98, in check_image_dims
    img = Image.open(os.path.join(self.mafa_base_dir, image_name)).convert("RGB")
  File "/usr/local/lib/python3.5/dist-packages/PIL/Image.py", line 2878, in open
    fp = builtins.open(filename, "rb")
FileNotFoundError: [Errno 2] No such file or directory: 'face-mask-detection-data/MAFA_Dataset/train-images\\images/train_00000001.jpg'

I supposed it had something to do with that double backlash, I changed the lines 47 mafa_base_dir = os.path.join(mafa_base_dir, 'train-images\images') to mafa_base_dir = os.path.join(mafa_base_dir, 'train-images/images') and 50 mafa_base_dir = os.path.join(mafa_base_dir, 'test-images\images') to mafa_base_dir = os.path.join(mafa_base_dir, 'test-images/images') in data2kitti.py

I managed to run it, this is my output

Directory Already Exists
Directory Already Exists
Kaggle Dataset: Total Mask faces: 0 and No-Mask faces:0
Total Mask Labelled:0 and No-Mask Labelled:0
Directory Already Exists
Directory Already Exists
/workspace/tlt-ds-face_mask_detect/data_utils/mafa2kitti.py:51: RuntimeWarning: overflow encountered in ubyte_scalars
  bbox = [_bbox_label[0], _bbox_label[1], _bbox_label[0]+_bbox_label[2], _bbox_label[1]+_bbox_label[3]]
MAFA Dataset: Total Mask faces: 6000 and No-Mask faces:232
Total Mask Labelled:6000 and No-Mask Labelled:232
Directory Already Exists
Directory Already Exists
FDDB Dataset: Mask Labelled:0 and No-Mask Labelled:0
Total Mask Labelled:6000 and No-Mask Labelled:232
WideFace: Total Mask Labelled:0 and No-Mask Labelled:6033
----------------------------
Final: Total Mask Labelled:6000
Total No-Mask Labelled:6265
----------------------------

[hectormdom]

Also if you will please, let me know if this warning is normal

/workspace/tlt-ds-face_mask_detect/data_utils/mafa2kitti.py:51: RuntimeWarning: overflow encountered in ubyte_scalars
  bbox = [_bbox_label[0], _bbox_label[1], _bbox_label[0]+_bbox_label[2], _bbox_label[1]+_bbox_label[3]]

And if not, if you have any leads as to how should I go solving it.

Cheers!

[ak-nv]

data2kitti.py

Thank you for working on this. In arguments; you have to have a absolute path to the directories. Thanks for pointing this out, I will update the readme according to this.

[ak-nv]

Also if you will please, let me know if this warning is normal

/workspace/tlt-ds-face_mask_detect/data_utils/mafa2kitti.py:51: RuntimeWarning: overflow encountered in ubyte_scalars
  bbox = [_bbox_label[0], _bbox_label[1], _bbox_label[0]+_bbox_label[2], _bbox_label[1]+_bbox_label[3]]

And if not, if you have any leads as to how should I go solving it.

Cheers!

This warning is okay, it mainly because [one of the functions] (https://github.com/NVIDIA-AI-IOT/face-mask-detection/blob/master/data_utils/mafa2kitti.py#L122) currently does not support bounding box scaling for some of the image sizes. You can proceed to next steps, without worrying too much about this warning,

[hectormdom]

Thanks to you, also is my output correct (besides the warning)? it seems to me like I'm not getting images from anywhere except MAFA and WideFace.

[hectormdom]

I just found out that, even though the output says this:

Final: Total Mask Labelled:6000
Total No-Mask Labelled:6265

The folder actually contains only 3182 images

[ak-nv]

Thanks to you, also is my output correct (besides the warning)? it seems to me like I'm not getting images from anywhere except MAFA and WideFace.

Thank you for diligently going through the repo. Let me check on my end, if any issue with generating FDDB dataset in Kitti format. Once you get FDDB dataset completely, you should have good accuracy.

Also, I am assuming, for accuracy evaluations, what batch size are you using?

[ak-nv]

I just found out that, even though the output says this:

Final: Total Mask Labelled:6000
Total No-Mask Labelled:6265

The folder actually contains only 3182 images

This is okay, as most of the images have multiple faces. Especially WiderFace dataset, has more number of people in a single image.

[hectormdom]

Hi, thanks for the reply, I intend to go through the whole thing and report back, I find it very interesting thanks for authoring this.

Thanks to you, also is my output correct (besides the warning)? it seems to me like I'm not getting images from anywhere except MAFA and WideFace.

Thank you for diligently going through the repo. Let me check on my end, if any issue with generating FDDB dataset in Kitti format. Once you get FDDB dataset completely, you should have good accuracy.

Also, I am assuming, for accuracy evaluations, what batch size are you using?

I tried running all the data_utils scripts individually and: kaggle2kitti.py no data, throws an error widerface2kitti.py no data, no error (it instantaneously says that it generated 1000 items but there's nothing in the folder and no processing time) mafa2kitti.py generates 13,882 images and labels fddb2kitti.py generates 780 images and labels (All the paths in the files are correctly set and absolute)

Originally I tried the default batch size 24 that was written on the file for my first run, the I upped it to 32 because I read it in an article, and now I'm going to try 16 as it says in the blog entry corresponding to this project.

[hectormdom]

In widerface2kitti.py I changed line 33 from file_image = os.path.splitext(image_name)[0].split('\\')[1] to file_image = os.path.splitext(os.path.split(image_name)[1])[0] ran it alone and it actually generated 413 images and labels

[ak-nv]

Ah. Nice. One more quick question, are you running this on Linux or Windows?

[hectormdom]

Linux, Ubuntu 18.04.5 LTS

[ak-nv]

Ok. Let me test on that as well. i am currently testing on Windows.

[hectormdom]

Thanks a lot, I've tried a ton of things through comparison of the scripts that do work against the kaggle one, but to no avail

[ak-nv]

Closing as no activity for more than 20days. Please re-open if needed.