doruksonmez
commented
3 years ago I have the same issue.
Open hosseinzadeh88 opened 3 years ago
doruksonmez
commented
3 years ago I have the same issue.
hosseinzadeh88
commented
3 years ago I have the same issue.
@doruksonmez could you please share your logs as well? At the moment, looking at the samples that are ignored (starting with 114_mask.txt) they all seem to be belonging to the Kaggle dataset. I believe I previously had the same issue with some images from the MAFA datase as well, but there is no sign of those in my logs anymore! I'm not even sure why, I changed a couple of lines of code mafa2kitti.py (as explained previously).
Also, looking at the 114_mask.xml and then looking at my initial comment logs of 114_mask.txt, it's not clear whats happening
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use None for the system default.
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/114_mask.txt.
Coordinates: x1 = 739, x2 = 478, y1: 90, y2: 388
Skipping this objectCompare this to the content of the 114_mask.xml
<xmin>924</xmin>
<ymin>133</ymin>
<xmax>598</xmax>
<ymax>571</ymax>Not sure if the kaggle image labels are not being processed correctly or the labels are incorrect!
doruksonmez
commented
3 years ago Hi,
I also have the same line along with the other similar outputs.
Top left coordinate must be less than bottom right.Error in object 0 of label_file /root/kitti-outs/train/labels/114_mask.txt.
Coordinates: x1 = 739, x2 = 478, y1: 90, y2: 388
Skipping this objectRegarding to other thing you mentioned about the pixel points, it shouldn't be an issue since data being rescaled to (960,544). However, I have a strange issue with my training procedure. It doesn't train at all and I can't see why.
Matching predictions to ground truth, class 1/2.: 100%|█| 16537/16537 [00:00<00:00, 192885.94it/s]
Matching predictions to ground truth, class 2/2.: 100%|█| 7640/7640 [00:00<00:00, 188193.63it/s]
Epoch 1/20
=========================
Validation cost: 0.015327
Mean average_precision (in %): nan
class name average precision (in %)
------------ --------------------------
mask nan
no-mask nan
...
Epoch 11/20
=========================
Validation cost: -0.000008
Mean average_precision (in %): 0.0000
class name average precision (in %)
------------ --------------------------
mask 0
no-mask 0
Median Inference Time: 0.009497Regarding to other thing you mentioned about the pixel points, it shouldn't be an issue since data being rescaled to (960,544). However, I have a strange issue with my training procedure. It doesn't train at all and I can't see why.
@doruksonmez you're absolutely correct about the scaling, I didn't consider the fact that images are rescaled.
So we still don't know why some of the Kaggle samples are ignored during the tlt-dataset-convert.
@doruksonmez . Just to clarify. You are having the problem with tlt-dataset-convert generating the tfrecords but still continuing the training, despite the issue? Also, are you using your own database? It seems to me as if your class 1/2 has 16537 samples and your class 2/2 has 7640. Is that what it is? As with your issue with the training, I have no idea why your training is not working!
doruksonmez
commented
3 years ago @hosseinzadeh88 Yes, I am using tlt-dataset-convert tool for tfrecords and it skips some of the data but 3533 of them are still there and usable. So I tried increasing the category-limit to increase the size of the data and retried to train with that. But no luck. I think that's what you are seeing with the different sample counts, I am not trying with a custom dataset.
Yeah, it is frustrating to see my model is not being trained. But I am still trying to figure out what is wrong.
ak-nv
commented
3 years ago @hosseinzadeh88 and @doruksonmez
Top left coordinate must be less than bottom right.Error in object 3 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/142_mask.txt. Coordinates: x1 = 419, x2 = 323, y1: 119, y2: 239
This is warning is if dataset has X2 <X1 , we convert in KITTI format which needs xmin, ymin, xmax, ymax format. You can have simple script update in mafa2kitti.py have a simple check here for xmin < xmax to avoid it. I have not check it yet though.
Also, are you using your own database? It seems to me as if your class 1/2 has 16537 samples and your class 2/2 has 7640. Is that what it is?
In this case, your classifier will be biased towards predicting class 1
As with your issue with the training, I have no idea why your training is not working!
Which dataset are you using? What are your hyper-params. Can you share your detectnet_v2_train_resnet18_kitti.txt
Some minor fixes to improve accuracy
I just realized that, in widerface dataset, faces are obscure sometimes, so I now limit number of faces in the image. This might improve accuracy for faces without mask.
doruksonmez
commented
3 years ago @ak-nv thanks for your comment.
I am using the exact same datasets you are using. You can find my conversion_spec.txt and detectnet_v2_train_resnet18_kitti.txt below.
conversion_spec.txt:
kitti_config {
root_directory_path: "/workspace/dataset/kitti-dataset"
image_dir_name: "train/images"
label_dir_name: "train/labels"
image_extension: ".jpg"
partition_mode: "random"
num_partitions: 2
val_split: 20
num_shards: 10
}
image_directory_path: "/workspace/dataset/kitti-dataset"detectnet_v2_train_resnet18_kitti.txt:
random_seed: 42
dataset_config {
data_sources {
tfrecords_path: "/workspace/dataset/tfrecords/*"
image_directory_path: "/workspace/dataset/kitti-dataset"
}
image_extension: "jpg"
target_class_mapping {
key: "Mask"
value: "mask"
}
target_class_mapping {
key: "No-Mask"
value: "no-mask"
}
validation_fold: 0
}
augmentation_config {
preprocessing {
output_image_width: 960
output_image_height: 544
min_bbox_width: 1.0
min_bbox_height: 1.0
output_image_channel: 3
}
spatial_augmentation {
hflip_probability: 0.5
zoom_min: 1.0
zoom_max: 1.0
translate_max_x: 8.0
translate_max_y: 8.0
}
color_augmentation {
hue_rotation_max: 25.0
saturation_shift_max: 0.20000000298
contrast_scale_max: 0.10000000149
contrast_center: 0.5
}
}
postprocessing_config {
target_class_config {
key: "Mask"
value {
clustering_config {
coverage_threshold: 0.00499999988824
dbscan_eps: 0.20000000298
dbscan_min_samples: 0.0500000007451
minimum_bounding_box_height: 20
}
}
}
target_class_config {
key: "No-Mask"
value {
clustering_config {
coverage_threshold: 0.00499999988824
dbscan_eps: 0.15000000596
dbscan_min_samples: 0.0500000007451
minimum_bounding_box_height: 20
}
}
}
}
model_config {
pretrained_model_file: "/workspace/dataset/resnet18.hdf5"
num_layers: 18
freeze_blocks: 0
use_pooling: False
all_projections: True
use_batch_norm: true
activation {
activation_type: "relu"
}
objective_set {
bbox {
scale: 35.0
offset: 0.5
}
cov {
}
}
training_precision {
backend_floatx: FLOAT32
}
arch: "resnet"
}
evaluation_config {
validation_period_during_training: 10
first_validation_epoch: 1
minimum_detection_ground_truth_overlap {
key: "Mask"
value: 0.699999988079
}
minimum_detection_ground_truth_overlap {
key: "No-Mask"
value: 0.5
}
evaluation_box_config {
key: "Mask"
value {
minimum_height: 20
maximum_height: 9999
minimum_width: 10
maximum_width: 9999
}
}
evaluation_box_config {
key: "No-Mask"
value {
minimum_height: 20
maximum_height: 9999
minimum_width: 10
maximum_width: 9999
}
}
average_precision_mode: INTEGRATE
}
cost_function_config {
target_classes {
name: "Mask"
class_weight: 1.0
coverage_foreground_weight: 0.0500000007451
objectives {
name: "cov"
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: "bbox"
initial_weight: 10.0
weight_target: 10.0
}
}
target_classes {
name: "No-Mask"
class_weight: 8.0
coverage_foreground_weight: 0.0500000007451
objectives {
name: "cov"
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: "bbox"
initial_weight: 10.0
weight_target: 1.0
}
}
enable_autoweighting: true
max_objective_weight: 0.999899983406
min_objective_weight: 9.99999974738e-05
}
training_config {
batch_size_per_gpu: 16
num_epochs: 20
learning_rate {
soft_start_annealing_schedule {
min_learning_rate: 5e-06
max_learning_rate: 5e-04
soft_start: 0.10000000149
annealing: 0.699999988079
}
}
regularizer {
type: L1
weight: 3.00000002618e-09
}
optimizer {
adam {
epsilon: 9.99999993923e-09
beta1: 0.899999976158
beta2: 0.999000012875
}
}
cost_scaling {
initial_exponent: 20.0
increment: 0.005
decrement: 1.0
}
checkpoint_interval: 10
}
bbox_rasterizer_config {
target_class_config {
key: "Mask"
value {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 0.40000000596
cov_radius_y: 0.40000000596
bbox_min_radius: 1.0
}
}
target_class_config {
key: "No-Mask"
value {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 1.0
cov_radius_y: 1.0
bbox_min_radius: 1.0
}
}
deadzone_radius: 0.400000154972
}Output for the conversion:
root@321bd4274bec:~/face-mask-detection# python3.6 data2kitti.py --kaggle-dataset-path /workspace/dataset/dataset/KaggleMedicalMaskDataset --mafa-dataset-path /workspace/dataset/dataset/MAFADataset --fddb-dataset-path /workspace/dataset/dataset/FDDBDataset --widerface-dataset-path /workspace/dataset/dataset/WiderFaceDataset --kitti-base-path /workspace/dataset/kitti-dataset --category-limit 6000 --tlt-input-dims_width 960 --tlt-input-dims_height 544 --train
Kaggle Dataset: Total Mask faces: 4154 and No-Mask faces:790
Total Mask Labelled:4154 and No-Mask Labelled:790
Directory Already Exists
Directory Already Exists
/root/face-mask-detection/data_utils/mafa2kitti.py:51: RuntimeWarning: overflow encountered in ubyte_scalars
bbox = [_bbox_label[0], _bbox_label[1], _bbox_label[0]+_bbox_label[2], _bbox_label[1]+_bbox_label[3]]
MAFA Dataset: Total Mask faces: 1846 and No-Mask faces:232
Total Mask Labelled:6000 and No-Mask Labelled:1022
Directory Already Exists
Directory Already Exists
FDDB Dataset: Mask Labelled:0 and No-Mask Labelled:2845
Total Mask Labelled:6000 and No-Mask Labelled:3867
WideFace: Total Mask Labelled:0 and No-Mask Labelled:2134
----------------------------
Final: Total Mask Labelled:6000
Total No-Mask Labelled:6001
----------------------------@doruksonmez could you please run this script on your training dataset. I tried this on my dataset after successfully (apparently) running the data2kitti.py code, and to my surprise, I DO NOT have 6000 faces labeled as masked nor have 6001 labels of No-Mask! data2kitti.py is showing an incorrect total in my opinion. Please run this, I've also attached the script for you (it's slightly different as I was trying to extract all the face as well using opencv). Just have to change the datasetDir variable to point to the train directory, inside which you have 2 folders, images and labels.
import glob
maskedCounter = 0
noMaskCounter = 0
undefCounter = 0
datasetDir = '/home/sal/Desktop/Data4Training/train' # Main directory that contains two folders named "images" and the "labels"
# getting all the label files from the label folder within the datasetDir
fileNames = glob.glob(datasetDir+'/labels/'+'*.txt')
print(f'Total of {len(fileNames)} files found in the labels directory.')
for fileName in fileNames:
name = fileName.split('/')[-1]
name = name.split('.')[0]
# read the image and the labels
labelFile = open(datasetDir + '/labels/' + name + '.txt')
lines = labelFile.readlines()
for line in lines:
classLabel = line.split()[0]
#extract the faces from the images and write them to a seperate file.
if classLabel == 'No-Mask':
noMaskCounter = noMaskCounter + 1
elif classLabel == 'Mask':
maskedCounter = maskedCounter + 1
else:
undefCounter = undefCounter + 1
print(f'Total "MASK" labels: {maskedCounter}\nTotal "No-Mask" labels: {noMaskCounter} \nTotal undefined labels: {undefCounter}')I think something is happening during the dataset generation, apparently, the widerfaces2kitti.py does not write to the disk or something like this, still trying to figure it out. @doruksonmez if you don't have 6000 "mask" and 6000 "no mask" labels please let me know so we can open another issue and inform the author.
This is what I get when I run this script:
Total of 3533 files found in the labels directory.
Total "MASK" labels: 5050
Total "No-Mask" labels: 3676
Total undefined labels: 0Try the latest update from the author, 7a9f2c00280e206339d0f054216570429be640f4 They have updated the wider2kitti.py file which was not writing the files to the disk and was not even raising an exception to notice! So much time spent on training the network without actually having the correct dataset (@ak-nv ) I hope this has resolved the issue.
ak-nv
commented
3 years ago @doruksonmez
In your detectnet_v2_train_resnet18_kitti.txt ; image_directory_path should be "/workspace/dataset/kitti-dataset/train"
I assume, you are copying resnet18.hdf5 to dataset folder.
Let me know if this helps.
doruksonmez
commented
3 years ago @ak-nv
Thanks for the comment. I tried to train with your detectnet_v2_train_resnet18_kitti.txt file inside tlt_specs and it worked!
This is the unpruned model evaluation output:
Validation cost: 0.001529
Mean average_precision (in %): 87.2487
class name average precision (in %)
------------ --------------------------
mask 88.3731
no-mask 86.1242And this is for the pruned one:
Validation cost: 0.001639
Mean average_precision (in %): 86.4396
class name average precision (in %)
------------ --------------------------
mask 88.9616
no-mask 83.9176However, when I export the pruned model and run DeepStream, I get a strange output bboxes as follows:
ak-nv
commented
3 years ago Good to hear that accuracy is close to what we had got. To resolve above issue with deepstream follow steps
Let me know if this solves it. I have also mentioned it in ReadMe.
lmarval
commented
3 years ago @hosseinzadeh88 and @doruksonmez
Top left coordinate must be less than bottom right.Error in object 3 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/142_mask.txt. Coordinates: x1 = 419, x2 = 323, y1: 119, y2: 239
This is warning is if dataset has X2 <X1 , we convert in KITTI format which needs xmin, ymin, xmax, ymax format. You can have simple script update in
mafa2kitti.pyhave a simple check here for xmin < xmax to avoid it. I have not check it yet though.Also, are you using your own database? It seems to me as if your class 1/2 has 16537 samples and your class 2/2 has 7640. Is that what it is?
In this case, your classifier will be biased towards predicting
class 1As with your issue with the training, I have no idea why your training is not working!
Which dataset are you using? What are your hyper-params. Can you share your
detectnet_v2_train_resnet18_kitti.txtSome minor fixes to improve accuracy
I just realized that, in widerface dataset, faces are obscure sometimes, so I now limit number of faces in the image. This might improve accuracy for faces without mask.
For the X2 <X1 problem, you could add in kaggle2kitti.py
(xmin, xmax) = (xmin, xmax) if xmax >xmin else (xmax, xmin) # some bounding boxes are stored in inverse order (ymin, ymax) = (ymin, ymax) if ymax >ymin else (ymax, ymin) # some bounding boxes are stored in inverse order
at line 74 and line 84(86).
hosseinzadeh88
commented
3 years ago Just use the dataset that I have shared on google drive. See my other comments etc, it has 4000 masked and 4000 non-masked
Hi @ak-nv I am having an issue with creating the tfrecords for training. I used data2kitti.py to generate the training dataset, resulted in the following log
Just to mention, there was an ubyte overflow warning in lines 51 and 81 of the maffa2kitti.py file. I added the bbox coordinates in an int() to avoid the overflow. Even before I amend the code I had problem with the tfrecords, so I don't think this change has caused the issue with the tfrecords generation. Hopefully, the change I made was more of a fix than creating and issue.
So, assuming that the dataset is created correctly, lets get to the issue now.
The boxes don't seem to have the correct corners coordinates! Following the Jupyter Notebook, when I run tlt-dataset-convert I end of with quite a bit data samples being ignored. I guess this was mentioned in #5. I have got the latest version of the code as well "f62edc0"
The logs from running the tlt-dataset-convert as follows:
Any suggestions as what is causing this issue and how to resolve it please? Thanks