MiaoRain
commented
4 years ago 5th Input Preprocessing and Input Size: Since I used Imagenet pretrained models which required three-channel color input, each input image in this competition was duplicated two times and transformed into color image. There was no other preprocessing performed. The input image size for model training was fixed to the original size of the images: 256x1600. During early stage of the model development, I roughly tried cropping a smaller size, e.g., 224x768 for training and the original size for testing, but the public leaderboard result with cropped input turned out to be slightly worse. Another reason to not using a cropped version for training was because there would be no benefit for inference speed.
Image Augmentation: Image augmentation is one of the key factors for the successful training of deep neural nets. This was particularly true for this competition because of the label imbalance. There are a lot more negative samples compared with positive ones. When using random input sampling to form training batches, the model would tend to predict with less confidence the positive samples after the first dozens of epochs training, which means we would need smaller thresholds (rather than 0.5) to segment the positive regions, and these thresholds could extra hyperprameters we would want to avoid that. This problem would gradually disappear after the model was trained long enough, but the risk of overfitting (due to overtraining) would increase, which could be solved with the help of heavy augmentation. The augmentations I used includes crop and resize back, contrast and brightness, gamma correction, blurring and sharpening, scale and shift (no rotation and shearing), horizontal and vertical mirroring. The crop and resize back augmentation (the randomcrop function in any of the train.py) first crops horizontally and vertically a random portion, for example, crop 90% horizontally and 80% vertically, and then resize back to the original size to introduce some distortion effect. The random shift augmentation is a little unusual: instead of using a small portion of shift (typically at most 20%), I used a 100% random shift, and made the border filling circular (cv2.BORDERWRAP). I expected this circular shift augmentation to be simulating the manufacturing pipeline. During test time, I used either horizontal or vertical flipping or both for augmentation. Totally there were four variants including the original version.
Model Types: I trained totally eight Unet models and averaged their results. All of them were based on qubvel’s github: https://github.com/qubvel/segmentation_models.pytorch. The encoders used were inceptionresnetv2, efficientnetv4, seresnext50, se_resnext101, all pretrained on Imagenet. I trained two models for each type of the encoders. Due to the nature of the evaluation metric, the benefit of correctly detecting a negative sample is larger than detecting and segmenting a positive sample. Therefore, removing false positives are the key to high score. To achieve this, I used the max of all the predicted pixel probabilities as something like a classification score to perform thresholding on the averaged (across eight models) max probabilities and rule out false positives. I also noticed that a lot of participants trained some classifiers to specifically perform this false positive removal, I didn’t do that because of the extra cost of training the classifiers and the extra time to run them during testing. Note that to further improve the performance, we can use the top K (K>1) pixel probabilities instead of a single max probability for a more reliable estimate, but I did not go that far during the competition.
Training: Every model was firstly trained for 40-50 epoches using the traditional BCE loss. After the model weights were trained to a good state, I reduced the learning rate and shifted to lovasz-hinge loss (https://github.com/bermanmaxim/LovaszSoftmax) due to its robustness for imbalanced dataset. The models were trained until convergence, and further tuned using pseudo-labeling on the public test data for 10 epoches. This also helped improved the performance a little.
Testing and Postprocessing: In the early stage when I started this competition, I noticed that my cross-validation results were much better than the public leaderboard. Later, some participants posted in the Kaggle forum that there are some duplicate or near duplicate images in the training set, possibly generated by the same manufacturing pipelines, and the labels are very likely correlated between the duplicates. On the one hand, this breaks the i.i.d. assumption of data when applying machine learning models, resulting in over-optimistic validation estimates due to the label leakage; on the other hand, we could take advantage of the duplicates by jointly predicting clusters of duplicated images instead of independently predicting the single images. However, the fact that the cross-validation performance on the training set was much better than the public test set indicated that the duplicates in the training and public test set may not be perfectly overlapped, which implied new manufacturing pipelines in the public test set. Apparently, the difference between the train and public test set could not be a coincidence, it was more like intentionally done to mimic some real world scenario that required the models to generalize to new data rather than taking advantage of the label leakage. So I asked myself if the private test set would be another new set. My guess was, yes, because if the public and private test set are very much alike the participants could still take advantage of the label leakage by probing the public leaderboard. So some method to take advantage of the duplicates in the private set was necessary. For this, I used the efficientnet_b4’s bottleneck features (because of the lower dimensions) generated from the last output of the encoder as features for each image. I then grouped the duplicated images into clusters by kmeans clustering the features of the test data. In each cluster, I averaged the max pixel probabilities mentioned earlier. So, for each of the test image, the averaged max probabilities of its cluster were compared with pre-defined thresholds ([0.6,0.6,0.5,0.5] for the four classes), and this image would be determined to be negative if smaller than the pre-defined thresholds, if otherwise, the total number of positive pixels (binarized with the threshold 0.5) would be compared with pre-defined thresholds ([547,715,1145,2959] corresponding to the 2% quantile thresholds found on the train data) to further remove false positives. Only the test images that survived these two rounds of false positive removal went to the run-length encoding process to generate final positive masks.
First I’ve tried to include all predicted test negative samples (this was easier, since I didn’t need the masks) into the train set (keeping folds unchanged and not including pseudo-labeled images into validation set, to be able to track improvement) and fine tune the models I had using this data. It didn’t help much, so I moved on to including all predicted test set images the same way into training set and training from scratch. This allowed to get a single model score equal to the ensemble score on pubLB, but ensembling such models didn’t provide any improvement – it made sense, since most likely the models were just “remembering” masks for all the samples from the public test set, and there was no variation between predictions on the public test set images.
Finally, I decided to do it the “proper” way, and included only confident samples into the training set. I measured confidence similar to approach of winners of TGS competition - count the number of pixels with confidence < 0.2 or > 0.8, and consider images that have more than N confident pixels to be reliable. I’ve noticed, that no matter how certain/good the predictions are, there is always an uncertain region along the predicted defect boundary (which makes sense, since the ground truth annotation was fairly arbitrary in terms of boundary). To avoid this uncertain area affecting the “prediction confidence” estimate, I’ve applied morphological operations to “grow” confident regions – this allowed to “close” uncertain areas along the boundaries, and still kept the “really” uncertain defect predictions. This was done differently for each defect type, since they had different areas and boundary uncertainties.
最终复现方案 https://www.kaggle.com/c/severstal-steel-defect-detection/discussion/114410
目标分割+分类 视觉分割类, 赛题是针对钢板表面的图像，区分出四种瑕疵类型，并且分割出每种瑕疵类型所对应的位置与区域---语义分割。评测指标是Dice coefficient。
evaluated on the mean Dice coefficient 来评价X与Y的相似度
Dice 数组趋近于1 说明相似度越大， 最后结果是mean of the Dice coefficients for each <ImageId, ClassId> pair in the test set. 空值为一一
EncodedPixels 以run-length encoding on the pixel values. 起始位置+pixels长度 '1 3 10 5' implies pixels 1,2,3,10,11,12,13,14 The pixels are numbered from top to bottom, then left to right: 1 is pixel (1,1), 2 is pixel (2,1), etc.
a Kernels-only competition
四种缺陷类型 ClassId = [1, 2, 3, 4]
难点 难点:
每个图片包含零个为6000,1个缺陷为6100， 2个缺陷为200，3个没有
1st Place Solution https://www.kaggle.com/c/severstal-steel-defect-detection/discussion/114254 Classification Classification is an important part of this competition. Even though classifiers can only slightly improve your score after 0.915 in public LB, they can work as a preliminary screening (初步筛选) by filtering out around half of images with no defects. This will enable us to ensemble more models in segmentation part. We trained our classifiers with random crop of 224x1568 and do inference on full size. This random crop gives slightly improvement on accuracy. Augmentations: Randomcrop, Hflip, Vflip, RandomBrightnessContrast (from albumentations) and a customized defect blackout Since this is a sematic segmentation task, we know exactly where the defects are. As a result, these defects components can be randomly blacked out (中断) and the label for this image will also change from 1 to 0 if all defects are blacked out. This augmentation indeed works on local CV and public LB. Here are some graphs of the training process of a ResNet34 classifier. Batchsize: 8 for efficientnet-b1, 16 for resnet34 (both accumulate gradients for 32 samples) Optimizer: SGD Model Ensemble: 3 x efficientnet-b1+1 x resnet34 TTA: None, Hflip, Vflip Threshold: 0.6,0.6,0.6,0.6
Segmentation We have to admit that we used models from @lightforever these models improved our score from 0.907 private LB to our current score. Train data: 256x512 crop images Augmentations: Hflip, Vflip, RandomBrightnessContrast (from albumentations) Batchsize: 12 or 24 (both accumulate gradients for 24 samples) Optimizer: Rectified Adam Models: Unet (efficientnet-b3), FPN (efficientnet-b3) from @pavel92 segmentationmodelspytorch Loss: BCE (with posweight = (2.0,2.0,1.0,1.5)) 0.75BCE+0.25DICE (with posweight = (2.0,2.0,1.0,1.5)) Model Ensemble: 1 x Unet(BCE loss) + 3 x FPN(first trained with BCE loss then finetuned with BCEDice loss) +2 x FPN(BCEloss)+ 3 x Unet from mlcomp+catalyst infer TTA: None, Hflip, Vflip Label Thresholds: 0.7, 0.7, 0.6, 0.6 Pixel Thresholds: 0.55,0.55,0.55,0.55 Postprocessing: Remove whole mask if total pixel < threshold (600,600,900,2000) + remove small components with size <150
Pesudo Label We did 2 rounds of pseudo labels in this competition. The first round is generated from a submission with 0.916 public LB, maybe it is too early? The second round was done several days before the end of this competition, generated from a submission with 0.91985 public LB. With pseudo label and public models, we finally improved from 0.91985 to 0.92124 on public LB and from 0.90663 to 0.90883 on private LB. The pseudo labels are chosen if classifiers and segmentation networks make the same decisions. We got this idea from Heng. An image will only be chosen if the probabilities from classifiers are all over 0.95 or below 0.05 and it gets same result from segmentation part. According to this rule, 1135 images are chosen and added to trainset.
Predictions on public LB: Defect 1: 97(128) Defect 2: 2(43) Defect 3: 611(741) Defect 4: 110(120) Sum Pos: 820(1032)