[Fixed!] Test mIoU using OFFICIAL cityscapes scripts

[Sixkplus]

I've been dealing with semantic segmentation for a couple of months. Guys, I'm curious whether you test your mIoU using evaluate.py or the code provided by this repo. I use the provided checkpoint train_30k, I don't think the result yields 67% mIoU. In fact, when I use the official code to test the accuracy on validation set(500 images), it only gives 27.9% mIoU. Or, if anyone has made the same experiment. Could you please share your mIoU on test(online)/validation set? Just like the following form. Or, if anyone has submitted results using this repo on https://www.cityscapes-dataset.com and get ~67% mIoU?

classes IoU nIoU

road : 0.924 nan sidewalk : 0.535 nan building : 0.517 nan wall : 0.042 nan fence : 0.141 nan pole : 0.106 nan traffic light : 0.020 nan traffic sign : 0.357 nan vegetation : 0.251 nan terrain : 0.179 nan sky : 0.353 nan person : 0.400 0.000 rider : 0.067 0.000 car : 0.752 0.000 truck : 0.071 0.000 bus : 0.336 0.000 train : 0.051 0.000 motorcycle : 0.009 0.000 bicycle : 0.196 0.000

Score Average : 0.279 0.000

What's more, my own model only gives 64%mIoU and I think the quality is better than the predicted result using this pre-trained model(67%mIoU). Or if I made something wrong during the configuration?

For comparison, the upper image is the result using my model, below is result using this repo, This is frankfurt_000000_000294_leftImg8bit.png in validation set.

[hellochick]

Hey @Sixkplus,

Thank you for the information, I have tested the mIoU using the official cityscapes script and translate the trainId into regular ID. I got the same result as your's.

I am now trying to find the reason which cause such strange results. If you use evaluate.py, you can get the desired results. Maybe there exists some label ID translation problem when use their script?

[hellochick]

I have found the problem! Finally got the desired results,

It seems that there are some shift in the image if you use inference.py to generate the output image. So if you use eveluate.py and store the output image, you can get this results!

[Sixkplus]

Thanks! @hellochick I'll try to fix it! :D

[Sixkplus]

Hey! @hellochick I found this problem is caused by the fact you forget to pass the parameter img_mean in mode configuration block to the function _infer_preprocess in model.py. I've fixed it as follows and it works well now. THx~

elif mode == 'inference':
            # Create placeholder and pre-process here.
            self.img_placeholder = tf.placeholder(dtype=tf.float32, shape=cfg.INFER_SIZE)
            self.images, self.o_shape, self.n_shape = _infer_preprocess(self.img_placeholder, cfg.IMG_MEAN)

            super().__init__(inputs={'data': self.images}, cfg=self.cfg)

            self.output = self.get_output_node()

[Sixkplus]

@hellochick BTW, Could you please tell me which model you use to get mIoU 67.3? I use the train_30k and get 64.2

[hellochick]

@Sixkplus , thank you a lot!! I will update the bug codes in recent days >_<!

I used the train_30k model, maybe you can take a look at the pre-processing procedure in evaluation phase, which are different with inference phase. For instance, the input image will first resize to [1025, 2049] at first in evaluation phase (here I just want to re-produce the original code which wrote in MATLAB).

[hellochick]

And also, the model definition is also a little bit different, you can check it in model.py.

[Sixkplus]

@hellochick It turns out that during inference, you should also implement zero-padding (make it from [1024, 2048] to [1025, 2049] ) and this will result in the same result as eval. ( I think this is quite strange as the resolution is large enough : (, but it really works).

Again, thank you for your suggestions and help !

[Sixkplus]

Sorry to disturb again, @hellochick ! Would you mind sharing your configurations for training cityscapes using train_30k mode to achieve over 67%mIoU. I tried to continue training while the accuracy drops, I think this might be caused by different configurations.

[shuyueL]

@Sixkplus I got 64.2% too. Could you share how to modify the code to get to 67.3%?