pix2pix - Testing on a new image

[s4chin]

The validation images generated during training appear to be in much better quality than the same images using the saved model(Using the saved model from the same epoch). I think it is due to some error on my testing code. I would be glad if you could help me out here -

    from scipy.misc import imsave
    img_path = os.path.join('path/to/input/image.jpg')
    img = cv2.imread(img_path)
    img = img[:, :, ::-1]  # GBR to RGB
    data = normalization(np.expand_dims(img, 0).transpose(0, 3, 1, 2)).transpose(0, 2, 3, 1)
    out = generator_model.predict(data) # generator_model is already loaded with weights
    out = inverse_normalization(out)
    print out.shape # (256, 256, 3)
    out=out[0, :, :, :]
    print out.shape # (256, 256, 3)
    imsave('path/to/output/image.jpg', out)

I think it has something to do with .asarray(np.float32) and np.uint8.

[tdeboissiere]

Which keras version are you using ? It could have something to do with the Batch Norm mode (in early versions, you have a mode to compute BN statistics for inference in the same way as for training, i.e. use a per batch mean and standard dev. If you are in this mode, then when you make predictions, the estimated BN statistics given only one image may not be very good)

Do you have image samples ? Do you get the same quality if you save with cv2.imwrite ?

[s4chin]

Keras version is 1.2.2 For using cv2.imwrite i have to multiply out by 255, but the saved images look more or less the same. Here are the images(at 350th epoch) generated by the plot_generated_batch- Here are the same generated images at same epoch by the above code - Also, I now run the above code every 50 epochs during training itself using the generator_model so as to remove any errors that may be caused due to saving and loading models.

[tdeboissiere]

If I'm not mistaken, during training, predictions are made in batch before being saved:

X_pred = generator.predict(X_batch) # X_batch is maybe (16,3,256,256) for i in range(X_pred.shape[0]): cv2.imwrite(X_pred[i])

However you call predict on a single image.

2 possibilities:

• You are in the BN mode where moving mean and stddev are computed at training to be re-used at test time. In this case I would expect the same results whether you predict on a single image of a whole batch of image.
• You are in the BN mode where moving mean and stddev are computed at test time for each new batch you pass. You will get different results depending on how many images are in your test batch. That may explain your artifacts.

Did you try running your image saving code on multiple images at the same time (i.e. data has more than 1 sample on axis=0) ? Do check the mode argument in the BatchNorm layer.

I met a few float/int issues before but these led to images far more deteriorated than the ones you show.

[s4chin]

Alright, it was the case 2. Predicting in a batch led to no artifacts, and patches. I predicted on multiple images together and it worked. :+1: Also, having a batch of 4 gave slightly better results in terms of colors than a batch of 16. I guess that has something to do with having a batch_size of 4 during training too. Thank you!

[tvpian]

Hi Sachin, I'm a newbie in the field of deep learning. Out of pure interest I have started to work on pix2pix. I need a small help. I completed the training of my model and now I want to test it on single new images. Can you please brief me how to do it. To be particular I'm working on the Layout-Building example.I used your testing code above to test my model too.Unfortunately the output that I obtained seem to vary a lot from the actual expected output.

Thank you. Tharun