Other imaging modalities, input size, network model

[overfrequency]

Hi, Can we use the code to transform other types of images? For example MRI to CT?

1. How can we change the input shape (i.e to a smaller size like 28x28)? Should we change the U-net parameters in that case?
2. Is it possible to use gray scale images instead of binary images at the input side?

Thanks,

[costapt]

Hi,

Yes, you can use the code for other image modalities. The only constraint is that you need pairs of images as input to the model.

Regarding question 1, to change the input shape you will need to change the models.py file. You need to redefine the g_unet and discriminator models so that they accept 28x28 inputs and then update the input shape on the pix2pix method. It is actually easier if the input shape is a power of 2 as you would only need to delete some of the existing layers. If you need help changing any of those methods I can help you with that.

The answer to the second question is yes. If your input image is grayscale you need to run the code with the following parameters: --is_a_binary False --is_a_grayscale True

[overfrequency]

Hi,

Thank you very much for the answer. I will try to do the modifications and will write the results here.

Note: To run the code seaborn package needs to be installed.

[overfrequency]

Hi,

Although my results are not as successful as yours, I was able to run the code with these modifications. Also, I need to add the following before using pyplot:

import matplotlib matplotlib.use('Agg')

Thanks.

[costapt]

Hi,

Thank you for your feedback.

You might need to tune your network architecture since you changed the size of the input images. In our case, the discriminator classifies 16x16 patches of size 63x63 pixels. Since you images are small, maybe you should reduce the size of the output of the discriminator.

Thanks.

[overfrequency]

Thank you for the suggestion.

My input images are 64x64 pixels. How should I change the output of the discriminator in this case?

Thanks again,

[costapt]

Can you show me your current discriminator model? I can make a suggestion then.

[overfrequency]

Here is my discriminator:

def discriminator(a_ch, b_ch, nf, opt=Adam(lr=2e-4, beta_1=0.5), name='d'):
    """Define the discriminator network.

    Parameters:
    - a_ch: the number of channels of the first image;
    - b_ch: the number of channels of the second image;
    - nf: the number of filters of the first layer.
    """
    i = Input(shape=(a_ch + b_ch, 64, 64))

    # (a_ch + b_ch) x 64 x 64
    conv1 = Convolution(nf)(i)
    x = LeakyReLU(0.2)(conv1)
    # nf x 32 x 32

    conv2 = Convolution(nf*2)(x)
    x = LeakyReLU(0.2)(conv2)
    # nf*2 x 16 x 16

    conv3 = Convolution(nf * 4)(x)
    x = LeakyReLU(0.2)(conv3)
    # nf*4 x 8 x 8

    conv4 = Convolution(nf * 8)(x)
    x = LeakyReLU(0.2)(conv4)
    # nf*8 x 4 x 4

    conv5 = Convolution(1)(x)
    out = Activation('sigmoid')(conv5)
    # 1 x 2 x 2

    d = Model(i, out, name=name)

    def d_loss(y_true, y_pred):
        L = objectives.binary_crossentropy(K.batch_flatten(y_true), K.batch_flatten(y_pred))
        return L

    d.compile(optimizer=opt, loss=d_loss)
    return d

[costapt]

I was also thinking about something like that.

I am not really sure what you can do. You can try to monitor the discriminator's and generator's losses to try to debug what is the problem.

If the discriminator's loss is low then you probably do not have enough data and you could reduce the size of the patch that the discriminator classifies, by decreasing its depth.

I am sorry I can not be of much help.

[overfrequency]

Hi,

You have been very helpful. Thank you very much. For training, I have 1418 image pairs of sizes 64x64 pixels. And my losses are like this:

Do you think the model overfits very quickly?

[costapt]

Yes, the discriminator's loss goes down rather quickly and the generator's loss is decreasing too slowly. You could remove one more layer of the discriminator and try again.

[overfrequency]

I removed three layers from the discrminator (conv2, conv3 and conv4). The losses are as follows but the visual results are not satisfying.

Do you have any other suggestions? Thank you very much for your help and for your time.

[costapt]

I am sorry for the delay in the response.

Now it looks like that you have the opposite problem. You discriminator is not being able to distinguish between real and fake pairs. You should add some layers.

I hope that you were able to solve your problem.

[overfrequency]

Thank you very much for your comment. Unfortunately adding or removing just one layer of the discriminator makes this difference. If I add one more layer then the discriminator's loss goes down rather quickly and the generator's loss is decreasing too slowly as before. Do you have any other suggestion?

Second thing is, generated images are not smooth. I can see borders of the patches clearly (below). What should I do to solve this and have a smooth image?

Again many thanks for your help.

[costapt]

That is strange.

You could try changing the alpha parameter. If you increase alpha, the model will give more weight to the L1 reconstruction loss. By default alpha is set to 100. Try increasing it and see what happens.

[overfrequency]

I will try this as soon as possible. Thanks a lot!

[overfrequency]

Hi, Increasing alpha did not improve the results (Losses were not stable). Then, I removed just one layer from the discriminator and kept the others. Now I have four convolutional layers. I have selected alpha=50. Here is the results:

Results are slightly better. But generated images are still see pixelated.

[costapt]

Hi!

I am sorry I did not respond sooner. Actually, those plots do not look that bad. You just need to let the model run for more epochs and the results will eventually get better. You can see that there's a slight trend upwards in the discriminator's loss and a downward trend in the pix2pix loss.

Did you manage to make it work?

[overfrequency]

Hi!

Thanks a lot! I will try to run for more epochs. My results are exactly at the same situation where I left. I could not find a better way to workaround the problems.

Another thing, is it possible to write the losses and epochs to a file so that we can plot it elsewhere?

[costapt]

Hi!

You can change the plot_loss method in util/util.py to save the losses to a file. I am sorry, I do not have time to make that change in the code at the moment :(

[GinkgoX]

Hi, Can we use the code to transform other types of images? For example MRI to CT?

1. How can we change the input shape (i.e to a smaller size like 28x28)? Should we change the U-net parameters in that case?
2. Is it possible to use gray scale images instead of binary images at the input side?

Thanks,

Hi, could you share with the requirement.txt file so that we can run the test demo? thanks a lot!