search

junyanz / pytorch-CycleGAN-and-pix2pix

Image-to-Image Translation in PyTorch
Other
22.39k stars 6.22k forks source link

CycleGan with Paired Images #131

Closed fschiffers closed 6 years ago

fschiffers commented 6 years ago

Hello,

I am dealing with the following problem: I have two paired medical image datasets from two different modalities. However, the images cannot be registered well, so pix2pix should not be a good option. CycleGan is a method that works fairly well and was already employed by Wolterink et al. for MRI/CT datasets in "Deep MR to CT Synthesis using Unpaired Data" using your DL framework.

However, CycleGAN seems to be not the perfect choice under these conditions since the information that there indeed exists a corresponding (though not registered) image pair is neglected.

I don't have a lot of experience in Machine or Deep Learning, so my thinking might be very naive: In CycleGan I think the idea to use a second generator to go also from B -> A instead of only A-> B is - simply put - great. But this should not only hold for unpaired image data, but also for paired data. Wouldn't it make sense to combine the strengthes of Pix2Pix and CycleGan ?

Best, Florian

junyanz commented 6 years ago

We haven't tried it by ourselves. Maybe you can simply try both pix2pix objective (conditional GANs and L1) and a small cycle consistency loss.

leochan2009 commented 3 years ago

we have a similar problem. we are trying to train a model that translate video data captured by one camera to a video that looks like captured by another camera. we have video clips from both cameras. Some video clips are paired but not registered (two cameras are bundled together and capture the same scene) and some clips are not paired. I think the model will be more robust if the pairing information can be leveraged during the training.