Generate photorealistic images from human-drawn sketches using GANs - any model suggestions?

[ehandorfer]

I am currently trying to implement a GAN that does transform human-draw sketches (very simple drawings) into photorealistic images. However, since the two domains actually do differ quite a lot, I am kind of lost on where to start. I tried the most popular GAN models, such as CycleGAN, which delivers pretty poor results due to the domain gap, and also pix2pix, but there is not really a paired dataset for such sketches and photos. I actually invested a lot of time in data preprocessing, such as performing data augmentation by adding (HED) egde maps to the sketches (but again edge maps are quite different from simple sketches), and I also transformed the sketches into distance fields to make the input data more dense. However, right now I do not really know which type of GAN is a good fit for such a kind of task, so any input is appreciated.

I invested most of the time training a CycleGAN instance, but the generated data is pretty bad. I thought that maybe the SketchyGAN model would be a good idea, but unfortunately the implementation is based on an old tensorflow version.

[JustinasLekavicius]

What generator parameters are you using? And what are your training results? The results may be improved by strengthening the generator, i.e. increasing the ngf value, or training for more epochs. Discriminator may also need to be either strengthened or weakened. Depends on the balance between them, and how the training results look.