How to generate images from the PixelCNN?

Hanzy1996 commented 4 years ago

The PixelCNN learn to model the prior q(z) in the paper and the code. For any given classes/labels, PixelCNN should model their prior q(z), as shown in the code https://github.com/ritheshkumar95/pytorch-vqvae/blob/8d123c0d043bebc8734d37785dd13dd20e7e5e0e/modules.py#L262 here. And the prior here is actually the index of some codes in the codebook.

I first generate the index for some given classes as the codes https://github.com/ritheshkumar95/pytorch-vqvae/blob/8d123c0d043bebc8734d37785dd13dd20e7e5e0e/modules.py#L262 do, which is q(z)=GatedPixelCNN.generate(label). After I got the index q(z), I try to generate the images based on the index using the decoder in VQVAE https://github.com/ritheshkumar95/pytorch-vqvae/blob/8d123c0d043bebc8734d37785dd13dd20e7e5e0e/modules.py#L142, which is images=VectorQuantizedVAE.decode(q(z)). However, these generated images look very unrealistic, unlike the reconstruction results.

Can we evaluate the PixelCNN based on the generated images? How can I get the realistic images based on the prior generated by PixelCNN?

Best wishes!

mitkina commented 4 years ago

I am having the same issue! I am wondering if the train_dataset._label_encoder() is not constant? After training, I run the evaluation script, and in every run the labels seem to be mapped to a different integer encoding. Also, is there a reason that shuffle is turned off for the training data?

Thanks!

enk100 commented 3 years ago

@Hanzy1996 Can you please share the code for sample images based on training of the PixelCNN model?

Hanzy1996 commented 3 years ago

@enk100 @mitkina Sorry, I did not succeed to train the PixelCNN, either.

enk100 commented 3 years ago

@mitkina @Hanzy1996 this is what is was able to achieve :

@Hanzy1996 is this similar to what you generate?

mitkina commented 3 years ago

@enk100 the CIFAR10 images look similar to what I got.

MichalisLazarou commented 3 years ago

Hey guys could you please write the code how did you sample from the pixelCNN and then generate these images? I did what @Hanzy1996 suggested but I get really bad images, can you please write the steps to go from sampling pixelCNN--->images using the given functions?

kwang42 commented 1 year ago

to generate fake samples, just add the following at the end of main method in pixelcnn_prior.py:

latents = prior.generate(torch.LongTensor([0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,3,3,3,3,3,3,3,3,4,4,4,4,4,4,4,4,5,5,5,5,5,5,5,5,6,6,6,6,6,6,6,6,7,7,7,7,7,7,7,7]).cuda())

this tensor above contains the labels (each int 0-9 corresponds to one of the ten classes in the CIFAR10 dataset which I am

using): length 64 to create 64 images in an 8x8 table

samps = model.decode(latents)

fixed_grid = make_grid(samps, nrow=8, range=(-1, 1), normalize=True) writer.add_image('fakes', fixed_grid, 0)

then check them in tensorboard by inserting the following in a jupyter notebook

%load_ext tensorboard

%tensorboard --logdir logs/pixelcnn_prior

ritheshkumar95 / pytorch-vqvae