Sampling a new data point using TFP distribution trained model

[fused-byte]

Hi @MokkeMeguru ,

Thanks for creating such a nice tutorial on flow using tensorflow probability. There is hardly anything available outside.

I have a question regarding, how we can generate new data points once the model is trained? I tried in the following ways: 1. just the sample() function, 2) sampling and then doing a bijector.forward() followed by inverse, 3) sampling followed by bijector.inverse(). None of them worked.

I also wanted to ask, while training once we get the latent space after bijector.forward(), is the latent space first sampled and then bijector.inverse() is called? I am asking this because our network is a bijection it will reproduce the same image as the same operations will be performed.

[MokkeMeguru]

Thanks for your comment @fused-byte . i have some question and answer.

questions

1. data point means the sampling image, right?

   answers

2. sample() function can sample an data point like x-y position see. https://github.com/MokkeMeguru/glow-realnvp-tutorial/blob/master/tips/RealNVP_tutorial_en.ipynb 's inference.

3. I think you should check your implemented bijector's mechanics. sometimes, tensorflow-probability often reverses the direction of data<->latent connection internally. (tensorflow-probability.distributions.TransformedDistribution internally does it, I remembered)

4. sry, I cannot understand what you did. I'll write some code...

Optional answers

I also wanted to ask, while training once we get the latent space after bijector.forward(), is the latent space first sampled and then bijector.inverse() is called? I am asking this because our network is a bijection it will reproduce the same image as the same operations will be performed.

I think this reversible network is too hard to implement with tfp... because they are too helpfully framework. we can try to use pixyz(PyTorch's generative model library https://github.com/masa-su/pixyz) or TFGENZOO(my library https://github.com/MokkeMeguru/TFGENZOO and it's tutorial is https://github.com/MokkeMeguru/TFGENZOO/blob/master/tutorials/01_What_is_the_invertible_layer.ipynb)

If you want to implement what you say with TFP, please let me know with detail requirements (e.g. mnist, cifer-10, glow, etc...).

[MokkeMeguru]

or you can see... https://www.tensorflow.org/probability/api_docs/python/tfp/bijectors/Glow#top_of_page ... (its internal implementation is too complex for me... but it seems correct )

[fused-byte]

@MokkeMeguru thank you for such a detailed response and suggestions.

Yes, by data point I meant sampling an image.

I followed this tutorial https://github.com/MokkeMeguru/glow-realnvp-tutorial/blob/master/tips/glow_mnist_en.ipynb and after training, I wanted to sample an image.

What I was essentially asking is - Whether the sample() function returns an image or a vector in latent space? If it is a vector in latent space then it needs to be passed through flow.bijector.inverse() function to get an image.

The other question I had was - There is gaussianisation in latent space on one half of the split from the output of the final step of flow in a level in the official implementation. Where that is happening in this implementation?

I read your TFGENZOO implementation and I started implementing my FLOW network like that but I got stuck and started working with this implementation of yours.

If you want to implement what you say with TFP, please let me know with detail requirements (e.g. mnist, cifer-10, glow, etc...).

I want to train the GLOW network on either of the dataset (MNIST or CIFAR-10) and after training, I want to be able to sample a new image for the given class.

[MokkeMeguru]

OK, @fused-byte

sample()'s algorithm is

1. sample from the latent distribution (like gaussian).
2. throw model's black box
3. returns the generated image

   inverse function to get an image, right?

   No (I think) see. https://github.com/tensorflow/probability/blob/master/tensorflow_probability/python/distributions/transformed_distribution.py#L343

they call forward function (and it's the horrible bug generator of this framework)

WARN: this TransformedDistribution's forward may mean our inverse, and their inverse may mean our forward ...

I want to train the Glow network (with given class)

If you want to build glow without label (because it is the original),. please wait for a week... I'll now re-format my TFGENZOO's example.

I think you want to get the network to obtain https://github.com/MokkeMeguru/TFGENZOO#news-2020616 's result

[fused-byte]

@MokkeMeguru

I understand the working of the sample function.

Thank you for pointing out the possible difference in the inverse and forward implementation of tensorflow.

I think you want to get the network to obtain https://github.com/MokkeMeguru/TFGENZOO#news-2020616 's result Yes, I am looking for this.

I know you are implementing it in TFGENZOO but Is it at all possible to do with TFP?

[MokkeMeguru]

Why you want to run within TFP? TFGENZOO is based on the basic Tensorflow Model system, tf.keras.layers.Layer (TFP is based on tf.keras.Module). but, you may wrap tfp.bijector module into my base-class. TFGENZOO needs

1. forward function & log determinant function
2. inverse function & inverse log determinant function

however, sampling is more difficult than tfp. because our library doesn't help it (tfp's helper, TransformDistribution will prevent factor-out technique and conditional input) so, you will write these code (we recommend this process writes into your tf.keras.Model)

sample_size = 128
sample = tfd.Normal().sample([sample_size, 32, 32, 1])
out = sample
for flow_module in reversed(flow_modules):
    out, ildj = flow_module(out, inverse=True)
generated_image = out

[fused-byte]

No, I was involved in a project and we thought if we could generate some results without heavily altering the implementation. So I asked if it is at all possible.

I will check out the TFGENZOO implementation. Thanks for all the help. You can close the issue if you want.