denizyuret
commented
2 years ago Dear Barışcan, could you try the following:
- Try your model without the batchnorm operations.
- Try your model with Array and/or CuArray instead of KnetArray for array type.
- Send me a minimal working example https://en.wikipedia.org/wiki/Minimal_working_example: complete source file or notebook which I can directly run to get the error. Your explanations above are good, but I cannot run the code.
Here is an mwe that I tried with the same logic, but it does not reproduce the error:
using AutoGrad, Random, CUDA, Knet
ZDIM=50
XDIM=100
BATCH=10
atype = Array{Float64}
target = atype(randn(XDIM))
w = Param(atype(randn(XDIM,ZDIM)))
z0 = Param(atype(zeros(ZDIM)))
decoder(z) = w*z
qloss(x,y) = sum(abs2, x .- y)
function loss(x)
d = @diff qloss(x, decoder(z0))
z = -grad(d,z0)
qloss(x, decoder(z))
end
J = @diff loss(target)
grad(J, w) |> summary |> println 
BariscanBozkurt
Hello.
I am currently using Julia Versio 1.6.3 on a Platform "OS: Linux (x86_64-pc-linux-gnu) CPU: Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz, GPU : CuDevice(0): Tesla T4". I am trying to implement a variational autoencoder called Gradient Origin Networks (GONs). GONs are introduced as a generative model which does not require encoders or hypernetworks. Assume Variational GON model called F. First, a zero vector z_0 is passed through the model F, and then the latent vector initialized as the minus gradient of the loss with respect to this zero vector. Therefore, the latent space is determined by only one gradient step. Let us call this latent vector as z. Then, the network parameters are optimized by using the loss with the reconstruction F(z).
I am currently performing my experiments on MNIST dataset where I linearly interpolated the images to the size of 32x32. The decoding and reparametrization functions are as follows. theta is a vector of model weights.
For the loss, it is used binary cross-entropy and KL-divergence. The code is given as follows.
Since there are two steps for GON (1-) Use the gradient w.r.t. origin to determine the latent space z, 2-) Use latent space for reconstruction) I need to track all the gradient w.r.t. model weights from the steps (1) and (2). Therefore, I wrote the following decoding function and loss function for training purpose.
However, I am not able to take the gradient of the " loss_train(theta, x)" function. I am getting the following error when I use the @diff macro of AutoGrad package. How can I handle to train this model which requires a second order derivative (I need the derivative of the function decode_train)? To reproduce this result, you can run the following notebook : https://github.com/BariscanBozkurt/Gradient-Origin-Networks/blob/main/GON_Implementation_Issue.ipynb My code:
@diff loss_train(theta, x)The error is: