search

orenkatzir / nfsd

Noise Free Score Distillation
67 stars 1 forks source link

Question: Have you tested your methods on stabledreamfusion? #2

Open YuyangYin opened 8 months ago

YuyangYin commented 8 months ago

Nice work! I try to implement your work and found it works well on 2d generation. The results are as follow: 2dNFSD

But when I test stabledreamfusion(https://github.com/ashawkey/stable-dreamfusion) with NFSD, I cann't get the same results in papers. The default sd version is 1.4 in stabledreamfusion. The results are as follow: 3dNFSD My core codes are as follow:

noise = torch.randn_like(latents)
  latents_noisy = self.scheduler.add_noise(latents, noise, t)
  latent_model_input = torch.cat([latents_noisy] * 3)
  tt = torch.cat([t] * 3)
  noise_pred = self.unet(latent_model_input, tt, encoder_hidden_states=text_embeddings).sample
  noise_pred_uncond, noise_pred_y, noise_pred_neg = noise_pred.chunk(3)
  noise_pred= (noise_pred_y- noise_pred_uncond)*guidance_scale  #delta_c

  if t<200:    #delta_d
      #epsilon(zt,none,t) 
      noise_pred +=noise_pred_uncond
  else:
      #epsilon(zt,none,t)-epsilon(zt,neg,t) 
      noise_pred +=(noise_pred_uncond-noise_pred_neg)
w = (1 - self.alphas[t])
grad = grad_scale * w[:, None, None, None] * (noise_pred)

I would like to ask if there are any errors in my implementation and have you tested your methods on stabledreamfusion?

orenkatzir commented 8 months ago

Hi, so the snippet of the code looks OK but indeed we haven't tested it on stable-dreamfusion and we have used SD2.1.base in our implementation. Can you try running with SD2.1.base? In addition can you try multiplying (noise_pred_uncond-noise_pred_neg) by some coefficient (say 3), just to check if any improvement is introduced?

zcai0612 commented 7 months ago

I am also reproducing NFSD for 3D generation now, can I have a comment with you? My email is zcai701@connect.hkust-gz.edu.cn