Intuition behind `nerf_init`?

[KokeCacao]

Hi there,

Thank you for providing a good reference code of prolific dreamers for the community. May I ask what the idea behind args.nerf_init is for sds latent-nerf?

    if args.nerf_init and args.rgb_as_latents and not args.use_mlp_particle:
        # current only support sds and experimental for only rgb_as_latents==True
        assert args.generation_mode == 'sds'
        with torch.no_grad():
            noise_pred = predict_noise0_diffuser(unet, particles, text_embeddings_vsd, t=999, guidance_scale=7.5, scheduler=scheduler)
        particles = scheduler.step(noise_pred, 999, particles).pred_original_sample

[yuanzhi-zhu]

Hi @KokeCacao ,

Good catch, this is from dreamtime https://arxiv.org/abs/2306.12422.

Best

[KokeCacao]

Hi @KokeCacao ,

Good catch, this is from dreamtime https://arxiv.org/abs/2306.12422.

Best

Thank you for answering. I have read their paper before. I don't recall such initialization for TP-SDS. May I ask which section does the code refer to? Or is it something you came up with? Does it generally work better?

[yuanzhi-zhu]

You can find it in Figure 4 :)

But I didn't make extensive experiments to test if it really helps to improve.

Thank you for answering. I have read their paper before. I don't recall such initialization for TP-SDS. May I ask which section does the code refer to? Or is it something you came up with? Does it generally work better?