I fine-tuned the same yannlecun image as yours, and the generated results are different from yours. I used the pre-trained model e4t-diffusion-ffhq-celebahq-v1 you released. Why are the results different?

[wangzhaolei]

The image generated by inference is as follows

CUDA_VISIBLE_DEVICES=1,3 accelerate launch --config_file 2_gpu.json tuning_e4t.py \ --pretrained_model_name_or_path="e4t-diffusion-ffhq-celebahq-v1" \ --prompt_template="a photo of {placeholder_token}" \ --reg_lambda=0.1 \ --gradient_accumulation_steps=8 \ --output_dir="tune_yann-lecun" \ --train_image_path="yann-lecun.png" \ --resolution=512 \ --train_batch_size=1 \ --learning_rate=1e-6 --scale_lr \ --max_train_steps=60 \ --checkpointing_steps=30 \ --mixed_precision="fp16" \ --enable_xformers_memory_efficient_attention

And I didn’t use data augmentation

[zetyquickly]

batch size is crucial, that what's the paper claims

[omergra]

Are you able to use the checkpoint? When using the checkpoint path, I am getting errors: Downloading (…)lve/main/config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 374/374 [00:00<00:00, 3.04MB/s] Traceback (most recent call last): File "/home/ubuntu/e4t-diffusion/tuning_e4t.py", line 365, in main() File "/home/ubuntu/e4t-diffusion/tuning_e4t.py", line 103, in main unet = load_e4t_unet( ^^^^^^^^^^^^^^ File "/home/ubuntu/e4t-diffusion/e4t/utils.py", line 99, in load_e4t_unet assert ckpt_path in MODELS, f"Choose from {list(MODELS.keys())}" ^^^^^^^^^^^^^^^^^^^ AssertionError: Choose from ['e4t-diffusion-ffhq-celebahq-v1']

When downloading the files separately, and providing the folder, I'm getting the error: File "/home/ubuntu/e4t-diffusion/tuning_e4t.py", line 103, in main unet = load_e4t_unet( ^^^^^^^^^^^^^^ File "/home/ubuntu/e4t-diffusion/e4t/utils.py", line 120, in load_e4t_unet unet = UNet2DConditionModel(*unet.config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ubuntu/diffusers/src/diffusers/configuration_utils.py", line 607, in inner_init init(self, args, **init_kwargs) TypeError: UNet2DConditionModel.init() got an unexpected keyword argument 'encoder_hid_dim'

[zetyquickly]

https://huggingface.co/mkshing/e4t-diffusion-ffhq-celebahq-v1

Have you downloaded it?

[Parisa-Bagherzadeh]

Are you able to use the checkpoint? When using the checkpoint path, I am getting errors: Downloading (…)lve/main/config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 374/374 [00:00<00:00, 3.04MB/s] Traceback (most recent call last): File "/home/ubuntu/e4t-diffusion/tuning_e4t.py", line 365, in main() File "/home/ubuntu/e4t-diffusion/tuning_e4t.py", line 103, in main unet = load_e4t_unet( ^^^^^^^^^^^^^^ File "/home/ubuntu/e4t-diffusion/e4t/utils.py", line 99, in load_e4t_unet assert ckpt_path in MODELS, f"Choose from {list(MODELS.keys())}" ^^^^^^^^^^^^^^^^^^^ AssertionError: Choose from ['e4t-diffusion-ffhq-celebahq-v1']

When downloading the files separately, and providing the folder, I'm getting the error: File "/home/ubuntu/e4t-diffusion/tuning_e4t.py", line 103, in main unet = load_e4t_unet( ^^^^^^^^^^^^^^ File "/home/ubuntu/e4t-diffusion/e4t/utils.py", line 120, in load_e4t_unet unet = UNet2DConditionModel(*unet.config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ubuntu/diffusers/src/diffusers/configuration_utils.py", line 607, in inner_init init(self, args, init_kwargs) TypeError: UNet2DConditionModel.init**() got an unexpected keyword argument 'encoder_hid_dim'

I have the same problem , did you find out what's the problem?

[Parisa-Bagherzadeh]

https://huggingface.co/mkshing/e4t-diffusion-ffhq-celebahq-v1

Have you downloaded it?

how can i get the unet.pt file?

[Parisa-Bagherzadeh]

https://huggingface.co/mkshing/e4t-diffusion-ffhq-celebahq-v1

Have you downloaded it?

I've downloaded it but the unet.pt is not in this link , how can i get this file?

[mkshing]

@Parisa-Bagherzadeh

I have the same problem , did you find out what's the problem?

I think this is because your diffusers version is incompatible. Please make sure to download v0.14.0. (see requirements.txt)

I've downloaded it but the unet.pt is not in this link , how can i get this file?

I don't know where unet.pt comes from but e4t-diffusion does use only diffusers weigths (https://huggingface.co/CompVis/stable-diffusion-v1-4).

[h3clikejava]

I run

python inference.py \
  --pretrained_model_name_or_path "./models" \
  --prompt "Times square in the style of *s" \
  --num_images_per_prompt 3 \
  --scheduler_type "ddim" \
  --image_path_or_url "./image/aaa.jpg" \
  --num_inference_steps 50 \
  --guidance_scale 7.5

get:

device: cuda
a: False /home/h3c/Documents/e4t-diffusion/models/unet.pt
Traceback (most recent call last):
  File "/home/h3c/Documents/e4t-diffusion/inference.py", line 157, in <module>
    main()
  File "/home/h3c/Documents/e4t-diffusion/inference.py", line 83, in main
    unet = load_e4t_unet(
  File "/home/h3c/Documents/e4t-diffusion/e4t/utils.py", line 100, in load_e4t_unet
    assert ckpt_path in MODELS, f"Choose from {list(MODELS.keys())}"
AssertionError: Choose from ['e4t-diffusion-ffhq-celebahq-v1']

Cause ./models/unet.pt not found. How Can i get the unet.pt file?

I'll download config.json, encoder.pt, weight_offsets.pt from huggingface in models folder. But don't have unet.pt.