code release

[zhang0557kui]

Hello, first of all, thank you for your achievements. May I ask when we can experience it，thank you.

[tvaranka]

Hello, first of all, thank you for your achievements. May I ask when we can experience it，thank you.

Thanks for your interest in the project. I am currently working on transforming the research code into a diffusers pipeline. I am hoping to have the inference part ready in a few weeks or so.

[tvaranka]

Code uploaded and tested with RoCM 5.6. Unfortunately not able to test with cuda at the moment. Happy to take any PRs if there are issues.

[johndpope]

thanks so much - any guidance on switching base models ? just any sd1.5 can be swapped in? if you had gpu compute - could a new model be upgraded to use sdxl?

[tvaranka]

thanks so much - any guidance on switching base models ? just any sd1.5 can be swapped in? if you had gpu compute - could a new model be upgraded to use sdxl?

As the fine-tuning has been performed on 2-1-base, these weights will only work for that model, hence other SD base models cannot be switched at the moment.

The model could be upgraded to SDXL. However, a potential limitation is the relatively low-resolution of the used training dataset, which may cause some blurriness in the results.

[johndpope]

i just ran the demo.py

diffusion_pytorch_model.safetensors:  99%|█████████████████████████████████████████████████▍| 3.43G/diffusion_pytorch_model.safetensors:  99%|█████████████████████████████████████████████████▋| 3.44G/diffusion_pytorch_model.safetensors: 100%|█████████████████████████████████████████████████▊| 3.45G/diffusion_pytorch_model.safetensors: 100%|█████████████████████████████████████████████████▉| 3.46G/diffusion_pytorch_model.safetensors: 100%|██████████████████████████████████████████████████| 3.46G/diffusion_pytorch_model.safetensors: 100%|██████████████████████████████████████████████████| 3.46G/3.46G [10:43<00:00, 5.07MB/s]
attn_processors.ckpt: 100%|██████████████████████████████████████| 102M/102M [00:24<00:00, 4.21MB/s]
/media/oem/12TB/fineface/fineface/fineface_pipeline.py:40: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  torch.load(hf_hub_download(fineface_repo_id, "attn_processors.ckpt"), map_location=self.device),
pytorch_lora_weights.safetensors: 100%|████████████████████████| 26.6M/26.6M [00:06<00:00, 4.36MB/s]
model_index.json: 100%|████████████████████████████████████████████| 543/543 [00:00<00:00, 2.62MB/s]
text_encoder/config.json: 100%|████████████████████████████████████| 613/613 [00:00<00:00, 2.97MB/s]
scheduler/scheduler_config.json: 100%|█████████████████████████████| 346/346 [00:00<00:00, 6.94MB/s]
tokenizer/special_tokens_map.json: 100%|███████████████████████████| 460/460 [00:00<00:00, 2.76MB/s]
tokenizer/tokenizer_config.json: 100%|█████████████████████████████| 807/807 [00:00<00:00, 2.92MB/s]
(…)ature_extractor/preprocessor_config.json: 100%|█████████████████| 342/342 [00:00<00:00, 1.97MB/s]
tokenizer/merges.txt: 100%|██████████████████████████████████████| 525k/525k [00:00<00:00, 3.16MB/s]
tokenizer/vocab.json: 100%|████████████████████████████████████| 1.06M/1.06M [00:00<00:00, 3.32MB/s]
vae/config.json: 100%|█████████████████████████████████████████████| 553/553 [00:00<00:00, 2.91MB/s]
diffusion_pytorch_model.safetensors: 100%|███████████████████████| 335M/335M [02:14<00:00, 2.48MB/s]
model.safetensors: 100%|███████████████████████████████████████| 1.36G/1.36G [05:24<00:00, 4.20MB/s]
Fetching 11 files: 100%|████████████████████████████████████████████| 11/11 [05:25<00:00, 29.60s/it]
Loading pipeline components...:   0%|                                         | 0/6 [00:00<?, ?it/s]/home/oem/miniconda3/envs/comfyui/lib/python3.11/site-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
  warnings.warn(
Loading pipeline components...: 100%|█████████████████████████████████| 6/6 [00:00<00:00, 21.53it/s]
au_encoder.ckpt: 100%|█████████████████████████████████████████| 4.91k/4.91k [00:00<00:00, 15.8MB/s]
/media/oem/12TB/fineface/fineface/fineface_pipeline.py:50: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  torch.load(hf_hub_download(fineface_repo_id, "au_encoder.ckpt"), map_location=self.device)
100%|███████████████████████████████████████████████████████████████| 50/50 [00:06<00:00,  8.02it/s

um... maybe it's missing vae?

[tvaranka]

Thanks for trying it out. I managed to run it on colab with CUDA and it seems to work for me. Perhaps there is an issue from your side? Could you check if you can run the standard text-to-image using just 2-1-base?

a closeup of a boy in a park {'AU1'_ 2.5, 'AU6'_ 2.5, 'AU12'_ 5}

[johndpope]

may help to prescribe conda env - this worked. I use later version of transformers - i wonder why it's breaking - there's not a lot of code here.

conda create -n fineface python==3.11 conda activate fineface

requirements.txt

one of these things is a must

torch
torchvision
diffusers==0.27.2
transformers==4.41.0
huggingface_hub==0.23.0
accelerate==0.28.0
peft==0.7.0
numpy==1.26.3

UPDATE

I do a pip install -r requirements.txt -U and only specify diffusers locked to 0.27.2

torch torchvision diffusers==0.27.2 transformers huggingface_hub accelerate peft numpy

fyi - Upgrading to diffusers-0.30.0 breaks your code.

Installing collected packages: diffusers
  Attempting uninstall: diffusers
    Found existing installation: diffusers 0.27.2
    Uninstalling diffusers-0.27.2:
      Successfully uninstalled diffusers-0.27.2
Successfully installed diffusers-0.30.0

-  i recreate that picture of grass.

[tvaranka]

Great! The libraries are changing so fast, sometimes even small changes can break things. I will keep the requirements with those versions to ensure it works in the future

[johndpope]

its just newbies will break things if theres no conda env.