how to train well on my pretrained model?

[guoyilin]

@luosiallen , great job. I do do latent consistency model distilling on my pretrained model according to your paper using ddim solver. the result is ok but blurry and step must be high(16 is ok , 4 is too bad) . I found the training is very tricky, especially lr and ema_rate, do you have the practical training guildance for distilling ? ?

[lonestar1990]

I have the same question, training details would be super helpful. Is the training pipeline will be open sourced as well? @luosiallen

[Redmond-AI]

Will the training code be published in the future? Even if it is not polished, it would be great to have.

[Redmond-AI]

@guoyilin would you mind putting your code in a repo? I would love to check it out, maybe I can help with quality.

[kelisiya]

+1

[howardgriffin]

Waiting for the training code

[wqdong8]

+1

[2120160608]

+1

[guoyilin]

https://github.com/huggingface/diffusers/blob/main/examples/consistency_distillation/train_lcm_distill_sd_wds.py , I found the code is release now. the result is ok but blurry and step must be high (hyperparameter influence, like ema rate and lr.)

[2120160608]

https://github.com/huggingface/diffusers/blob/main/examples/consistency_distillation/train_lcm_distill_sd_wds.py , I found the code is release now. the result is ok but blurry and step must be high (hyperparameter influence, like ema rate and lr.)

可不可以加个微信，我的601663546

[kelisiya]

https://github.com/huggingface/diffusers/blob/main/examples/consistency_distillation/train_lcm_distill_sd_wds.py , I found the code is release now. the result is ok but blurry and step must be high (hyperparameter influence, like ema rate and lr.)

When I run this code , i didn't read s3-laion dataset in aws , how to solve it ?

[PeiqinSun]

@guoyilin I also found the lr and ema-rate is very sensitive to the performance. Have any experiences to share with?

[kopyl]

Waiting for the training code as well.

Or any other methods to convert a trained model to an LCM one...

If you don't mind, i'd really appreciate if you could ping me in Telegram when someone posts it @kopyl

[MRG-DOT]

https://github.com/huggingface/diffusers/blob/main/examples/consistency_distillation/train_lcm_distill_sd_wds.py , I found the code is release now. the result is ok but blurry and step must be high (hyperparameter influence, like ema rate and lr.)

When I run this code , i didn't read s3-laion dataset in aws , how to solve it ?

Have you resolved it?

[MRG-DOT]

I have many issues when i use aws to load the data

[foocker]

https://github.com/huggingface/diffusers/blob/main/examples/consistency_distillation/train_lcm_distill_sd_wds.py , I found the code is release now. the result is ok but blurry and step must be high (hyperparameter influence, like ema rate and lr.)

When I run this code , i didn't read s3-laion dataset in aws , how to solve it ?

maybe u should have a visa card, and register aws, and the config aws on your local env.

[foocker]

Who can share the data in aliyun pan or baiduyun or google pan for chinese users? is the data have licence, we cant share it?

[MRG-DOT]

https://github.com/huggingface/diffusers/blob/main/examples/consistency_distillation/train_lcm_distill_sd_wds.py , I found the code is release now. the result is ok but blurry and step must be high (hyperparameter influence, like ema rate and lr.)

When I run this code , i didn't read s3-laion dataset in aws , how to solve it ?

maybe u should have a visa card, and register aws, and the config aws on your local env.

I have tried it,but also have some errors.

[akameswa]

1. Download data from: https://huggingface.co/datasets/ChristophSchuhmann/improved_aesthetics_6plus
2. Use img2dataset to convert into ‘webdataset’ format: https://github.com/rom1504/img2dataset

[foocker]

1. Download data from: https://huggingface.co/datasets/ChristophSchuhmann/improved_aesthetics_6plus

   2. Use img2dataset to convert into ‘webdataset’ format: https://github.com/rom1504/img2dataset
2. download it from hf.
3. but why img2datasets will download something again... the network is error again.

[akameswa]

You can use the downloaded path instead of the URL. Also, keep the specifications of thread_count and processes_count according to your CPU.

Attaching sample preprocessing procedure for your reference: Link.

[foocker]

You can use the downloaded path instead of the URL. Also, keep the specifications of thread_count and processes_count according to your CPU.

Attaching sample preprocessing procedure for your reference: Link.

confused. first

img2dataset --url_list dowload_from_hf_dir --input_format "parquet"\
        --url_col "URL" --caption_col "TEXT" --output_format webdataset\
        --output_folder /xx/data_webdataset --processes_count 16 --thread_count 32 --image_size 224 --resize_only_if_bigger --resize_mode=keep_ratio_largest \
        --save_additional_columns '["WIDTH","HEIGHT","punsafe","similarity"]' --enable_wandb True

and then

MODEL_DIR="/data/modelscope_cache/AI-ModelScope/stable-diffusion-xl-base-1.0"
OUTPUT_DIR="../sdxl_lora"
CUDA_VISIBLE_DEVICES=1 python train_lcm_distill_lora_sdxl_wds.py \
    --pretrained_teacher_model=$MODEL_DIR \
    --pretrained_vae_model_name_or_path=$MODEL_DIR/vae \
    --output_dir=$OUTPUT_DIR \
    --mixed_precision=fp16 \
    --resolution=1024 \
    --lora_rank=64 \
    --learning_rate=1e-6 --loss_type="huber" --use_fix_crop_and_size --adam_weight_decay=0.0 \
    --max_train_steps=1000 \
    --max_train_samples=4000000 \
    --dataloader_num_workers=8 \
    --train_shards_path_or_url="/xx/data_webdataset" \
    --validation_steps=200 \
    --checkpointing_steps=200 --checkpoints_total_limit=10 \
    --train_batch_size=4 \
    --gradient_checkpointing --enable_xformers_memory_efficient_attention \
    --gradient_accumulation_steps=1 \
    --use_8bit_adam \
    --resume_from_checkpoint=latest \
    --report_to=wandb \
    --seed=453645634

but why

UserWarning: IsADirectoryError(21, 'Is a directory', '/xx/data_webdataset')

and it seems stoped, not running.

[akameswa]

--train_shards_path_or_url="/xx/data_webdataset/{00000..01210}.tar"

[RunpuWei]

1. Download data from: https://huggingface.co/datasets/ChristophSchuhmann/improved_aesthetics_6plus
2. Use img2dataset to convert into ‘webdataset’ format: https://github.com/rom1504/img2dataset

Hello, I cannot download the LAION-Aesthetics V2 6.5+ dataset right now, and the links I can find are all 404😭. Is there any other place I can download it, thanks!

[akameswa]

Check this out: RobinWZQe/improved_aesthetics_6.5plus

[LILI1KA]

You can use the downloaded path instead of the URL. Also, keep the specifications of thread_count and processes_count according to your CPU. Attaching sample preprocessing procedure for your reference: Link.

confused. first

img2dataset --url_list dowload_from_hf_dir --input_format "parquet"\
        --url_col "URL" --caption_col "TEXT" --output_format webdataset\
        --output_folder /xx/data_webdataset --processes_count 16 --thread_count 32 --image_size 224 --resize_only_if_bigger --resize_mode=keep_ratio_largest \
        --save_additional_columns '["WIDTH","HEIGHT","punsafe","similarity"]' --enable_wandb True

and then

MODEL_DIR="/data/modelscope_cache/AI-ModelScope/stable-diffusion-xl-base-1.0"
OUTPUT_DIR="../sdxl_lora"
CUDA_VISIBLE_DEVICES=1 python train_lcm_distill_lora_sdxl_wds.py \
    --pretrained_teacher_model=$MODEL_DIR \
    --pretrained_vae_model_name_or_path=$MODEL_DIR/vae \
    --output_dir=$OUTPUT_DIR \
    --mixed_precision=fp16 \
    --resolution=1024 \
    --lora_rank=64 \
    --learning_rate=1e-6 --loss_type="huber" --use_fix_crop_and_size --adam_weight_decay=0.0 \
    --max_train_steps=1000 \
    --max_train_samples=4000000 \
    --dataloader_num_workers=8 \
    --train_shards_path_or_url="/xx/data_webdataset" \
    --validation_steps=200 \
    --checkpointing_steps=200 --checkpoints_total_limit=10 \
    --train_batch_size=4 \
    --gradient_checkpointing --enable_xformers_memory_efficient_attention \
    --gradient_accumulation_steps=1 \
    --use_8bit_adam \
    --resume_from_checkpoint=latest \
    --report_to=wandb \
    --seed=453645634

but why

UserWarning: IsADirectoryError(21, 'Is a directory', '/xx/data_webdataset')

and it seems stoped, not running.

I followed what's done here and used --train_shards_path_or_url="/xx/data_webdataset/{00000..01210}.tar".

But now I'm getting this error when doing training: /python3.11/site-packages/webdataset/handlers.py:34: UserWarning: ValueError("didn't find ['text', 'txt', 'caption'] in ['key', 'url', 'jpg', 'json']"). (I added all columns 'URL', 'TEXT', 'WIDTH', 'HEIGHT', 'similarity', 'punsafe', 'pwatermark', 'AESTHETIC_SCORE', 'hash', 'index_level_0' into the parquet file)

Has anyone experienced similar situation?