Open zRzRzRzRzRzRzR opened 2 months ago
5b image to video please! I2V would be lovely!
The 3D VAE model consumes significantly more memory compared to diffusion models, which is severely limiting the batch size for fine-tuning. Any suggestions or optimizations to reduce memory usage would be greatly appreciated.
The 3D VAE model consumes significantly more memory compared to diffusion models, which is severely limiting the batch size for fine-tuning. Any suggestions or optimizations to reduce memory usage would be greatly appreciated.
You make a very good point. We will work together with the Diffusers team to modify the fake quantization (fakecp) process in the VAE section to optimize it for lower memory usage. Please give us some time, as we will collaborate with the Diffusers team to develop a version of the model that is fine-tuned specifically for Diffusers, which is expected to save a significant amount of memory.
First of all, thank you for your excellent work!
The dataset format used SAT way for fine tuning & full training be the same as the format that will be used for fine-tuning Diffusers version models?
+ wrong discord link
We are currently completing several tasks
Work that has been completed
When will vertical video generation be supported?
When will vertical video generation be supported?
The current model cannot generate vertical videos, such as 480x720 resolution. We are working on fine-tuning to reach this capability, but it’s still in progress. Once we have any updates, we will share them as soon as possible.
Two related issues working now:
Diffusers supports CogVideoX-I2V. This PR has been merged, but the patch has not yet been released.
Fine-tuning the Diffusers version of CogVideoX-2B T2V without using the SAT model, running directly under the Diffusers framework. It can run on a single A100 GPU. This PR is still under debugging. I am working with members of the Diffusers team to attempt fine-tuning the CogVideoX-5B and I2V models. We will provide a small dataset (a few dozen samples) for this PR, which is sufficient for LoRA fine-tuning CogVideoX.
Many thanks to @a-r-r-o-w for the help with these two tasks!
when will CogVideoX-2B-I2V be released?
@zRzRzRzRzRzRzR Many thanks to you and the team! I know fine-tuning vae is not very useful, but I'm curious is there any way I can just fine-tuning decoder part?
Our publicly available fine-tuning code is for the fine-tuning of the transformers part, not for vae. We indeed have not updated the training and fine-tuning parts of vae (because I have not received the corresponding permissions either). Additionally, fine-tuning vae alone seems to have little significance for the overall model effect. If you want to try fine-tuning the diffusers model, all fine-tuning of the transformers module is already in dev, currently it is lora, and it is expected to implement SFT by early October.
Our publicly available fine-tuning code is for the fine-tuning of the transformers part, not for vae. We indeed have not updated the training and fine-tuning parts of vae (because I have not received the corresponding permissions either). Additionally, fine-tuning vae alone seems to have little significance for the overall model effect. If you want to try fine-tuning the diffusers model, all fine-tuning of the transformers module is already in dev, currently it is lora, and it is expected to implement SFT by early October.
@zRzRzRzRzRzRzR thank you but actually I'm asking how to fine-tuning VAE decoder, any advice?
Hi @zRzRzRzRzRzRzR, what's your plan about diffuser I2V lora fine-tune code? Thanks!
@zRzRzRzRzRzRzR
Thank you for your great works!
I would like to covert a full-finetuned 2b model weight in sat into a model weight in diffusers.
My full-finetuned model weight in sat is approx. 22GB.
Therefore, I do not convert the model by your conversion code: python ../tools/convert_weight_sat2hf.py
.
This model weight may includes transformer and optimizer weight and so on.
Therefore, I tried to extract transformer from the model weight only.
However, something went on.
How can I do it?
I could convert the full-finetuned weight into the diffusers weight. We need extract limited keys by the script:
import torch
# ファイルAとファイルBのパス
file_a_path = 'file_a.pt'
file_b_path = 'file_b.pt'
file_c_path = 'file_c.pt'
# ファイルAをロードしてキーを取得
file_a = torch.load(file_a_path)
# ファイルBをロード
file_b = torch.load(file_b_path)
# 新しいファイルCに入れるデータを保存する辞書
file_c_data = {}
# 再帰的に辞書のキーに基づいて値を取り出す関数
def extract_values_from_b(a_dict, b_dict):
result = {}
for key, value in a_dict.items():
if isinstance(value, dict): # 値がさらに辞書の場合は再帰処理
if key in b_dict and isinstance(b_dict[key], dict):
result[key] = extract_values_from_b(value, b_dict[key])
else:
print(f"Key '{key}' not found or not a dict in file B")
else:
if key in b_dict:
result[key] = b_dict[key]
else:
print(f"Key '{key}' not found in file B")
return result
# ファイルAの構造に基づいてファイルBから値を取得
file_c_data = extract_values_from_b(file_a, file_b)
# 新しいファイルCに保存
torch.save(file_c_data, file_c_path)
print(f"New .pt file saved at: {file_c_path}")
Please feel free to ask me for further detail.
video outpainting fintune scripts
I want to enable multiple gpus, but it go wrong
# 3. Enable CPU offload for the model.
# turn off if you have multiple GPUs or enough GPU memory(such as H100) and it will cost less time in inference
# and enable to("cuda")
pipe.to("cuda")
# pipe.enable_sequential_cpu_offload()
pipe.vae.enable_slicing()
pipe.vae.enable_tiling()
$ python cli_demo.py --prompt "A girl riding a bike." --model_path THUDM/CogVideoX-5b --generate_type "t2v"
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████| 2/2 [00:02<00:00, 1.13s/it]
Loading pipeline components...: 100%|████████████████████████████████████████████████████████████████████████████| 5/5 [00:06<00:00, 1.37s/it]
Traceback (most recent call last):
File "/home/ubuntu/gamehub/CogVideo/inference/cli_demo.py", line 177, in enable_sequential_cpu_offload
, but are now attempting to move the pipeline to GPU. This is not compatible with offloading. Please, move your pipeline .to('cpu')
or consider removing the move altogether if you use sequential offloading.
Tasks that have been identified and scheduled:
已经明确并排期的任务:
如果你有更多诉求,欢迎在这里提出