westfish commented 8 months ago

新增模型Hotshot-XL

任务描述

任务背景

Hotshot-XL是一个text-to-GIF模型，可以与SDXL、ControlNet兼容工作。
完成步骤
1. 模型、代码已经开源。
2. 根据开源代码进行网络结构转换，训练、推理，代码链接。
3. 前向推理和训练跑通即可。
4. 提交代码PR到PaddleMIX。

参考PR：https://github.com/PaddlePaddle/PaddleMIX/pull/401

提交内容：

co63oc commented 7 months ago

Hotshot-XL 版本要求是 diffusers==0.21.4，ppdiffusers develop分支是0.19.4，那是要用upgrade_ppdiffusers0240分支ppdiffusers吗

westfish commented 7 months ago

是的目前先基于upgrade_ppdiffusers0240开发 ppdiffusers0.24.0会在最近一两周发版

co63oc commented 7 months ago

模型文件较大 14G，是不是能转为paddle格式下载使用，text_encoder和tokenizer转换规则和stable diffusion类似，hsxl_temporal_layers 如果转换规则不同可以本地再修改 https://huggingface.co/hotshotco/Hotshot-XL/

westfish commented 7 months ago

可以呀你那边可以转吗我帮你上传到bos

co63oc commented 7 months ago

@westfish

模型文件 https://huggingface.co/co63oc/hotshotxl/tree/main/ hotshot_output 主模型 controlnet_depth depth模型 controlnet_canny canny模型

文件MD5

4ce6c3a202b3f09c16e20fe0e980dff4  hotshot_output/scheduler/scheduler_config.json
aa6b3cff8d3a1a8aac5e559078ea89ef  hotshot_output/text_encoder/config.json
c9edb592edf1ea5b8217330271b65f3f  hotshot_output/text_encoder/model.safetensors
5bb3a717878dd8da03c442dc145eb3f5  hotshot_output/text_encoder_2/config.json
d57475443511011ad42ace27b155868a  hotshot_output/text_encoder_2/model.safetensors
d2bc10797e70c920c14c69729ecdab34  hotshot_output/tokenizer/merges.txt
c05624787db53245d5a2241c746b874c  hotshot_output/tokenizer/special_tokens_map.json
8a6faaf8756b741480a66d989f811188  hotshot_output/tokenizer/tokenizer_config.json
a3933521163a0e76cee6094f1d72a9cc  hotshot_output/tokenizer/vocab.json
d2bc10797e70c920c14c69729ecdab34  hotshot_output/tokenizer_2/merges.txt
e00a54896b39f2016de02da8bd2d8262  hotshot_output/tokenizer_2/special_tokens_map.json
4e37b134d337fb7714d09f0adfbb722d  hotshot_output/tokenizer_2/tokenizer_config.json
a3933521163a0e76cee6094f1d72a9cc  hotshot_output/tokenizer_2/vocab.json
486063173e391354422f39c671e138ef  hotshot_output/unet/config.json
4d419b79192500788b9abb054a5cff5a  hotshot_output/unet/diffusion_paddle_model-00001-of-00002.safetensors
69326c511b5d645a5b8e5ec75fb3e048  hotshot_output/unet/diffusion_paddle_model-00002-of-00002.safetensors
a0c481c7eb23feebe7d3d5bb9ee804c0  hotshot_output/unet/diffusion_paddle_model.safetensors.index.json
788437f8eb005679f7af78a21cb05ad2  hotshot_output/vae/config.json
8c3607c69cbb32e109282516c3eef46c  hotshot_output/vae/diffusion_paddle_model.safetensors
703efefcf1da148538f2ca58aa3d2dce  controlnet_canny/config.json
53fba6b47328313794a8dc698fe7d8b2  controlnet_canny/diffusion_paddle_model.safetensors
027cdc84d0ed3ed8eebaed74bfe109c1  controlnet_depth/config.json
0ad635132307d931450d0f4d90507711  controlnet_depth/diffusion_paddle_model.safetensors

PR https://github.com/PaddlePaddle/PaddleMIX/pull/449

westfish commented 7 months ago

模型已上传bos，id为co63oc/hotshotxl

co63oc commented 7 months ago

@westfish https://bj.bcebos.com/paddlenlp/models/community/co63oc/hotshotxl/controlnet_depth/diffusion_paddle_model.safetensors 怎么这个路径不存在，这个是controlnet的模型

https://huggingface.co/co63oc/hotshotxl/tree/main/controlnet_depth