search

IDEA-CCNL / Fengshenbang-LM

Fengshenbang-LM(封神榜大模型)是IDEA研究院认知计算与自然语言研究中心主导的大模型开源体系，成为中文AIGC和认知智能的基础设施。
Apache License 2.0
4.01k stars 376 forks source link

[Done] Add checkpoint and webui code for IDEA-CCNL/Taiyi-Stable-Diffusion-1B-Chinese-v0.1 #186

Closed ganzhiruyi closed 1 year ago

ganzhiruyi commented 1 year ago

I have writen a readme , everyone can configuration Taiyi-Stable-Diffusion-1B-Chinese-v0.1 in webui step by step. https://github.com/IDEA-CCNL/stable-diffusion-webui/blob/master/README.md

We use the script https://github.com/huggingface/diffusers/blob/main/scripts/convert_diffusers_to_original_stable_diffusion.py to covert our model to a Stable Diffusion checkpoint, you can find the ckpt file in https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-1B-Chinese-v0.1/tree/main. For stable-diffusion-webui, because our model use roberta as text encoder and it not support yet, we clone the stable-diffusion-webui and update the code to support IDEA-CCNL/Taiyi-Stable-Diffusion-1B-Chinese-v0.1, you can find our updates from https://github.com/IDEA-CCNL/stable-diffusion-webui/commit/61ece0cec1097ab8f5e2b52c8d340ca203c5917b

chenkai-666 commented 1 year ago

I put Taiyi's ckpt in the webui's models/Table diffusion directory, and updated the three configuration files according to your instructions. But the generated image is not like the result of Taiyi, but more like the result of the original stable-diffusion model, I wonder where is the error, thanks!

ganzhiruyi commented 1 year ago

First, if you load our model correctly, and type in chinese prompt, the model can perform correct image, howerver the original stable-diffusion can't solve chinese yet.

chenkai-666 commented 1 year ago

首先，如果你正确加载我们的模型，并输入中文提示，模型可以执行正确的图像，而原来的稳定扩散还不能解决中文问题。 You are right! I removed all the model files in the stable-diffusion directory, leaving only the taiyi model, and tested it with Chinese many times and got very good results, thank you for your reply, and excellent work

lizekui commented 1 year ago

How can I solve this problem, thanks ! I put the model into ckpt folder, and replace the hijack code. can @ganzhiruyi help me?

OSError: Can't load tokenizer for 'IDEA-CCNL/Taiyi-Stable-Diffusion-1B-Chinese-v0.1/'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'IDEA-CCNL/Taiyi-Stable-Diffusion-1B-Chinese-v0.1/' is the correct path to a directory containing all relevant files for a BertTokenizer tokenizer.

ganzhiruyi commented 1 year ago

@lizekui It seems like network or transformers version problem, which lead to BertTokenizer can't find the tokenizer files. You can git clone the model and change the 'version' to your local model path like this. https://github.com/IDEA-CCNL/stable-diffusion-webui/blob/master/repositories/stable-diffusion/configs/stable-diffusion/v1-inference.yaml#L75 https://github.com/IDEA-CCNL/stable-diffusion-webui/blob/master/repositories/stable-diffusion/ldm/modules/encoders/modules.py#L170

ganzhiruyi commented 1 year ago

We use the script https://github.com/huggingface/diffusers/blob/main/scripts/convert_diffusers_to_original_stable_diffusion.py to covert our model to a Stable Diffusion checkpoint, you can find the ckpt file in https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-1B-Chinese-v0.1/tree/main. For stable-diffusion-webui, because our model use roberta as text encoder and it not support yet, we clone the stable-diffusion-webui and update the code to support IDEA-CCNL/Taiyi-Stable-Diffusion-1B-Chinese-v0.1, you can find our updates from IDEA-CCNL/stable-diffusion-webui@61ece0c

If you do everything right, you can use Taiyi-Stable-Diffusion-1B-Chinese-v0.1 in webui, such as text2img and img2img inpaint, enjoy it!

image image
lizekui commented 1 year ago

Oh dear @ganzhiruyi , Thanks a lot!

I replace the path of version property in https://github.com/IDEA-CCNL/stable-diffusion-webui/blob/master/repositories/stable-diffusion/configs/stable-diffusion/v1-inference.yaml#L75 and value is /stable-diffusion-webui/models/Stable-diffusion/Taiyi-Stable-Diffusion-1B-Chinese-v0.1/ which is whole filefolder download from huggingface. And webui repo is download from IDEA fork version.

folder structure is below:

[root@0304-ais Taiyi-Stable-Diffusion-1B-Chinese-v0.1]# ll -h
总用量 3.9G
drwxr-xr-x 2 38 11月 10 12:32 feature_extractor
-rw-r--r-- 1  539 11月 10 12:32 model_index.json
-rw-r--r-- 1 6.9K 11月 10 12:32 README.md
drwxr-xr-x 2  187 11月 10 12:32 result_examples
drwxr-xr-x 2   50 11月 10 12:32 safety_checker
drwxr-xr-x 2   35 11月 10 12:32 scheduler
-rw-r--r-- 1 3.9G 11月 10 12:33 Taiyi-Stable-Diffusion-1B-Chinese-v0.1.ckpt
drwxr-xr-x 2   50 11月 10 12:32 text_encoder
drwxr-xr-x 2   83 11月 10 12:32 tokenizer
drwxr-xr-x 2   60 11月 10 12:32 unet
drwxr-xr-x 2   60 11月 10 12:33 vae

but when I choice the Taiyi Model in the webui, I met the following bug log :

webui-docker-auto-1  | Running on local URL:  http://0.0.0.0:7860
webui-docker-auto-1  | 
webui-docker-auto-1  | To create a public link, set `share=True` in `launch()`.
webui-docker-auto-1  | Traceback (most recent call last):
webui-docker-auto-1  |   File "/usr/local/lib/python3.10/site-packages/gradio/routes.py", line 275, in run_predict
webui-docker-auto-1  |     output = await app.blocks.process_api(
webui-docker-auto-1  |   File "/usr/local/lib/python3.10/site-packages/gradio/blocks.py", line 785, in process_api
webui-docker-auto-1  |     result = await self.call_function(fn_index, inputs, iterator)
webui-docker-auto-1  |   File "/usr/local/lib/python3.10/site-packages/gradio/blocks.py", line 694, in call_function
webui-docker-auto-1  |     prediction = await anyio.to_thread.run_sync(
webui-docker-auto-1  |   File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
webui-docker-auto-1  |     return await get_asynclib().run_sync_in_worker_thread(
webui-docker-auto-1  |   File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
webui-docker-auto-1  |     return await future
webui-docker-auto-1  |   File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
webui-docker-auto-1  |     result = context.run(func, *args)
webui-docker-auto-1  |   File "/stable-diffusion-webui/modules/ui.py", line 1636, in <lambda>
webui-docker-auto-1  |     fn=lambda value, k=k: run_settings_single(value, key=k),
webui-docker-auto-1  |   File "/stable-diffusion-webui/modules/ui.py", line 1478, in run_settings_single
webui-docker-auto-1  |     opts.data_labels[key].onchange()
webui-docker-auto-1  |   File "/stable-diffusion-webui/repositories/stable-diffusion/../../webui.py", line 41, in f
webui-docker-auto-1  |     res = func(*args, **kwargs)
webui-docker-auto-1  |   File "/stable-diffusion-webui/repositories/stable-diffusion/../../webui.py", line 83, in <lambda>
webui-docker-auto-1  |     shared.opts.onchange("sd_model_checkpoint", wrap_queued_call(lambda: modules.sd_models.reload_model_weights()))
webui-docker-auto-1  |   File "/stable-diffusion-webui/modules/sd_models.py", line 286, in reload_model_weights
webui-docker-auto-1  |     sd_hijack.model_hijack.undo_hijack(sd_model)
webui-docker-auto-1  |   File "/stable-diffusion-webui/modules/sd_hijack.py", line 105, in undo_hijack
webui-docker-auto-1  |     if type(model_embeddings.word_embeddings) == EmbeddingsWithFixes:
webui-docker-auto-1  |   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1207, in __getattr__
webui-docker-auto-1  |     raise AttributeError("'{}' object has no attribute '{}'".format(
webui-docker-auto-1  | AttributeError: 'CLIPTextEmbeddings' object has no attribute 'word_embeddings'

meanwhile, the webui's choice text is not your screenshot 'xxx.ckpt', but show 'Taiyi-Stable-Diffusion-1B-Chinese-v0.1/xxx.ckpt',

Any wrong with the path?

gravitywp commented 1 year ago

@lizekui

only keep Taiyi-Stable-Diffusion-1B-Chinese-v0.1/xxx.ckpt model , and restart webui, works for me.

ganzhiruyi commented 1 year ago

I have writen a readme , everyone can configuration Taiyi-Stable-Diffusion-1B-Chinese-v0.1 in webui step by step. https://github.com/IDEA-CCNL/stable-diffusion-webui/blob/master/README.md

fingerx commented 1 year ago

I have writen a readme , everyone can configuration Taiyi-Stable-Diffusion-1B-Chinese-v0.1 in webui step by step. https://github.com/IDEA-CCNL/stable-diffusion-webui/blob/master/README.md

thanks for job but follow this step in win11 show errors:

image

@ganzhiruyi

sherlockeded commented 1 year ago

Thank you for your great work. I am wandering if you can add support for Taiyi-Stable-Diffusion-1B-Chinese-EN-v0.1 .

I download Taiyi-Stable-Diffusion-1B-Chinese-EN-v0.1, covert the bin file of Taiyi-Stable-Diffusion-1B-Chinese-EN-v0.1 to ckpt and load it into webui. However I got following error: 

To create a public link, set `share=True` in `launch()`.
Traceback (most recent call last):
  File "/data/wzy/chinese_stable-diffusion-webui/venv/lib/python3.8/site-packages/gradio/routes.py", line 283, in run_predict
    output = await app.blocks.process_api(
  File "/data/wzy/chinese_stable-diffusion-webui/venv/lib/python3.8/site-packages/gradio/blocks.py", line 936, in process_api
    result = await self.call_function(fn_index, inputs, iterator)
  File "/data/wzy/chinese_stable-diffusion-webui/venv/lib/python3.8/site-packages/gradio/blocks.py", line 777, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/data/wzy/chinese_stable-diffusion-webui/venv/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/data/wzy/chinese_stable-diffusion-webui/venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/data/wzy/chinese_stable-diffusion-webui/venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/data/wzy/chinese_stable-diffusion-webui/modules/ui.py", line 443, in update_token_counter
    tokens, token_count, max_length = max([model_hijack.tokenize(prompt) for prompt in prompts], key=lambda args: args[1])
  File "/data/wzy/chinese_stable-diffusion-webui/modules/ui.py", line 443, in <listcomp>
    tokens, token_count, max_length = max([model_hijack.tokenize(prompt) for prompt in prompts], key=lambda args: args[1])
  File "/data/wzy/chinese_stable-diffusion-webui/modules/sd_hijack.py", line 125, in tokenize
    _, remade_batch_tokens, _, _, _, token_count = self.clip.process_text([text])
  File "/data/wzy/chinese_stable-diffusion-webui/modules/sd_hijack.py", line 234, in process_text
    remade_tokens, fixes, multipliers, current_token_count = self.tokenize_line(line, used_custom_terms, hijack_comments)
  File "/data/wzy/chinese_stable-diffusion-webui/modules/sd_hijack.py", line 166, in tokenize_line
    tokenized = self.wrapped.tokenizer([text for text, _ in parsed], truncation=False, add_special_tokens=False)["input_ids"]
  File "/data/wzy/chinese_stable-diffusion-webui/venv/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2477, in __call__
    return self.batch_encode_plus(
  File "/data/wzy/chinese_stable-diffusion-webui/venv/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2668, in batch_encode_plus
    return self._batch_encode_plus(
  File "/data/wzy/chinese_stable-diffusion-webui/venv/lib/python3.8/site-packages/transformers/tokenization_utils.py", line 734, in _batch_encode_plus
    batch_outputs = self._batch_prepare_for_model(
  File "/data/wzy/chinese_stable-diffusion-webui/venv/lib/python3.8/site-packages/transformers/tokenization_utils.py", line 806, in _batch_prepare_for_model
    batch_outputs = self.pad(
  File "/data/wzy/chinese_stable-diffusion-webui/venv/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2827, in pad
    raise ValueError(
ValueError: type of None unknown: <class 'NoneType'>. Should be one of a python, numpy, pytorch or tensorflow object.
Traceback (most recent call last):
  File "/data/wzy/chinese_stable-diffusion-webui/venv/lib/python3.8/site-packages/gradio/routes.py", line 283, in run_predict
    output = await app.blocks.process_api(
  File "/data/wzy/chinese_stable-diffusion-webui/venv/lib/python3.8/site-packages/gradio/blocks.py", line 936, in process_api
    result = await self.call_function(fn_index, inputs, iterator)
  File "/data/wzy/chinese_stable-diffusion-webui/venv/lib/python3.8/site-packages/gradio/blocks.py", line 777, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/data/wzy/chinese_stable-diffusion-webui/venv/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/data/wzy/chinese_stable-diffusion-webui/venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/data/wzy/chinese_stable-diffusion-webui/venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/data/wzy/chinese_stable-diffusion-webui/modules/ui.py", line 443, in update_token_counter
    tokens, token_count, max_length = max([model_hijack.tokenize(prompt) for prompt in prompts], key=lambda args: args[1])
  File "/data/wzy/chinese_stable-diffusion-webui/modules/ui.py", line 443, in <listcomp>
    tokens, token_count, max_length = max([model_hijack.tokenize(prompt) for prompt in prompts], key=lambda args: args[1])
  File "/data/wzy/chinese_stable-diffusion-webui/modules/sd_hijack.py", line 125, in tokenize
    _, remade_batch_tokens, _, _, _, token_count = self.clip.process_text([text])
  File "/data/wzy/chinese_stable-diffusion-webui/modules/sd_hijack.py", line 234, in process_text
    remade_tokens, fixes, multipliers, current_token_count = self.tokenize_line(line, used_custom_terms, hijack_comments)
  File "/data/wzy/chinese_stable-diffusion-webui/modules/sd_hijack.py", line 166, in tokenize_line
    tokenized = self.wrapped.tokenizer([text for text, _ in parsed], truncation=False, add_special_tokens=False)["input_ids"]
  File "/data/wzy/chinese_stable-diffusion-webui/venv/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2477, in __call__
    return self.batch_encode_plus(
  File "/data/wzy/chinese_stable-diffusion-webui/venv/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2668, in batch_encode_plus
    return self._batch_encode_plus(
  File "/data/wzy/chinese_stable-diffusion-webui/venv/lib/python3.8/site-packages/transformers/tokenization_utils.py", line 734, in _batch_encode_plus
    batch_outputs = self._batch_prepare_for_model(
  File "/data/wzy/chinese_stable-diffusion-webui/venv/lib/python3.8/site-packages/transformers/tokenization_utils.py", line 806, in _batch_prepare_for_model
    batch_outputs = self.pad(
  File "/data/wzy/chinese_stable-diffusion-webui/venv/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2827, in pad
    raise ValueError(
ValueError: type of None unknown: <class 'NoneType'>. Should be one of a python, numpy, pytorch or tensorflow object.

Thank you! @ganzhiruyi

ebziw commented 1 year ago

I have successfully run Taiyi model on a fresh installed Centos 9-stream. A fiew things about the toturial.

  1. Step 1 is OK
  2. Step 2 for new system, "python3 launch.py" won't work, should be "sh webui.sh" to start from begining.
  3. Step 3 is confusing, took some time to figure out: " Download Taiyi model files from https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-1B-Chinese-v0.1 place your local_path. " actually means: a. Make sure you have git lfs support (do something like "dnf install git-lfs” in Centos) b. Do a "git clone https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-1B-Chinese-v0.1" , it should download a bunch of files with a total size of 10G+. Only 5G is useful, everything under .git can be deleted. Did'nt find out but if there is a way to download only the necessary files excluding the .git directory, it should be provided to save some space and time. c. After replacing the two files mentioned in the toturial, open v1-inference.yaml, at line 75 change "local_path" to the directory where you just download the diffusion models to. BE WARE: absolute path only, no relative path such as ~yourname/modules/TaiyiModel/ d. There is a typo: "/stable-diffusionldm" should be "/stable-diffusion/ldm", small things matter, especially for someone who is new.
  4. Step 4 is OK
  5. Step 5, in order to start web service, the launch command should be “python3 launch.py --listen --ckpt path-to-ckpt-file"
ganzhiruyi commented 1 year ago

I have successfully run Taiyi model on a fresh installed Centos 9-stream. A fiew things about the toturial.

  1. Step 1 is OK
  2. Step 2 for new system, "python3 launch.py" won't work, should be "sh webui.sh" to start from begining.
  3. Step 3 is confusing, took some time to figure out: " Download Taiyi model files from https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-1B-Chinese-v0.1 place your local_path. " actually means: a. Make sure you have git lfs support (do something like "dnf install git-lfs” in Centos) b. Do a "git clone https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-1B-Chinese-v0.1" , it should download a bunch of files with a total size of 10G+. Only 5G is useful, everything under .git can be deleted. Did'nt find out but if there is a way to download only the necessary files excluding the .git directory, it should be provided to save some space and time. c. After replacing the two files mentioned in the toturial, open v1-inference.yaml, at line 75 change "local_path" to the directory where you just download the diffusion models to. BE WARE: absolute path only, no relative path such as ~yourname/modules/TaiyiModel/ d. There is a typo: "/stable-diffusionldm" should be "/stable-diffusion/ldm", small things matter, especially for someone who is new.
  4. Step 4 is OK
  5. Step 5, in order to start web service, the launch command should be “python3 launch.py --listen --ckpt path-to-ckpt-file"

Thanks for advices, I have updated the readme

ganzhiruyi commented 1 year ago

I have writen a readme , everyone can configuration Taiyi-Stable-Diffusion-1B-Chinese-v0.1 in webui step by step. https://github.com/IDEA-CCNL/stable-diffusion-webui/blob/master/README.md

thanks for job but follow this step in win11 show errors:

image

@ganzhiruyi

I don't run webui on windows, it seems like your tokenizer loaded error, you can check your model path have a 'tokenizer' subfolder.

ebziw commented 1 year ago

Me again.Taiyi CN version works fine. I'm Trying to get EN version work but got no luck. Here's what I did:

  1. git clone https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-1B-Chinese-EN-v0.1/, put it under stable diffusion models directory where the CN version lies.
  2. wget https://raw.githubusercontent.com/IDEA-CCNL/Fengshenbang-LM/main/fengshen/utils/convert_diffusers_to_original_stable_diffusion.py
  3. python3 convert_diffusers_to_sd.py --model_path ./Taiyi-Stable-Diffusion-1B-Chinese-EN-v0.1/ --checkpoint_path ./Taiyi-Stable-Diffusion-1B-Chinese-EN-v0.1/Taiyi-Stable-Diffusion-1B-Chinese-EN-v0.1.ckpt It gives some errors like: Reshaping encoder.mid.attn_1.q.weight for SD format Reshaping encoder.mid.attn_1.k.weight for SD format Reshaping encoder.mid.attn_1.v.weight for SD format Reshaping encoder.mid.attn_1.proj_out.weight for SD format Reshaping decoder.mid.attn_1.q.weight for SD format Reshaping decoder.mid.attn_1.k.weight for SD format Reshaping decoder.mid.attn_1.v.weight for SD format Reshaping decoder.mid.attn_1.proj_out.weight for SD format
  4. Now, after restart webui, I can see the EN version ckpt under dropdown menu. Now I try to load the EN ckpt file, the system reports error. Background shows such error: Traceback (most recent call last): File "/home/liyong/stable-diffusion-webui/venv/lib64/python3.9/site-packages/gradio/routes.py", line 284, in run_predict output = await app.blocks.process_api( File "/home/liyong/stable-diffusion-webui/venv/lib64/python3.9/site-packages/gradio/blocks.py", line 936, in process_api result = await self.call_function(fn_index, inputs, iterator) File "/home/liyong/stable-diffusion-webui/venv/lib64/python3.9/site-packages/gradio/blocks.py", line 777, in call_function prediction = await anyio.to_thread.run_sync( File "/home/liyong/stable-diffusion-webui/venv/lib64/python3.9/site-packages/anyio/to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "/home/liyong/stable-diffusion-webui/venv/lib64/python3.9/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "/home/liyong/stable-diffusion-webui/venv/lib64/python3.9/site-packages/anyio/_backends/_asyncio.py", line 867, in run result = context.run(func, args) File "/home/liyong/taiyi-stable-diffusion-webui/modules/ui.py", line 1636, in fn=lambda value, k=k: run_settings_single(value, key=k), File "/home/liyong/taiyi-stable-diffusion-webui/modules/ui.py", line 1478, in run_settings_single opts.data_labels[key].onchange() File "/home/liyong/taiyi-stable-diffusion-webui/webui.py", line 41, in f res = func(args, **kwargs) File "/home/liyong/taiyi-stable-diffusion-webui/webui.py", line 83, in shared.opts.onchange("sd_model_checkpoint", wrap_queued_call(lambda: modules.sd_models.reload_model_weights())) File "/home/liyong/taiyi-stable-diffusion-webui/modules/sd_models.py", line 286, in reload_model_weights sd_hijack.model_hijack.undo_hijack(sd_model) File "/home/liyong/taiyi-stable-diffusion-webui/modules/sd_hijack.py", line 103, in undo_hijack if type(model_embeddings.token_embedding) == EmbeddingsWithFixes: File "/home/liyong/stable-diffusion-webui/venv/lib64/python3.9/site-packages/torch/nn/modules/module.py", line 1207, in getattr raise AttributeError("'{}' object has no attribute '{}'".format( AttributeError: 'BertEmbeddings' object has no attribute 'token_embedding'

What's is wrong?

ganzhiruyi commented 1 year ago

Me again.Taiyi CN version works fine. I'm Trying to get EN version work but got no luck. Here's what I did:

  1. git clone https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-1B-Chinese-EN-v0.1/, put it under stable diffusion models directory where the CN version lies.
  2. wget https://raw.githubusercontent.com/IDEA-CCNL/Fengshenbang-LM/main/fengshen/utils/convert_diffusers_to_original_stable_diffusion.py
  3. python3 convert_diffusers_to_sd.py --model_path ./Taiyi-Stable-Diffusion-1B-Chinese-EN-v0.1/ --checkpoint_path ./Taiyi-Stable-Diffusion-1B-Chinese-EN-v0.1/Taiyi-Stable-Diffusion-1B-Chinese-EN-v0.1.ckpt It gives some errors like: Reshaping encoder.mid.attn_1.q.weight for SD format Reshaping encoder.mid.attn_1.k.weight for SD format Reshaping encoder.mid.attn_1.v.weight for SD format Reshaping encoder.mid.attn_1.proj_out.weight for SD format Reshaping decoder.mid.attn_1.q.weight for SD format Reshaping decoder.mid.attn_1.k.weight for SD format Reshaping decoder.mid.attn_1.v.weight for SD format Reshaping decoder.mid.attn_1.proj_out.weight for SD format
  4. Now, after restart webui, I can see the EN version ckpt under dropdown menu. Now I try to load the EN ckpt file, the system reports error. Background shows such error: Traceback (most recent call last): File "/home/liyong/stable-diffusion-webui/venv/lib64/python3.9/site-packages/gradio/routes.py", line 284, in run_predict output = await app.blocks.process_api( File "/home/liyong/stable-diffusion-webui/venv/lib64/python3.9/site-packages/gradio/blocks.py", line 936, in process_api result = await self.call_function(fn_index, inputs, iterator) File "/home/liyong/stable-diffusion-webui/venv/lib64/python3.9/site-packages/gradio/blocks.py", line 777, in call_function prediction = await anyio.to_thread.run_sync( File "/home/liyong/stable-diffusion-webui/venv/lib64/python3.9/site-packages/anyio/to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "/home/liyong/stable-diffusion-webui/venv/lib64/python3.9/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "/home/liyong/stable-diffusion-webui/venv/lib64/python3.9/site-packages/anyio/_backends/_asyncio.py", line 867, in run result = context.run(func, args) File "/home/liyong/taiyi-stable-diffusion-webui/modules/ui.py", line 1636, in fn=lambda value, k=k: run_settings_single(value, key=k), File "/home/liyong/taiyi-stable-diffusion-webui/modules/ui.py", line 1478, in run_settings_single opts.data_labels[key].onchange() File "/home/liyong/taiyi-stable-diffusion-webui/webui.py", line 41, in f res = func(args, kwargs) File "/home/liyong/taiyi-stable-diffusion-webui/webui.py", line 83, in shared.opts.onchange("sd_model_checkpoint", wrap_queued_call(lambda: modules.sd_models.reload_model_weights())) File "/home/liyong/taiyi-stable-diffusion-webui/modules/sd_models.py", line 286, in reload_model_weights sd_hijack.model_hijack.undo_hijack(sd_model) File "/home/liyong/taiyi-stable-diffusion-webui/modules/sd_hijack.py", line 103, in undo_hijack if type(model_embeddings.token_embedding) == EmbeddingsWithFixes: File "/home/liyong/stable-diffusion-webui/venv/lib64/python3.9/site-packages/torch/nn/modules/module.py", line 1207, in getattr** raise AttributeError("'{}' object has no attribute '{}'".format( AttributeError: 'BertEmbeddings' object has no attribute 'token_embedding'

What's is wrong?

参考https://github.com/IDEA-CCNL/stable-diffusion-webui/blob/master/README.md ，最新代码已经支持了 https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-1B-Chinese-EN-v0.1/ 这个模型

ebziw commented 1 year ago

OK, works now. But, if I tried to switch back to the Chinese only model, error occured. Here's what's happened:

Traceback (most recent call last): File "/home/liyong/taiyi-stable-diffusion-webui/venv/lib64/python3.9/site-packages/gradio/routes.py", line 284, in run_predict output = await app.blocks.process_api( File "/home/liyong/taiyi-stable-diffusion-webui/venv/lib64/python3.9/site-packages/gradio/blocks.py", line 936, in process_api result = await self.call_function(fn_index, inputs, iterator) File "/home/liyong/taiyi-stable-diffusion-webui/venv/lib64/python3.9/site-packages/gradio/blocks.py", line 777, in call_function prediction = await anyio.to_thread.run_sync( File "/home/liyong/taiyi-stable-diffusion-webui/venv/lib64/python3.9/site-packages/anyio/to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "/home/liyong/taiyi-stable-diffusion-webui/venv/lib64/python3.9/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "/home/liyong/taiyi-stable-diffusion-webui/venv/lib64/python3.9/site-packages/anyio/_backends/_asyncio.py", line 867, in run result = context.run(func, args) File "/home/liyong/taiyi-stable-diffusion-webui/modules/ui.py", line 1636, in fn=lambda value, k=k: run_settings_single(value, key=k), File "/home/liyong/taiyi-stable-diffusion-webui/modules/ui.py", line 1478, in run_settings_single opts.data_labels[key].onchange() File "/home/liyong/taiyi-stable-diffusion-webui/webui.py", line 41, in f res = func(args, **kwargs) File "/home/liyong/taiyi-stable-diffusion-webui/webui.py", line 83, in shared.opts.onchange("sd_model_checkpoint", wrap_queued_call(lambda: modules.sd_models.reload_model_weights())) File "/home/liyong/taiyi-stable-diffusion-webui/modules/sd_models.py", line 286, in reload_model_weights sd_hijack.model_hijack.undo_hijack(sd_model) File "/home/liyong/taiyi-stable-diffusion-webui/modules/sd_hijack.py", line 105, in undo_hijack if type(model_embeddings.word_embeddings) == EmbeddingsWithFixes: File "/home/liyong/taiyi-stable-diffusion-webui/venv/lib64/python3.9/site-packages/torch/nn/modules/module.py", line 1207, in getattr raise AttributeError("'{}' object has no attribute '{}'".format( AttributeError: 'CLIPTextEmbeddings' object has no attribute 'word_embeddings'

It would be nice to have the two models co-exist. It's alright if not. Nice work, keep on.

发件人: ganzhiruyi @.> 发送时间: 2022年11月19日 13:55 收件人: IDEA-CCNL/Fengshenbang-LM @.> 抄送: Smiler @.>; Comment @.> 主题: Re: [IDEA-CCNL/Fengshenbang-LM] [Done] Add checkpoint and webui code for IDEA-CCNL/Taiyi-Stable-Diffusion-1B-Chinese-v0.1 (Issue #186)

Me again.Taiyi CN version works fine. I'm Trying to get EN version work but got no luck. Here's what I did:

  1. git clone https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-1B-Chinese-EN-v0.1/, put it under stable diffusion models directory where the CN version lies.
  2. wget https://raw.githubusercontent.com/IDEA-CCNL/Fengshenbang-LM/main/fengshen/utils/convert_diffusers_to_original_stable_diffusion.py
  3. python3 convert_diffusers_to_sd.py --model_path ./Taiyi-Stable-Diffusion-1B-Chinese-EN-v0.1/ --checkpoint_path ./Taiyi-Stable-Diffusion-1B-Chinese-EN-v0.1/Taiyi-Stable-Diffusion-1B-Chinese-EN-v0.1.ckpt It gives some errors like: Reshaping encoder.mid.attn_1.q.weight for SD format Reshaping encoder.mid.attn_1.k.weight for SD format Reshaping encoder.mid.attn_1.v.weight for SD format Reshaping encoder.mid.attn_1.proj_out.weight for SD format Reshaping decoder.mid.attn_1.q.weight for SD format Reshaping decoder.mid.attn_1.k.weight for SD format Reshaping decoder.mid.attn_1.v.weight for SD format Reshaping decoder.mid.attn_1.proj_out.weight for SD format
  4. Now, after restart webui, I can see the EN version ckpt under dropdown menu. Now I try to load the EN ckpt file, the system reports error. Background shows such error: Traceback (most recent call last): File "/home/liyong/stable-diffusion-webui/venv/lib64/python3.9/site-packages/gradio/routes.py", line 284, in run_predict output = await app.blocks.process_api( File "/home/liyong/stable-diffusion-webui/venv/lib64/python3.9/site-packages/gradio/blocks.py", line 936, in process_api result = await self.call_function(fn_index, inputs, iterator) File "/home/liyong/stable-diffusion-webui/venv/lib64/python3.9/site-packages/gradio/blocks.py", line 777, in call_function prediction = await anyio.to_thread.run_sync( File "/home/liyong/stable-diffusion-webui/venv/lib64/python3.9/site-packages/anyio/to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "/home/liyong/stable-diffusion-webui/venv/lib64/python3.9/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "/home/liyong/stable-diffusion-webui/venv/lib64/python3.9/site-packages/anyio/_backends/_asyncio.py", line 867, in run result = context.run(func, args) File "/home/liyong/taiyi-stable-diffusion-webui/modules/ui.py", line 1636, in fn=lambda value, k=k: run_settings_single(value, key=k), File "/home/liyong/taiyi-stable-diffusion-webui/modules/ui.py", line 1478, in run_settings_single opts.data_labels[key].onchange() File "/home/liyong/taiyi-stable-diffusion-webui/webui.py", line 41, in f res = func(args, **kwargs) File "/home/liyong/taiyi-stable-diffusion-webui/webui.py", line 83, in shared.opts.onchange("sd_model_checkpoint", wrap_queued_call(lambda: modules.sd_models.reload_model_weights())) File "/home/liyong/taiyi-stable-diffusion-webui/modules/sd_models.py", line 286, in reload_model_weights sd_hijack.model_hijack.undo_hijack(sd_model) File "/home/liyong/taiyi-stable-diffusion-webui/modules/sd_hijack.py", line 103, in undo_hijack if type(model_embeddings.token_embedding) == EmbeddingsWithFixes: File "/home/liyong/stable-diffusion-webui/venv/lib64/python3.9/site-packages/torch/nn/modules/module.py", line 1207, in getattr raise AttributeError("'{}' object has no attribute '{}'".format( AttributeError: 'BertEmbeddings' object has no attribute 'token_embedding'

What's is wrong?

参考https://github.com/IDEA-CCNL/stable-diffusion-webui/blob/master/README.md ，最新代码已经支持了 https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-1B-Chinese-EN-v0.1/ 这个模型

― Reply to this email directly, view it on GitHubhttps://github.com/IDEA-CCNL/Fengshenbang-LM/issues/186#issuecomment-1320811146, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A4GRQI6GJOK3KR2JIBNDLA3WJBTVDANCNFSM6AAAAAARYOT34U. You are receiving this because you commented.Message ID: @.***>

soulteary commented 1 year ago

可以试试这个 issue 里的方法，使用容器来运行项目，应该可以规避掉帖子里所有的因为运行环境安装配置、PyPi 软件包版本、下载文件放置路径不对的问题。

第一步：下载运行环境镜像

docker pull soulteary/stable-diffusion:taiyi-0.1

第二步：想运行“太乙”，除了需要下载“模型游乐场”镜像之外，我们还需要获取“太乙模型”文件

git clone https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-1B-Chinese-v0.1

整个仓库尺寸比较大（大概有 18GB），需要花费一些时间：

Cloning into 'Taiyi-Stable-Diffusion-1B-Chinese-v0.1'...
remote: Enumerating objects: 157, done.
remote: Counting objects: 100% (157/157), done.
remote: Compressing objects: 100% (155/155), done.
remote: Total 157 (delta 77), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (157/157), 3.06 MiB | 22.25 MiB/s, done.
Resolving deltas: 100% (77/77), done.
Filtering content: 100% (5/5), 8.92 GiB | 11.48 MiB/s, done.

原始项目启用了 git lfs，所以添加不添加 --depth 参数没有差别，耐心等待模型下载完毕。

第三步：编写一个容器编排文件，来启动模型应用

version: "2"
services:

  taiyi:
    image: soulteary/stable-diffusion:taiyi-0.1
    container_name: taiyi
    restart: always
    runtime: nvidia
    ipc: host
    ports:
      - "7860:7860"
    volumes:
      - ./Taiyi-Stable-Diffusion-1B-Chinese-v0.1:/stable-diffusion-webui/models/Taiyi-Stable-Diffusion-1B-Chinese-v0.1

将上面的内容保存为 docker-compose.yml 之后，执行 docker compose up -d，稍等片刻，在浏览器访问启动服务的 IP 地址和对应端口，比如：http://localhost:7860，就能够正常使用啦。

generate

@ganzhiruyi @chenkai-666 @lizekui @fingerx @sherlockeded @ebziw