Closed aa956 closed 3 months ago
OwenTruong
commented
7 months ago Hi, I managed to fix the issue in my windows virtual machine in bug/utf-write branch just now.
For now I have merged it into master since it is a pretty critical bug. Can you give it a go? New version is v2.0.1 on pypi :)
aa956
commented
7 months ago So, just pulled the master branch and installed from git with pip install --upgrade .
Now sample images and metadata are downloaded
New error is division by 0, so I suppose that this fix worked and we've got to the next (probably separate) error :D
PS E:\models\Stable-diffusion\sdxl\realism-engine-sdxl> civitdl --verbose --limit-rate 5M https://civitai.com/models/152525?modelVersionId=293240 .
Active code page: 65001
Arguments: {'source_strings': ['https://civitai.com/models/152525?modelVersionId=293240'], 'rootdir': '.', 'sorter': 'basic', 'max_images': 3, 'api_key': '', 'with_prompt': True, 'limit_rate': '5M', 'retry_count': 3, 'pause_time': 3.0, 'cache_mode': '1', 'model_overwrite': False, 'verbose': True}
Chosen Sorter Description: This is useful when you only just want all of the lora/checkpoint/etc. in the same root directory. Metadata + images are downloaded inside a directory under the parent directory.
Requesting model metadata.
Metadata API Request URL: https://civitai.com/api/v1/models/152525
Finished requesting model metadata.
Requesting model metadata.
Metadata API Request URL: https://civitai.com/api/v1/model-versions/293240
Finished requesting model metadata.
Now downloading "Realism Engine SDXL"...
- Model ID: 152525
- Version ID: 293240
Preparing to send download model request...
Model Download API URL: https://civitai.com/api/download/models/293240
Now requesting images...
Finished requesting images...
Images: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3.00/3.00 [00:00<00:00, 285iB/s]
---------
Traceback (most recent call last):
File "C:\Users\username\projects\civitdl\venv\Lib\site-packages\civitdl\batch\batch_download.py", line 32, in batch_download
download_model(
File "C:\Users\username\projects\civitdl\venv\Lib\site-packages\civitdl\batch\_get_model.py", line 268, in download_model
_download_prompts(sorter_data.prompt_dir_path, prompts_basenames,
File "C:\Users\username\projects\civitdl\venv\Lib\site-packages\civitdl\batch\_get_model.py", line 146, in _download_prompts
write_to_files(dirpath, basenames, [dumps(
File "C:\Users\username\projects\civitdl\venv\Lib\site-packages\helpers\utils.py", line 182, in write_to_files
write_contents(file, content_chunks)
File "C:\Users\username\projects\civitdl\venv\Lib\site-packages\helpers\utils.py", line 107, in write_contents
speed = bytes_downloaded / download_time
~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~
ZeroDivisionError: float division by zero
float division by zero
---------
Pausing for 3 seconds...
Waking up!
PS E:\models\Stable-diffusion\sdxl\realism-engine-sdxl> ls ".\Realism Engine SDXL\extra_data-vid_293240\"
Directory: E:\models\Stable-diffusion\sdxl\realism-engine-sdxl\Realism Engine SDXL\extra_data-vid_293240
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 24.01.2024 16:14 25209 5342532.jpeg
-a--- 24.01.2024 16:14 42749 5344273.jpeg
-a--- 24.01.2024 16:14 16 5344273.json
-a--- 24.01.2024 16:14 32632 5344319.jpeg
-a--- 24.01.2024 16:14 128759 model_dict-mid_152525-vid_293240.json
PS E:\models\Stable-diffusion\sdxl\realism-engine-sdxl>
Your computer/network is pretty fast :)
I made some changes so that when download_time is 0, it won't divide (branch is bug/download-time). download_time is a variable to check how much time has passed per chunk downloaded.
Would you be able to check that out?
# for convenience reasons
git pull
git checkout bug/download-time
pip install .Apparently unicode bug has been in windows version of civitdl in v1 of the script... I can't believe I never found out about this bug...
aa956
commented
7 months ago Sorry, took some time to return to my desktop.
branch is
bug/download-time
It's working, everything looks correct now! Thank you.
Your computer/network is pretty fast :)
I'd suspect time resolution on Python Windows vs Python Linux as it looks like error was triggered in pretty small file, 5344273.json accompanying the sample image was broken before - just 16 bytes was saved in the listing above when division by 0 occured and 1258 bytes now (still smaller than typical ethernet MTU so probably was downloaded in single packet but can't be sure) after branch bug/download-time run through all of metadata and sample images and started the download of the actual 6.5Gb checkpoint:
PS E:\models\Stable-diffusion\sdxl\realism-engine-sdxl> ls ".\Realism Engine SDXL\extra_data-vid_293240\"
Directory: E:\models\Stable-diffusion\sdxl\realism-engine-sdxl\Realism Engine SDXL\extra_data-vid_293240
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 24.01.2024 18:49 25209 5342532.jpeg
-a--- 24.01.2024 18:49 1609 5342532.json
-a--- 24.01.2024 18:49 42749 5344273.jpeg
-a--- 24.01.2024 18:49 1258 5344273.json
-a--- 24.01.2024 18:49 32632 5344319.jpeg
-a--- 24.01.2024 18:49 1648 5344319.json
-a--- 24.01.2024 18:49 128759 model_dict-mid_152525-vid_293240.jsonApparently unicode bug has been in windows version of civitdl in v1 of the script... I can't believe I never found out about this bug...
There's some pretty broken HTML in Realism Engine SDXL model description:
F7ZE0hKIGXtkR4qfjKEwYou\"><span style=\"color:rgb(193, 194, 197)\">You</span></a><span style=\"color:rgb(193, 194, 197)\"> can also add hires_fix_refiner_passAs can be seen there are two U+FFFC characters before You\">
OwenTruong
commented
7 months ago Apparently unicode bug has been in windows version of civitdl in v1 of the script... I can't believe I never found out about this bug...
There's some pretty broken HTML in Realism Engine SDXL model description:
F7ZE0hKIGXtkR4qfjKEwYou\"><span style=\"color:rgb(193, 194, 197)\">You</span></a><span style=\"color:rgb(193, 194, 197)\"> can also add hires_fix_refiner_passAs can be seen there are two U+FFFC characters before
You\">
Hmm looking at the html in civitai.com/models/152525, it also gave the same broke url with two U+FFFC for some reason on the model's page. I tried changing to utf-32 encoding for the metadata, but still the same issue.
Not sure if this is the author's mistake, or if it is something we can fix on civitdl's end.
I think this link / anchor element was suppose to display a url or image that leads to some kind of example image on how add hires_fix_refiner_pass to some quick list setting.
<a target=\"_blank\" rel=\"ugc\" href=\"https://private-user-images.githubusercontent.com/655287/268127943-b3b3177b-29f5-4013-a984-33e19c8bc13b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDQzOTUwMTMsIm5iZiI6MTcwNDM5NDcxMywicGF0aCI6Ii82NTUyODcvMjY4MTI3OTQzLWIzYjMxNzdiLTI5ZjUtNDAxMy1hOTg0LTMzZTE5YzhiYzEzYi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwMTA0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDEwNFQxODU4MzNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT02NGFlYzE0YjNmMmY5ZDdjYzYyNjE0MDY1YjUyOTMyYTE5YmY1YWM5N2NlNjdhOWM1NTA3YmExYmI4OTY0NGQ1JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.oH2ZYe6EYUC84DKAa0ovqnxF7ZE0hKIGXtkR4qfjKEwYou\"><span style=\"color:rgb(193, 194, 197)\">You</span></a>So looks most probably like the result of partially HTML formatted copy/paste of the description by model author, and that's the reason why this error never surfaced before.
Although I'm slightly surprised too that no one tried to download some models with e.g. Japanese or Chinese language descriptions on the Windows box with the default Latin locale.
Edit: And I suppose there's nothing to fix here on client side? Description is downloaded as is and is probably best left for web browsers to interpret as is. If someone would like to process the JSON file further it's simple to pipe the HTML parts through html-tidy or something similar.
OwenTruong
commented
7 months ago And I suppose there's nothing to fix here on client side?
- Unfortunately so if the author copy pasted their html then only the author themselves probably have the correct html.
- Unless I am missing anything else here
Description is downloaded as is and is probably best left for web browsers to interpret as is. If someone would like to process the JSON file further it's simple to pipe the HTML parts through html-tidy or something similar.
Do you mean that piping the html part might fix the link?
I was curious what html-tidy did so I tried to run the description of Realism Engine SDXL on this site (not sure if this site does what html-tidy is supposed to do).
Don't know if it actually looked prettier.
Before:
<h2 id=\"heading-167\"><span style=\"color:rgb(34, 139, 230)\">Version 3.0 of the Realism Engine</span></h2><p>In this version we improved the skin realism, the eyes. We also improved the Male anatomy.</p><p>Keep in mind that this is not a NSFW model. For that I recommend SDXXXL.</p><p></p><h2 id=\"heading-167\"><span style=\"color:rgb(34, 139, 230)\">Introducing Version 2.0 of the Realism Engine</span></h2><p>We are excited to announce that Version 2.0 of the Realism Engine is now available, featuring significant enhancements.</p><p></p><p>This updated model exhibits superior prompt responsiveness and offers markedly improved overall coherence, including more facial expressions, a greater variety of faces, more poses, and improved hands.</p><p></p><p>Additionally, we have addressed several quirks from the previous version, such as the challenges in generating nighttime scenes and the issue where increasing the CFG Scale enhanced color vividness without proportionally adjusting contrast. While being capable of producing NSFW content it is less likely to produce it without asking.</p><p></p><p><em>This version include the VAE to help newcomers with their setup.</em></p><p></p><p><strong>Happy Prompting</strong></p><p></p><h3 id=\"heading-13\"><span style=\"color:rgb(34, 139, 230)\">Good news for the fans of the original Realism Engine</span></h3><p><strong>Realism Engine SDXL is here</strong></p><p></p><p>Version 1.0 is officially out.</p><p></p><p><strong>Here's the recommended setting for Auto1111</strong></p><p>Sampler: DPM++ 2S a,</p><p>CFG scale range: 5-9,</p><p>Hires sampler: DPM++ SDE Karras,</p><p>Hires upscaler: ESRGAN_4x,</p><p>Refiner switch at: 0.9 (<strong>ON THE FIRST PASS ONLY)</strong></p><p><img src=\"https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/51dd7ec2-b64d-4c63-b7e0-7aea6f4413c3/width=525/51dd7ec2-b64d-4c63-b7e0-7aea6f4413c3.jpeg\" /></p><p><span style=\"color:rgb(193, 194, 197)\">Look for this in Setting -> Stable Diffusion</span><br /><a target=\"_blank\" rel=\"ugc\" href=\"https://private-user-images.githubusercontent.com/655287/268127943-b3b3177b-29f5-4013-a984-33e19c8bc13b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDQzOTUwMTMsIm5iZiI6MTcwNDM5NDcxMywicGF0aCI6Ii82NTUyODcvMjY4MTI3OTQzLWIzYjMxNzdiLTI5ZjUtNDAxMy1hOTg0LTMzZTE5YzhiYzEzYi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwMTA0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDEwNFQxODU4MzNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT02NGFlYzE0YjNmMmY5ZDdjYzYyNjE0MDY1YjUyOTMyYTE5YmY1YWM5N2NlNjdhOWM1NTA3YmExYmI4OTY0NGQ1JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.oH2ZYe6EYUC84DKAa0ovqnxF7ZE0hKIGXtkR4qfjKEwYou\"><span style=\"color:rgb(193, 194, 197)\">You</span></a><span style=\"color:rgb(193, 194, 197)\"> can also add hires_fix_refiner_pass to you quick list setting to have it on top of you Ui</span></p><p></p><p>While this model hit some of the key goals I was reaching for, it will continue to be trained to fix the weaknesses.</p><p></p><p>I appreciate all the good feedback from the community.</p>After:
<div>
<div><h2 id=\"heading-167\"><span style=\"color:rgb(34, 139, 230)\">Version 3.0 of the Realism Engine</span></h2><p>In this version we improved the skin realism, the eyes. We also improved the Male anatomy.</p><p>Keep in mind that this is not a NSFW model. For that I recommend SDXXXL.</p><p></p><h2 id=\"heading-167\"><span style=\"color:rgb(34, 139, 230)\">Introducing Version 2.0 of the Realism Engine</span></h2><p>We are excited to announce that Version 2.0 of the Realism Engine is now available, featuring significant enhancements.</p><p></p><p>This updated model exhibits superior prompt responsiveness and offers markedly improved overall coherence, including more facial expressions, a greater variety of faces, more poses, and improved hands.</p><p></p><p>Additionally, we have addressed several quirks from the previous version, such as the challenges in generating nighttime scenes and the issue where increasing the CFG Scale enhanced color vividness without proportionally adjusting contrast. While being capable of producing NSFW content it is less likely to produce it without asking.</p><p></p><p><em>This version include the VAE to help newcomers with their setup.</em></p><p></p><p><strong>Happy Prompting</strong></p><p></p><h3 id=\"heading-13\"><span style=\"color:rgb(34, 139, 230)\">Good news for the fans of the original Realism Engine</span></h3><p><strong>Realism Engine SDXL is here</strong></p><p></p><p>Version 1.0 is officially out.</p><p></p><p><strong>Here's the recommended setting for Auto1111</strong></p><p>Sampler: DPM++ 2S a,</p><p>CFG scale range: 5-9,</p><p>Hires sampler: DPM++ SDE Karras,</p><p>Hires upscaler: ESRGAN_4x,</p><p>Refiner switch at: 0.9 (<strong>ON THE FIRST PASS ONLY)</strong></p><p><img src=\"https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/51dd7ec2-b64d-4c63-b7e0-7aea6f4413c3/width=525/51dd7ec2-b64d-4c63-b7e0-7aea6f4413c3.jpeg\" /></p><p><span style=\"color:rgb(193, 194, 197)\">Look for this in Setting -&gt; Stable Diffusion</span><br /><a target=\"_blank\" rel=\"ugc\" href=\"https://private-user-images.githubusercontent.com/655287/268127943-b3b3177b-29f5-4013-a984-33e19c8bc13b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDQzOTUwMTMsIm5iZiI6MTcwNDM5NDcxMywicGF0aCI6Ii82NTUyODcvMjY4MTI3OTQzLWIzYjMxNzdiLTI5ZjUtNDAxMy1hOTg0LTMzZTE5YzhiYzEzYi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwMTA0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDEwNFQxODU4MzNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT02NGFlYzE0YjNmMmY5ZDdjYzYyNjE0MDY1YjUyOTMyYTE5YmY1YWM5N2NlNjdhOWM1NTA3YmExYmI4OTY0NGQ1JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.oH2ZYe6EYUC84DKAa0ovqnxF7ZE0hKIGXtkR4qfjKEwYou\"><span style=\"color:rgb(193, 194, 197)\">You</span></a><span style=\"color:rgb(193, 194, 197)\"> can also add hires_fix_refiner_pass to you quick list setting to have it on top of you Ui</span></p><p></p><p>While this model hit some of the key goals I was reaching for, it will continue to be trained to fix the weaknesses.</p><p></p><p>I appreciate all the good feedback from the community.</p></div>
</div>I don't think you miss anything, broken html content is probably the result of failed (or partially successful) copy/paste by the model author on civitai.
I think that this tool should not try to fix the errors on civitai site - it should be ok to save the downloaded data as is.
Re: tidy, it's an html sanitizer.
Here is the source https://github.com/htacg/tidy-html5
Docs are here: https://www.html-tidy.org/documentation/
That was just an example of if e.g. I would sometime in the future need a nice clean text description extracted from the json metadata then it's much simpler to do something like this:
$ sudo apt install jq tidy html2text
$ jq .description ./extra_data-vid_293240/model_dict-mid_152525-vid_293240.json \
| tidy \
-indent \
--indent-spaces 2 \
-quiet \
--tidy-mark no \
2>/dev/null \
| html2text
"
***** Version 3.0 of the Realism Engine *****
In this version we improved the skin realism, the eyes. We also improved the
Male anatomy.
Keep in mind that this is not a NSFW model. For that I recommend SDXXXL.
***** Introducing Version 2.0 of the Realism Engine *****
We are excited to announce that Version 2.0 of the Realism Engine is now
available, featuring significant enhancements.Than to try to process all possible variants during download?
OwenTruong
commented
7 months ago Keeping this open in case I missed a bug
Got the UnicodeEncodeError on Windows for latest master branch (at the moment 537dc51880ee0cc636eb721b1b8fe9cab003bff6):
Changing the environment to UTF-8 (
chcp 65001for windows) did not fix this.Maybe it'd be a good idea to write everything as downloaded, in binary mode?
Or even better, encode all text files to utf8 by default?