search

AUTOMATIC1111 / stable-diffusion-webui

Stable Diffusion web UI
GNU Affero General Public License v3.0
141.51k stars 26.74k forks source link

[Bug]: Tensorflow error in Colab #7791

Open mykeehu opened 1 year ago

mykeehu commented 1 year ago

Is there an existing issue for this?

What happened?

I started the SD in Colab and got this error after loading the arguments:

2023-02-13 12:24:16.140877: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-02-13 12:24:18.687780: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.8/dist-packages/cv2/../../lib64:/usr/lib64-nvidia
2023-02-13 12:24:18.688049: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.8/dist-packages/cv2/../../lib64:/usr/lib64-nvidia

Steps to reproduce the problem

  1. Start SD in Colab.
  2. See Console

What should have happened?

I don't know what this bug is, but it's not good.

Commit where the problem happens

3715ece0adce7bf7c5e9c5ab3710b2fdc3848f39

What platforms do you use to access the UI ?

Other/Cloud

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

"--share --gradio-queue --enable-insecure-extension-access --gradio-auth mykee:diffusion --ckpt-dir /content/drive/MyDrive/SDmodels --vae-dir /content/drive/MyDrive/VAE --lora-dir /content/drive/MyDrive/Lora --esrgan-models-path /content/drive/MyDrive/ESRGAN --ui-config-file /content/drive/MyDrive/1my/ui-config-my.json --ui-settings-file /content/drive/MyDrive/1my/config-my.json --styles-file /content/drive/MyDrive/1my/styles.csv --embeddings-dir /content/drive/MyDrive/TI-Embeddings --hypernetwork-dir /content/drive/MyDrive/Hypernetworks --gradio-img2img-tool color-sketch" REQS_FILE="requirements.txt" python launch.py

List of extensions

None

Console logs

Python 3.8.10 (default, Nov 14 2022, 12:59:47) 
[GCC 9.4.0]
Commit hash: ea9bd9fc7409109adcd61b897abc2c8881161256
Installing requirements for Web UI
Launching Web UI with arguments: --share --gradio-queue --enable-insecure-extension-access --gradio-auth mykee:diffusion --ckpt-dir /content/drive/MyDrive/SDmodels --vae-dir /content/drive/MyDrive/VAE --lora-dir /content/drive/MyDrive/Lora --esrgan-models-path /content/drive/MyDrive/ESRGAN --ui-config-file /content/drive/MyDrive/1my/ui-config-my.json --ui-settings-file /content/drive/MyDrive/1my/config-my.json --styles-file /content/drive/MyDrive/1my/styles.csv --embeddings-dir /content/drive/MyDrive/TI-Embeddings --hypernetwork-dir /content/drive/MyDrive/Hypernetworks --gradio-img2img-tool color-sketch
2023-02-13 12:34:43.647542: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-02-13 12:34:46.242366: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.8/dist-packages/cv2/../../lib64:/usr/lib64-nvidia
2023-02-13 12:34:46.242640: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.8/dist-packages/cv2/../../lib64:/usr/lib64-nvidia
2023-02-13 12:34:46.242669: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
No module 'xformers'. Proceeding without it.
Checkpoint !MyMixed/oof85a_11050-theAllySMixIIChurned-mix90-fixed.safetensors [329141fd49] not found; loading fallback model.ckpt [fe4efff1e1]
Loading weights [fe4efff1e1] from /content/drive/MyDrive/SDmodels/model.ckpt
Creating model from config: /content/stable-diffusion-webui/configs/v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Loading VAE weights specified in settings: /content/drive/MyDrive/VAE/SD-1.5-vae-ft-mse-840000-ema-pruned.pt
Applying cross attention optimization (Doggettx).
Textual inversion embeddings loaded(0): 
Model loaded in 32.0s (load weights from disk: 19.3s, create model: 0.9s, apply weights to model: 2.0s, apply half(): 1.2s, load VAE: 8.0s, move model to device: 0.7s).
/usr/local/lib/python3.8/dist-packages/gradio/deprecation.py:43: UserWarning: You have unused kwarg parameters in Slider, please remove them: {'min': 0, 'max': 500}
  warnings.warn(
Running on local URL:  http://127.0.0.1:7860
Running on public URL: https://5843f040-d3c8-4841.gradio.live

This share link expires in 72 hours. For free permanent hosting and GPU upgrades (NEW!), check out Spaces: https://huggingface.co/spaces
Traceback (most recent call last):
  File "launch.py", line 361, in <module>
    start()
  File "launch.py", line 356, in start
    webui.webui()
  File "/content/stable-diffusion-webui/webui.py", line 232, in webui
    app.add_middleware(GZipMiddleware, minimum_size=1000)
  File "/usr/local/lib/python3.8/dist-packages/starlette/applications.py", line 135, in add_middleware
    raise RuntimeError("Cannot add middleware after an application has started")
RuntimeError: Cannot add middleware after an application has started
Killing tunnel 127.0.0.1:7860 <> https://5843f040-d3c8-4841.gradio.live

Additional information

No response

GrennKren commented 1 year ago

Because your commit was ea9bd9f and not the most recent one, 3715ece. you should update your webui.

mykeehu commented 1 year ago

@GrennKren Anyway, 3715ece also gives the same error

vt-idiot commented 1 year ago 
[GCC 9.4.0]
Commit hash: ea9bd9fc7409109adcd61b897abc2c8881161256
Installing requirements for Web UI
Launching Web UI with arguments: --share --gradio-auth x --xformers --api --gradio-queue --enable-insecure-extension-access --no-half-vae
2023-02-13 11:51:55.255746: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-02-13 11:51:58.983494: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.8/dist-packages/cv2/../../lib64:/usr/lib64-nvidia
2023-02-13 11:51:58.984356: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.8/dist-packages/cv2/../../lib64:/usr/lib64-nvidia
2023-02-13 11:51:58.984388: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
Loading weights [d179dccea9] from /content/stable-diffusion-webui/models/

but you're not using the latest commit either

the two commits after this are just changes to requirements.txt for the fastapi bug, which anyone successfully using it on colab was already working around with a !pip install... command.

vt-idiot commented 1 year ago

I tried some stuff I thought would make the warning (W - it's a warning, not an error) go away, but it requires a lot of shenanigans and tomfoolery. And I'm honestly not even sure it's necessary.

!apt-get install -y libnvinfer-dev=8.4.3-1+cuda11.6 libnvinfer-plugin-dev=8.4.3-1+cuda11.6 libnvparsers-dev=8.4.3-1+cuda11.6 libnvonnxparsers-dev=8.4.3-1+cuda11.6 python3-libnvinfer-dev=8.4.3-1+cuda11.6 libnvinfer8=8.4.3-1+cuda11.6 libnvinfer-plugin8=8.4.3-1+cuda11.6 libnvparsers8=8.4.3-1+cuda11.6 libnvonnxparsers8=8.4.3-1+cuda11.6 python3-libnvinfer=8.4.3-1+cuda11.6
!pip install protobuf
!apt-get install -y graphsurgeon-tf=8.4.3-1+cuda11.6 uff-converter-tf=8.4.3-1+cuda11.6
!ln -s /usr/lib/x86_64-linux-gnu/libnvinfer.so.8.4.3 /usr/lib64-nvidia/libnvinfer.so.7
!ln -s /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.8.4.3 /usr/lib64-nvidia/libnvinfer_plugin.so.7

It does work though, warnings are both gone. Now I just get the info message about AVX

Python 3.8.10 (default, Nov 14 2022, 12:59:47) 
[GCC 9.4.0]
Commit hash: 3715ece0adce7bf7c5e9c5ab3710b2fdc3848f39
Installing requirements for Web UI
Launching Web UI with arguments: --share --gradio-auth xx:xx --xformers --api --gradio-queue --enable-insecure-extension-access --no-half-vae
2023-02-13 19:27:56.040768: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Loading weights [d179dccea9] from /content/stable-diffusion-webui/models/...
LagNeto commented 1 year ago

i fixed those Warnings by doing: !pip install -U tensorflow==2.9.2 !pip install fastapi==0.90.0

SteakTree commented 1 year ago

@vt-idiot I added that code to my colab notebook before launching SD and that seemed to work. Took a bit longer to launch, and left me with a with this message:

I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags

Not sure if that is critical though. Thanks for the workaround.

vt-idiot commented 1 year ago

No problem. Like I said, I've noticed no difference in image generation or in performance with or without running that bit of code to "fix" it, I don't use it myself since the extra couple of minutes it takes seem like a waste. The W after the timestamp means those are Warnings, not errors; "generative" (txt2img, img2img, etc) usage of webui doesn't seem to rely on libvinfer for anything at all. libvinfer - as in "inference". If it's used for anything at all, it's only for training. And I imagine the authors of Colab notebooks specifically meant for training checkpoints/Dreambooth/embeddings/Lora already have their bases covered.

2023-02-13 11:51:58.983494: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.8/dist-packages/cv2/../../lib64:/usr/lib64-nvidia
2023-02-13 11:51:58.984356: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.8/dist-packages/cv2/../../lib64:/usr/lib64-nvidia
2023-02-13 11:51:58.984388: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.

The remaining message you see begins with I - information. Tensorflow is just letting you know it's only using AVX2/FMA (SIMD instructions - faster CPU instructions for CPUs newer than Ivy Bridge) in "performance-critical operations" - but not everywhere. I imagine it'd hardly be worth recompiling TensorFlow from source with different compiler flags just to make it go away lol.