Trying to run the downloadable deepdanbooru-v3/v4 always ends up in the same error
Loading deepdanbooru-v3-20211112-sgd-e28 from <DirEntry 'deepdanbooru-v3-20211112-sgd-e28'> 2024-03-30 17:12:30.152946: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 2024-03-30 17:12:30.174784: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 2024-03-30 17:12:30.178338: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 2024-03-30 17:12:30.182152: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 2024-03-30 17:12:30.185918: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 2024-03-30 17:12:30.189107: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 2024-03-30 17:12:30.192417: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 2024-03-30 17:12:30.195551: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 2024-03-30 17:12:30.199125: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 2024-03-30 17:12:30.202313: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6744 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3060, pci bus id: 0000:10:00.0, compute capability: 8.6 Loaded deepdanbooru-v3-20211112-sgd-e28 model from <DirEntry 'deepdanbooru-v3-20211112-sgd-e28'> 2024-03-30 17:12:38.330697: W external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:504] Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice. This may result in compilation or runtime failures, if the program we try to run uses routines from libdevice. Searched for CUDA in the following directories: ./cuda_sdk_lib /usr/local/cuda-12.2 /usr/local/cuda /mnt/ts512/stable-diffusion-webui-forge/venv/lib/python3.10/site-packages/tensorflow/python/platform/../../../nvidia/cuda_nvcc /mnt/ts512/stable-diffusion-webui-forge/venv/lib/python3.10/site-packages/tensorflow/python/platform/../../../../nvidia/cuda_nvcc You can choose the search directory by setting xla_gpu_cuda_data_dir in HloModule's DebugOptions. For most apps, setting the environment variable XLA_FLAGS=--xla_gpu_cuda_data_dir=/path/to/cuda will work. 2024-03-30 17:12:38.516224: W external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:542] libdevice is required by this HLO module but was not found at ./libdevice.10.bc error: libdevice not found at ./libdevice.10.bc 2024-03-30 17:12:38.516496: E tensorflow/compiler/mlir/tools/kernel_gen/tf_framework_c_interface.cc:207] INTERNAL: Generating device code failed. 2024-03-30 17:12:38.516839: W tensorflow/core/framework/op_kernel.cc:1827] UNKNOWN: JIT compilation failed. *** Error completing request *** Arguments: (<PIL.Image.Image image mode=RGB size=850x1363 at 0x7B5A310A8BE0>, 'deepdanbooru-v3-20211112-sgd-e28', 'ba', '', '', '', '', '') {} Traceback (most recent call last): File "/mnt/ts512/stable-diffusion-webui-forge/modules/call_queue.py", line 57, in f res = list(func(*args, **kwargs)) File "/mnt/ts512/stable-diffusion-webui-forge/modules/call_queue.py", line 36, in f res = func(*args, **kwargs) File "/mnt/ts512/stable-diffusion-webui-forge/extensions/stable-diffusion-webui-wd14-tagger/tagger/ui.py", line 113, in on_interrogate_image_submit interrogator.interrogate_image(image) File "/mnt/ts512/stable-diffusion-webui-forge/extensions/stable-diffusion-webui-wd14-tagger/tagger/interrogator.py", line 150, in interrogate_image data = ('', '', fi_key) + self.interrogate(image) File "/mnt/ts512/stable-diffusion-webui-forge/extensions/stable-diffusion-webui-wd14-tagger/tagger/interrogator.py", line 309, in interrogate image = ddd.load_image_for_evaluate( File "/mnt/ts512/stable-diffusion-webui-forge/venv/lib/python3.10/site-packages/deepdanbooru/data/__init__.py", line 26, in load_image_for_evaluate image = tf.image.resize( File "/mnt/ts512/stable-diffusion-webui-forge/venv/lib/python3.10/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler raise e.with_traceback(filtered_tb) from None File "/mnt/ts512/stable-diffusion-webui-forge/venv/lib/python3.10/site-packages/tensorflow/python/framework/ops.py", line 5883, in raise_from_not_ok_status raise core._status_to_exception(e) from None # pylint: disable=protected-access tensorflow.python.framework.errors_impl.UnknownError: {{function_node __wrapped__Round_device_/job:localhost/replica:0/task:0/device:GPU:0}} JIT compilation failed. [Op:Round] name:
Traceback (most recent call last): File "/mnt/ts512/stable-diffusion-webui-forge/venv/lib/python3.10/site-packages/gradio/routes.py", line 488, in run_predict output = await app.get_blocks().process_api( File "/mnt/ts512/stable-diffusion-webui-forge/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1434, in process_api data = self.postprocess_data(fn_index, result["prediction"], state) File "/mnt/ts512/stable-diffusion-webui-forge/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1297, in postprocess_data self.validate_outputs(fn_index, predictions) # type: ignore File "/mnt/ts512/stable-diffusion-webui-forge/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1272, in validate_outputs raise ValueError( ValueError: An event handler (on_interrogate_image_submit) didn't receive enough output values (needed: 7, received: 3). Wanted outputs: [state, html, html, label, label, label, html] Received outputs: [None, "", "<div class='error'>UnknownError: {{function_node __wrapped__Round_device_/job:localhost/replica:0/task:0/device:GPU:0}} JIT compilation failed. [Op:Round] name: </div><div class='performance'><p class='time'>Time taken: <wbr><span class='measurement'>8.4 sec.</span></p><p class='vram'><abbr title='Active: peak amount of video memory used during generation (excluding cached data)'>A</abbr>: <span class='measurement'>1.73 GB</span>, <wbr><abbr title='Reserved: total amout of video memory allocated by the Torch library '>R</abbr>: <span class='measurement'>1.75 GB</span>, <wbr><abbr title='System: peak amout of video memory allocated by all running programs, out of total capacity'>Sys</abbr>: <span class='measurement'>5.5/11.5391 GB</span> (47.9%)</p></div>"]
Trying to run the downloadable
deepdanbooru-v3/v4always ends up in the same errorLoading deepdanbooru-v3-20211112-sgd-e28 from <DirEntry 'deepdanbooru-v3-20211112-sgd-e28'> 2024-03-30 17:12:30.152946: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 2024-03-30 17:12:30.174784: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 2024-03-30 17:12:30.178338: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 2024-03-30 17:12:30.182152: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 2024-03-30 17:12:30.185918: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 2024-03-30 17:12:30.189107: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 2024-03-30 17:12:30.192417: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 2024-03-30 17:12:30.195551: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 2024-03-30 17:12:30.199125: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 2024-03-30 17:12:30.202313: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6744 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3060, pci bus id: 0000:10:00.0, compute capability: 8.6 Loaded deepdanbooru-v3-20211112-sgd-e28 model from <DirEntry 'deepdanbooru-v3-20211112-sgd-e28'> 2024-03-30 17:12:38.330697: W external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:504] Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice. This may result in compilation or runtime failures, if the program we try to run uses routines from libdevice. Searched for CUDA in the following directories: ./cuda_sdk_lib /usr/local/cuda-12.2 /usr/local/cuda /mnt/ts512/stable-diffusion-webui-forge/venv/lib/python3.10/site-packages/tensorflow/python/platform/../../../nvidia/cuda_nvcc /mnt/ts512/stable-diffusion-webui-forge/venv/lib/python3.10/site-packages/tensorflow/python/platform/../../../../nvidia/cuda_nvcc You can choose the search directory by setting xla_gpu_cuda_data_dir in HloModule's DebugOptions. For most apps, setting the environment variable XLA_FLAGS=--xla_gpu_cuda_data_dir=/path/to/cuda will work. 2024-03-30 17:12:38.516224: W external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:542] libdevice is required by this HLO module but was not found at ./libdevice.10.bc error: libdevice not found at ./libdevice.10.bc 2024-03-30 17:12:38.516496: E tensorflow/compiler/mlir/tools/kernel_gen/tf_framework_c_interface.cc:207] INTERNAL: Generating device code failed. 2024-03-30 17:12:38.516839: W tensorflow/core/framework/op_kernel.cc:1827] UNKNOWN: JIT compilation failed. *** Error completing request *** Arguments: (<PIL.Image.Image image mode=RGB size=850x1363 at 0x7B5A310A8BE0>, 'deepdanbooru-v3-20211112-sgd-e28', 'ba', '', '', '', '', '') {} Traceback (most recent call last): File "/mnt/ts512/stable-diffusion-webui-forge/modules/call_queue.py", line 57, in f res = list(func(*args, **kwargs)) File "/mnt/ts512/stable-diffusion-webui-forge/modules/call_queue.py", line 36, in f res = func(*args, **kwargs) File "/mnt/ts512/stable-diffusion-webui-forge/extensions/stable-diffusion-webui-wd14-tagger/tagger/ui.py", line 113, in on_interrogate_image_submit interrogator.interrogate_image(image) File "/mnt/ts512/stable-diffusion-webui-forge/extensions/stable-diffusion-webui-wd14-tagger/tagger/interrogator.py", line 150, in interrogate_image data = ('', '', fi_key) + self.interrogate(image) File "/mnt/ts512/stable-diffusion-webui-forge/extensions/stable-diffusion-webui-wd14-tagger/tagger/interrogator.py", line 309, in interrogate image = ddd.load_image_for_evaluate( File "/mnt/ts512/stable-diffusion-webui-forge/venv/lib/python3.10/site-packages/deepdanbooru/data/__init__.py", line 26, in load_image_for_evaluate image = tf.image.resize( File "/mnt/ts512/stable-diffusion-webui-forge/venv/lib/python3.10/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler raise e.with_traceback(filtered_tb) from None File "/mnt/ts512/stable-diffusion-webui-forge/venv/lib/python3.10/site-packages/tensorflow/python/framework/ops.py", line 5883, in raise_from_not_ok_status raise core._status_to_exception(e) from None # pylint: disable=protected-access tensorflow.python.framework.errors_impl.UnknownError: {{function_node __wrapped__Round_device_/job:localhost/replica:0/task:0/device:GPU:0}} JIT compilation failed. [Op:Round] name:Traceback (most recent call last): File "/mnt/ts512/stable-diffusion-webui-forge/venv/lib/python3.10/site-packages/gradio/routes.py", line 488, in run_predict output = await app.get_blocks().process_api( File "/mnt/ts512/stable-diffusion-webui-forge/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1434, in process_api data = self.postprocess_data(fn_index, result["prediction"], state) File "/mnt/ts512/stable-diffusion-webui-forge/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1297, in postprocess_data self.validate_outputs(fn_index, predictions) # type: ignore File "/mnt/ts512/stable-diffusion-webui-forge/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1272, in validate_outputs raise ValueError( ValueError: An event handler (on_interrogate_image_submit) didn't receive enough output values (needed: 7, received: 3). Wanted outputs: [state, html, html, label, label, label, html] Received outputs: [None, "", "<div class='error'>UnknownError: {{function_node __wrapped__Round_device_/job:localhost/replica:0/task:0/device:GPU:0}} JIT compilation failed. [Op:Round] name: </div><div class='performance'><p class='time'>Time taken: <wbr><span class='measurement'>8.4 sec.</span></p><p class='vram'><abbr title='Active: peak amount of video memory used during generation (excluding cached data)'>A</abbr>: <span class='measurement'>1.73 GB</span>, <wbr><abbr title='Reserved: total amout of video memory allocated by the Torch library '>R</abbr>: <span class='measurement'>1.75 GB</span>, <wbr><abbr title='System: peak amout of video memory allocated by all running programs, out of total capacity'>Sys</abbr>: <span class='measurement'>5.5/11.5391 GB</span> (47.9%)</p></div>"]