Yesterday, our system was still running normally, but today when we tried to use it, we encountered an error report, after a preliminary investigation, we suspect that this may be due to some packages installed in the environment caused by dependency conflicts.
Colab cell output
📂 Connecting to Google Drive...
Mounted at /content/drive
💿 Checking dataset...
📁MyDrive/Loras/bangbangv3/dataset
📈 Found 181 images with 10 repeats, equaling 1810 steps.
📉 Divide 1810 steps by 2 batch size to get 905.0 steps per epoch.
🔮 There will be 10 epochs, for around 9050 total training steps.
🏭 Installing dependencies...
Cloning into '/content/kohya-trainer'...
remote: Enumerating objects: 5921, done.
remote: Counting objects: 100% (2786/2786), done.
remote: Compressing objects: 100% (359/359), done.
remote: Total 5921 (delta 2619), reused 2433 (delta 2427), pack-reused 3135
Receiving objects: 100% (5921/5921), 9.37 MiB | 14.79 MiB/s, done.
Resolving deltas: 100% (4207/4207), done.
HEAD is now at 9a67e0d Merge pull request #610 from lubobill1990/patch-1
38 packages can be upgraded. Run 'apt list --upgradable' to see them.
The following additional packages will be installed:
libaria2-0 libc-ares2
The following NEW packages will be installed:
aria2 libaria2-0 libc-ares2
0 upgraded, 3 newly installed, 0 to remove and 38 not upgraded.
Need to get 1,513 kB of archives.
After this operation, 5,441 kB of additional disk space will be used.
Selecting previously unselected package libc-ares2:amd64.
(Reading database ... 121752 files and directories currently installed.)
Preparing to unpack .../libc-ares2_1.18.1-1ubuntu0.22.04.3_amd64.deb ...
Unpacking libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.3) ...
Selecting previously unselected package libaria2-0:amd64.
Preparing to unpack .../libaria2-0_1.36.0-1_amd64.deb ...
Unpacking libaria2-0:amd64 (1.36.0-1) ...
Selecting previously unselected package aria2.
Preparing to unpack .../aria2_1.36.0-1_amd64.deb ...
Unpacking aria2 (1.36.0-1) ...
Setting up libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.3) ...
Setting up libaria2-0:amd64 (1.36.0-1) ...
Setting up aria2 (1.36.0-1) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.4) ...
/sbin/ldconfig.real: /usr/local/lib/libtbb.so.12 is not a symbolic link
/sbin/ldconfig.real: /usr/local/lib/libtbbbind.so.3 is not a symbolic link
/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_0.so.3 is not a symbolic link
/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc_proxy.so.2 is not a symbolic link
/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc.so.2 is not a symbolic link
/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_5.so.3 is not a symbolic link
Processing /content/kohya-trainer
Preparing metadata (setup.py) ... done
Collecting accelerate==0.15.0 (from -r requirements.txt (line 1))
Downloading accelerate-0.15.0-py3-none-any.whl (191 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 191.5/191.5 kB 3.0 MB/s eta 0:00:00
Collecting transformers==4.26.0 (from -r requirements.txt (line 2))
Downloading transformers-4.26.0-py3-none-any.whl (6.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.3/6.3 MB 62.7 MB/s eta 0:00:00
Collecting ftfy==6.1.1 (from -r requirements.txt (line 3))
Downloading ftfy-6.1.1-py3-none-any.whl (53 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.1/53.1 kB 7.2 MB/s eta 0:00:00
Requirement already satisfied: albumentations==1.3.1 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 4)) (1.3.1)
Requirement already satisfied: opencv-python==4.8.0.76 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 5)) (4.8.0.76)
Collecting einops==0.6.0 (from -r requirements.txt (line 6))
Downloading einops-0.6.0-py3-none-any.whl (41 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.6/41.6 kB 6.1 MB/s eta 0:00:00
Collecting diffusers==0.10.2 (from -r requirements.txt (line 7))
Downloading diffusers-0.10.2-py3-none-any.whl (503 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 503.1/503.1 kB 49.8 MB/s eta 0:00:00
Collecting pytorch-lightning==1.9.0 (from -r requirements.txt (line 8))
Downloading pytorch_lightning-1.9.0-py3-none-any.whl (825 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 825.8/825.8 kB 69.0 MB/s eta 0:00:00
Collecting bitsandbytes==0.41.3.post2 (from -r requirements.txt (line 9))
Downloading bitsandbytes-0.41.3.post2-py3-none-any.whl (92.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 92.6/92.6 MB 8.0 MB/s eta 0:00:00
Collecting tensorflow==2.14.0 (from -r requirements.txt (line 10))
Downloading tensorflow-2.14.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (489.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 489.8/489.8 MB 2.7 MB/s eta 0:00:00
Collecting safetensors==0.4.1 (from -r requirements.txt (line 11))
Downloading safetensors-0.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 76.2 MB/s eta 0:00:00
Requirement already satisfied: toml==0.10.2 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 12)) (0.10.2)
Collecting voluptuous==0.13.1 (from -r requirements.txt (line 13))
Downloading voluptuous-0.13.1-py3-none-any.whl (29 kB)
Collecting lion_pytorch==0.0.6 (from -r requirements.txt (line 14))
Downloading lion_pytorch-0.0.6-py3-none-any.whl (4.2 kB)
Collecting dadaptation==3.1 (from -r requirements.txt (line 15))
Downloading dadaptation-3.1.tar.gz (13 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting prodigyopt==1.0 (from -r requirements.txt (line 16))
Downloading prodigyopt-1.0-py3-none-any.whl (5.5 kB)
Collecting lycoris_lora==0.1.4 (from -r requirements.txt (line 17))
Downloading lycoris_lora-0.1.4.tar.gz (37 kB)
Preparing metadata (setup.py) ... done
Collecting xformers==0.0.22.post7 (from -r requirements.txt (line 18))
Downloading xformers-0.0.22.post7-cp310-cp310-manylinux2014_x86_64.whl (211.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 211.8/211.8 MB 2.4 MB/s eta 0:00:00
Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.15.0->-r requirements.txt (line 1)) (1.25.2)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.15.0->-r requirements.txt (line 1)) (24.0)
Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate==0.15.0->-r requirements.txt (line 1)) (5.9.5)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from accelerate==0.15.0->-r requirements.txt (line 1)) (6.0.1)
Requirement already satisfied: torch>=1.4.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.15.0->-r requirements.txt (line 1)) (2.2.1+cu121)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers==4.26.0->-r requirements.txt (line 2)) (3.13.1)
Requirement already satisfied: huggingface-hub<1.0,>=0.11.0 in /usr/local/lib/python3.10/dist-packages (from transformers==4.26.0->-r requirements.txt (line 2)) (0.20.3)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.26.0->-r requirements.txt (line 2)) (2023.12.25)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers==4.26.0->-r requirements.txt (line 2)) (2.31.0)
Collecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers==4.26.0->-r requirements.txt (line 2))
Downloading tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.8/7.8 MB 94.4 MB/s eta 0:00:00
Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers==4.26.0->-r requirements.txt (line 2)) (4.66.2)
Requirement already satisfied: wcwidth>=0.2.5 in /usr/local/lib/python3.10/dist-packages (from ftfy==6.1.1->-r requirements.txt (line 3)) (0.2.13)
Requirement already satisfied: scipy>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from albumentations==1.3.1->-r requirements.txt (line 4)) (1.11.4)
Requirement already satisfied: scikit-image>=0.16.1 in /usr/local/lib/python3.10/dist-packages (from albumentations==1.3.1->-r requirements.txt (line 4)) (0.19.3)
Requirement already satisfied: qudida>=0.0.4 in /usr/local/lib/python3.10/dist-packages (from albumentations==1.3.1->-r requirements.txt (line 4)) (0.0.4)
Requirement already satisfied: opencv-python-headless>=4.1.1 in /usr/local/lib/python3.10/dist-packages (from albumentations==1.3.1->-r requirements.txt (line 4)) (4.9.0.80)
Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.10/dist-packages (from diffusers==0.10.2->-r requirements.txt (line 7)) (7.0.2)
Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from diffusers==0.10.2->-r requirements.txt (line 7)) (9.4.0)
Requirement already satisfied: fsspec[http]>2021.06.0 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (2023.6.0)
Collecting torchmetrics>=0.7.0 (from pytorch-lightning==1.9.0->-r requirements.txt (line 8))
Downloading torchmetrics-1.3.1-py3-none-any.whl (840 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 840.4/840.4 kB 68.2 MB/s eta 0:00:00
Requirement already satisfied: typing-extensions>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (4.10.0)
Collecting lightning-utilities>=0.4.2 (from pytorch-lightning==1.9.0->-r requirements.txt (line 8))
Downloading lightning_utilities-0.10.1-py3-none-any.whl (24 kB)
Requirement already satisfied: absl-py>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (1.4.0)
Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (1.6.3)
Requirement already satisfied: flatbuffers>=23.5.26 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (24.3.7)
Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (0.5.4)
Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (0.2.0)
Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (3.9.0)
Requirement already satisfied: libclang>=13.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (16.0.6)
Requirement already satisfied: ml-dtypes==0.2.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (0.2.0)
Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (3.3.0)
Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (3.20.3)
Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (67.7.2)
Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (1.16.0)
Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (2.4.0)
Requirement already satisfied: wrapt<1.15,>=1.11.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (1.14.1)
Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (0.36.0)
Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (1.62.1)
Collecting tensorboard<2.15,>=2.14 (from tensorflow==2.14.0->-r requirements.txt (line 10))
Downloading tensorboard-2.14.1-py3-none-any.whl (5.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.5/5.5 MB 95.3 MB/s eta 0:00:00
Collecting tensorflow-estimator<2.15,>=2.14.0 (from tensorflow==2.14.0->-r requirements.txt (line 10))
Downloading tensorflow_estimator-2.14.0-py2.py3-none-any.whl (440 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 440.7/440.7 kB 47.1 MB/s eta 0:00:00
Collecting keras<2.15,>=2.14.0 (from tensorflow==2.14.0->-r requirements.txt (line 10))
Downloading keras-2.14.0-py3-none-any.whl (1.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 64.0 MB/s eta 0:00:00
Collecting torch>=1.4.0 (from accelerate==0.15.0->-r requirements.txt (line 1))
Downloading torch-2.1.0-cp310-cp310-manylinux1_x86_64.whl (670.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 670.2/670.2 MB 2.0 MB/s eta 0:00:00
Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) (1.12)
Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) (3.2.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) (3.1.3)
Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
Downloading nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23.7/23.7 MB 35.0 MB/s eta 0:00:00
Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
Downloading nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 823.6/823.6 kB 54.4 MB/s eta 0:00:00
Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
Downloading nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.1/14.1 MB 55.2 MB/s eta 0:00:00
Collecting nvidia-cudnn-cu12==8.9.2.26 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
Downloading nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 731.7/731.7 MB 2.1 MB/s eta 0:00:00
Collecting nvidia-cublas-cu12==12.1.3.1 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
Downloading nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 410.6/410.6 MB 3.0 MB/s eta 0:00:00
Collecting nvidia-cufft-cu12==11.0.2.54 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
Downloading nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.6/121.6 MB 7.3 MB/s eta 0:00:00
Collecting nvidia-curand-cu12==10.3.2.106 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
Downloading nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.5/56.5 MB 11.2 MB/s eta 0:00:00
Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
Downloading nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 124.2/124.2 MB 7.6 MB/s eta 0:00:00
Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
Downloading nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 196.0/196.0 MB 3.1 MB/s eta 0:00:00
Collecting nvidia-nccl-cu12==2.18.1 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
Downloading nvidia_nccl_cu12-2.18.1-py3-none-manylinux1_x86_64.whl (209.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 209.8/209.8 MB 4.4 MB/s eta 0:00:00
Collecting nvidia-nvtx-cu12==12.1.105 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
Downloading nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 99.1/99.1 kB 16.3 MB/s eta 0:00:00
Collecting triton==2.1.0 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
Downloading triton-2.1.0-0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (89.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 89.2/89.2 MB 8.0 MB/s eta 0:00:00
Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
Downloading nvidia_nvjitlink_cu12-12.4.99-py3-none-manylinux2014_x86_64.whl (21.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.1/21.1 MB 71.9 MB/s eta 0:00:00
Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from astunparse>=1.6.0->tensorflow==2.14.0->-r requirements.txt (line 10)) (0.43.0)
Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /usr/local/lib/python3.10/dist-packages (from fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (3.9.3)
Requirement already satisfied: scikit-learn>=0.19.1 in /usr/local/lib/python3.10/dist-packages (from qudida>=0.0.4->albumentations==1.3.1->-r requirements.txt (line 4)) (1.2.2)
Requirement already satisfied: imageio>=2.4.1 in /usr/local/lib/python3.10/dist-packages (from scikit-image>=0.16.1->albumentations==1.3.1->-r requirements.txt (line 4)) (2.31.6)
Requirement already satisfied: tifffile>=2019.7.26 in /usr/local/lib/python3.10/dist-packages (from scikit-image>=0.16.1->albumentations==1.3.1->-r requirements.txt (line 4)) (2024.2.12)
Requirement already satisfied: PyWavelets>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-image>=0.16.1->albumentations==1.3.1->-r requirements.txt (line 4)) (1.5.0)
Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (2.27.0)
Collecting google-auth-oauthlib<1.1,>=0.5 (from tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10))
Downloading google_auth_oauthlib-1.0.0-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (3.5.2)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (0.7.2)
Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (3.0.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.26.0->-r requirements.txt (line 2)) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.26.0->-r requirements.txt (line 2)) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.26.0->-r requirements.txt (line 2)) (2.0.7)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.26.0->-r requirements.txt (line 2)) (2024.2.2)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata->diffusers==0.10.2->-r requirements.txt (line 7)) (3.17.0)
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (1.3.1)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (23.2.0)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (1.4.1)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (6.0.5)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (1.9.4)
Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (4.0.3)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (5.3.3)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (0.3.0)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (4.9)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<1.1,>=0.5->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (1.4.0)
Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.19.1->qudida>=0.0.4->albumentations==1.3.1->-r requirements.txt (line 4)) (1.3.2)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.19.1->qudida>=0.0.4->albumentations==1.3.1->-r requirements.txt (line 4)) (3.3.0)
Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/dist-packages (from werkzeug>=1.0.1->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (2.1.5)
Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) (1.3.0)
Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (0.5.1)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<1.1,>=0.5->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (3.2.2)
Building wheels for collected packages: dadaptation, lycoris_lora, library
Building wheel for dadaptation (pyproject.toml) ... done
Created wheel for dadaptation: filename=dadaptation-3.1-py3-none-any.whl size=22788 sha256=711cb41973d95c7ce0e4bd199fcb74b3dec8cff2c9eb399bcc1d56ce581e7dc2
Stored in directory: /root/.cache/pip/wheels/34/25/92/b7708738900a69a166f41de56c776039b5a05599b36075c34f
Building wheel for lycoris_lora (setup.py) ... done
Created wheel for lycoris_lora: filename=lycoris_lora-0.1.4-py3-none-any.whl size=26134 sha256=7f3f78d978836dd99f155c2acd8f963970f9e5022cf1026ec4f169f727f98edd
Stored in directory: /root/.cache/pip/wheels/65/d4/14/c7a61ec6ce0398272767c355bd02af1aa309cecd7102b22080
Building wheel for library (setup.py) ... done
Created wheel for library: filename=library-0.0.0-py3-none-any.whl size=84081 sha256=6955abd0aed633ab95d29096622dee016fa7b133014b2663ea3403cba9c8134d
Stored in directory: /root/.cache/pip/wheels/d8/ef/ac/94db7d75dc3a8e81533bc2d001d79467c693ec69ed02bbee59
Successfully built dadaptation lycoris_lora library
Installing collected packages: voluptuous, tokenizers, library, bitsandbytes, triton, tensorflow-estimator, safetensors, prodigyopt, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, lightning-utilities, keras, ftfy, einops, dadaptation, nvidia-cusparse-cu12, nvidia-cudnn-cu12, transformers, nvidia-cusolver-cu12, google-auth-oauthlib, diffusers, torch, tensorboard, xformers, torchmetrics, tensorflow, lycoris_lora, lion_pytorch, accelerate, pytorch-lightning
Attempting uninstall: tokenizers
Found existing installation: tokenizers 0.15.2
Uninstalling tokenizers-0.15.2:
Successfully uninstalled tokenizers-0.15.2
Attempting uninstall: triton
Found existing installation: triton 2.2.0
Uninstalling triton-2.2.0:
Successfully uninstalled triton-2.2.0
Attempting uninstall: tensorflow-estimator
Found existing installation: tensorflow-estimator 2.15.0
Uninstalling tensorflow-estimator-2.15.0:
Successfully uninstalled tensorflow-estimator-2.15.0
Attempting uninstall: safetensors
Found existing installation: safetensors 0.4.2
Uninstalling safetensors-0.4.2:
Successfully uninstalled safetensors-0.4.2
Attempting uninstall: keras
Found existing installation: keras 2.15.0
Uninstalling keras-2.15.0:
Successfully uninstalled keras-2.15.0
Attempting uninstall: transformers
Found existing installation: transformers 4.38.2
Uninstalling transformers-4.38.2:
Successfully uninstalled transformers-4.38.2
Attempting uninstall: google-auth-oauthlib
Found existing installation: google-auth-oauthlib 1.2.0
Uninstalling google-auth-oauthlib-1.2.0:
Successfully uninstalled google-auth-oauthlib-1.2.0
Attempting uninstall: torch
Found existing installation: torch 2.2.1+cu121
Uninstalling torch-2.2.1+cu121:
Successfully uninstalled torch-2.2.1+cu121
Attempting uninstall: tensorboard
Found existing installation: tensorboard 2.15.2
Uninstalling tensorboard-2.15.2:
Successfully uninstalled tensorboard-2.15.2
Attempting uninstall: tensorflow
Found existing installation: tensorflow 2.15.0
Uninstalling tensorflow-2.15.0:
Successfully uninstalled tensorflow-2.15.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tf-keras 2.15.1 requires tensorflow<2.16,>=2.15, but you have tensorflow 2.14.0 which is incompatible.
torchaudio 2.2.1+cu121 requires torch==2.2.1, but you have torch 2.1.0 which is incompatible.
torchtext 0.17.1 requires torch==2.2.1, but you have torch 2.1.0 which is incompatible.
torchvision 0.17.1+cu121 requires torch==2.2.1, but you have torch 2.1.0 which is incompatible.
Successfully installed accelerate-0.15.0 bitsandbytes-0.41.3.post2 dadaptation-3.1 diffusers-0.10.2 einops-0.6.0 ftfy-6.1.1 google-auth-oauthlib-1.0.0 keras-2.14.0 library-0.0.0 lightning-utilities-0.10.1 lion_pytorch-0.0.6 lycoris_lora-0.1.4 nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-8.9.2.26 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-nccl-cu12-2.18.1 nvidia-nvjitlink-cu12-12.4.99 nvidia-nvtx-cu12-12.1.105 prodigyopt-1.0 pytorch-lightning-1.9.0 safetensors-0.4.1 tensorboard-2.14.1 tensorflow-2.14.0 tensorflow-estimator-2.14.0 tokenizers-0.13.3 torch-2.1.0 torchmetrics-1.3.1 transformers-4.26.0 triton-2.1.0 voluptuous-0.13.1 xformers-0.0.22.post7
✅ Installation finished in 291 seconds.
🔄 Downloading model...
Download Results:
gid |stat|avg speed |path/URI
======+====+===========+=======================================================
0c3920|OK | 153MiB/s|//content/animefull-final-pruned-fp16.safetensors
Status Legend:
(OK):download completed.
📄 Config saved to /content/drive/MyDrive/Loras/bangbangv3/training_config.toml
📄 Dataset config saved to /content/drive/MyDrive/Loras/bangbangv3/dataset_config.toml
⭐ Starting trainer...
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /content/kohya-trainer/train_network.py:17 in <module> │
│ │
│ 14 from accelerate.utils import set_seed │
│ 15 from diffusers import DDPMScheduler │
│ 16 │
│ ❱ 17 import library.train_util as train_util │
│ 18 from library.train_util import ( │
│ 19 │ DreamBoothDataset, │
│ 20 ) │
│ │
│ /content/kohya-trainer/library/train_util.py:36 in <module> │
│ │
│ 33 import torch │
│ 34 from torch.nn.parallel import DistributedDataParallel as DDP │
│ 35 from torch.optim import Optimizer │
│ ❱ 36 from torchvision import transforms │
│ 37 from transformers import CLIPTokenizer │
│ 38 import transformers │
│ 39 import diffusers │
│ │
│ /usr/local/lib/python3.10/dist-packages/torchvision/__init__.py:6 in <module> │
│ │
│ 3 from modulefinder import Module │
│ 4 │
│ 5 import torch │
│ ❱ 6 from torchvision import _meta_registrations, datasets, io, models, ops, transforms, util │
│ 7 │
│ 8 from .extension import _HAS_OPS │
│ 9 │
│ │
│ /usr/local/lib/python3.10/dist-packages/torchvision/_meta_registrations.py:164 in <module> │
│ │
│ 161 │
│ 162 │
│ 163 @torch._custom_ops.impl_abstract("torchvision::nms") │
│ ❱ 164 def meta_nms(dets, scores, iou_threshold): │
│ 165 │ torch._check(dets.dim() == 2, lambda: f"boxes should be a 2d tensor, got {dets.dim() │
│ 166 │ torch._check(dets.size(1) == 4, lambda: f"boxes should have 4 elements in dimension │
│ 167 │ torch._check(scores.dim() == 1, lambda: f"scores should be a 1d tensor, got {scores. │
│ │
│ /usr/local/lib/python3.10/dist-packages/torch/_custom_ops.py:253 in inner │
│ │
│ 250 │ """ │
│ 251 │ │
│ 252 │ def inner(func): │
│ ❱ 253 │ │ custom_op = _find_custom_op(qualname, also_check_torch_library=True) │
│ 254 │ │ custom_op.impl_abstract(_stacklevel=3)(func) │
│ 255 │ │ return func │
│ 256 │
│ │
│ /usr/local/lib/python3.10/dist-packages/torch/_custom_op/impl.py:1076 in _find_custom_op │
│ │
│ 1073 │ │ raise RuntimeError( │
│ 1074 │ │ │ f"Could not find custom op \"{qualname}\". Did you register it via " │
│ 1075 │ │ │ f"the torch._custom_ops API?") │
│ ❱ 1076 │ overload = get_op(qualname) │
│ 1077 │ result = custom_op_from_existing(overload) │
│ 1078 │ return result │
│ 1079 │
│ │
│ /usr/local/lib/python3.10/dist-packages/torch/_custom_op/impl.py:1062 in get_op │
│ │
│ 1059 │ │ error_not_found() │
│ 1060 │ opnamespace = getattr(torch.ops, ns) │
│ 1061 │ if not hasattr(opnamespace, name): │
│ ❱ 1062 │ │ error_not_found() │
│ 1063 │ packet = getattr(opnamespace, name) │
│ 1064 │ if not hasattr(packet, 'default'): │
│ 1065 │ │ error_not_found() │
│ │
│ /usr/local/lib/python3.10/dist-packages/torch/_custom_op/impl.py:1052 in error_not_found │
│ │
│ 1049 │
│ 1050 def get_op(qualname): │
│ 1051 │ def error_not_found(): │
│ ❱ 1052 │ │ raise ValueError( │
│ 1053 │ │ │ f"Could not find the operator {qualname}. Please make sure you have " │
│ 1054 │ │ │ f"already registered the operator and (if registered from C++) " │
│ 1055 │ │ │ f"loaded it via torch.ops.load_library.") │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Could not find the operator torchvision::nms. Please make sure you have already
registered the operator and (if registered from C++) loaded it via torch.ops.load_library.
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /usr/local/bin/accelerate:8 in <module> │
│ │
│ 5 from accelerate.commands.accelerate_cli import main │
│ 6 if __name__ == '__main__': │
│ 7 │ sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) │
│ ❱ 8 │ sys.exit(main()) │
│ 9 │
│ │
│ /usr/local/lib/python3.10/dist-packages/accelerate/commands/accelerate_cli.py:45 in main │
│ │
│ 42 │ │ exit(1) │
│ 43 │ │
│ 44 │ # Run │
│ ❱ 45 │ args.func(args) │
│ 46 │
│ 47 │
│ 48 if __name__ == "__main__": │
│ │
│ /usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py:1104 in launch_command │
│ │
│ 1101 │ elif defaults is not None and defaults.compute_environment == ComputeEnvironment.AMA │
│ 1102 │ │ sagemaker_launcher(defaults, args) │
│ 1103 │ else: │
│ ❱ 1104 │ │ simple_launcher(args) │
│ 1105 │
│ 1106 │
│ 1107 def main(): │
│ │
│ /usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py:567 in simple_launcher │
│ │
│ 564 │ process = subprocess.Popen(cmd, env=current_env) │
│ 565 │ process.wait() │
│ 566 │ if process.returncode != 0: │
│ ❱ 567 │ │ raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) │
│ 568 │
│ 569 │
│ 570 def multi_gpu_launcher(args): │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
CalledProcessError: Command '['/usr/bin/python3', 'train_network.py',
'--dataset_config=/content/drive/MyDrive/Loras/bangbangv3/dataset_config.toml',
'--config_file=/content/drive/MyDrive/Loras/bangbangv3/training_config.toml']' returned non-zero
exit status 1.
Which colab and model(s) were you using when the error occurred?
What happened?
Yesterday, our system was still running normally, but today when we tried to use it, we encountered an error report, after a preliminary investigation, we suspect that this may be due to some packages installed in the environment caused by dependency conflicts.
Colab cell output
Which colab and model(s) were you using when the error occurred?
https://github.com/camenduru/stable-diffusion-webui-colab/tree/training
Which Public WebUI Colab URL were you using when the error occurred?
remote.moe
If you used HiRes mode when the error occurred, please provide the Hires info
No response