search

hollowstrawberry / kohya-colab

Accessible Google Colab notebooks for Stable Diffusion Lora training, based on the work of kohya-ss and Linaqruf
GNU General Public License v3.0
599 stars 87 forks source link

CUDA backend failed to initialize: Found CUDA version 12010, but JAX was built against version 12020, which is newer. #94

Closed ridhoyp closed 6 months ago

ridhoyp commented 6 months ago

Hi,

Please help, i cant use the collab..

CUDA backend failed to initialize: Found CUDA version 12010, but JAX was built against version 12020, which is newer. The copy of CUDA that is installed must be at least as new as the version against which JAX was built. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)

yesterday was good, but today is like this..

image

did anybody have same issue?

stadiffs commented 6 months ago

I have the same issue, I try to repair the code but I'm not an expert, failed several times, it looks like It has some dependencies conflict.

RivaryF commented 6 months ago

Same here. This is my first time, though. image

stadiffs commented 6 months ago

5 m ▶️ Setup Your project name will be the same as the folder containing your images. Spaces aren't allowed.

project_name: Astro The folder structure doesn't matter and is purely for comfort. Make sure to always pick the same one. I like organizing by project.

folder_structure:

Organize by project (MyDrive/Loras/project_name/dataset) Decide the model that will be downloaded and used for training. These options should produce clean and consistent results. You can also choose your own by pasting its download link.

training_model:

Stable Diffusion (sd-v1-5-pruned-noema-fp16.safetensors) optional_custom_training_model_url: Insertar text aquí custom_model_is_based_on_sd2:

▶️ Processing Resolution of 512 is standard for Stable Diffusion 1.5. Higher resolution training is much slower but can lead to better details.

Images will be automatically scaled while training to produce the best results, so you don't need to crop or resize anything yourself.

resolution:

512 This option will train your images both normally and flipped, for no extra cost, to learn more from them. Turn it on specially if you have less than 20 images.

Turn it off if you care about asymmetrical elements in your Lora.

flip_aug:

Shuffling anime tags in place improves learning and prompting. An activation tag goes at the start of every text file and will not be shuffled.

shuffle_tags:

activation_tags:

1 ▶️ Steps Your images will repeat this number of times during training. I recommend that your images multiplied by their repeats is between 200 and 400.

num_repeats: 4 Choose how long you want to train for. A good starting point is around 10 epochs or around 2000 steps.

One epoch is a number of steps equal to: your number of images multiplied by their repeats, divided by batch size.

preferred_unit:

Epochs how_many: 10 Saving more epochs will let you compare your Lora's progress better.

save_every_n_epochs: 1 keep_only_last_n_epochs: 10 Increasing the batch size makes training faster, but may make learning worse. Recommended 2 or 3.

train_batch_size:

2 ▶️ Learning The learning rate is the most important for your results. If you want to train slower with lots of images, or if your dim and alpha are high, move the unet to 2e-4 or lower.

The text encoder helps your Lora learn concepts slightly better. It is recommended to make it half or a fifth of the unet. If you're training a style you can even set it to 0.

unet_lr: 5e-4 text_encoder_lr: 1e-4 The scheduler is the algorithm that guides the learning rate. If you're not sure, pick constant and ignore the number. I personally recommend cosine_with_restarts with 3 restarts.

lr_scheduler:

cosine_with_restarts lr_scheduler_number: 3 Steps spent "warming up" the learning rate during training for efficiency. I recommend leaving it at 5%.

lr_warmup_ratio:

0.05 New feature that adjusts loss over time, makes learning much more efficient, and training can be done with about half as many epochs. Uses a value of 5.0 as recommended by the paper.

min_snr_gamma:

▶️ Structure LoRA is the classic type and good for a variety of purposes. LoCon is good with artstyles as it has more layers to learn more aspects of the dataset.

lora_type:

LoRA Below are some recommended values for the following settings:

type network_dim network_alpha conv_dim conv_alpha LoRA 16 8
LoCon 16 8 8 4 More dim means larger Lora, it can hold more information but more isn't always better. A dim between 8-32 is recommended, and alpha equal to half the dim.

network_dim:

16 network_alpha:

8 The following two values only apply to the additional layers of LoCon.

conv_dim:

8 conv_alpha:

4 ▶️ Ready You can now run this cell to cook your Lora. Good luck!

📂 Connecting to Google Drive... Mounted at /content/drive

💿 Checking dataset... 📁MyDrive/Loras/Astro/dataset 📈 Found 106 images with 4 repeats, equaling 424 steps. 📉 Divide 424 steps by 2 batch size to get 212.0 steps per epoch. 🔮 There will be 10 epochs, for around 2120 total training steps.

🏭 Installing dependencies...

Cloning into '/content/kohya-trainer'... remote: Enumerating objects: 5921, done. remote: Counting objects: 100% (2786/2786), done. remote: Compressing objects: 100% (359/359), done. remote: Total 5921 (delta 2619), reused 2433 (delta 2427), pack-reused 3135 Receiving objects: 100% (5921/5921), 9.37 MiB | 15.00 MiB/s, done. Resolving deltas: 100% (4207/4207), done. HEAD is now at 9a67e0d Merge pull request #610 from lubobill1990/patch-1 38 packages can be upgraded. Run 'apt list --upgradable' to see them. The following additional packages will be installed: libaria2-0 libc-ares2 The following NEW packages will be installed: aria2 libaria2-0 libc-ares2 0 upgraded, 3 newly installed, 0 to remove and 38 not upgraded. Need to get 1,513 kB of archives. After this operation, 5,441 kB of additional disk space will be used. Selecting previously unselected package libc-ares2:amd64. (Reading database ... 121752 files and directories currently installed.) Preparing to unpack .../libc-ares2_1.18.1-1ubuntu0.22.04.3_amd64.deb ... Unpacking libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.3) ... Selecting previously unselected package libaria2-0:amd64. Preparing to unpack .../libaria2-0_1.36.0-1_amd64.deb ... Unpacking libaria2-0:amd64 (1.36.0-1) ... Selecting previously unselected package aria2. Preparing to unpack .../aria2_1.36.0-1_amd64.deb ... Unpacking aria2 (1.36.0-1) ... Setting up libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.3) ... Setting up libaria2-0:amd64 (1.36.0-1) ... Setting up aria2 (1.36.0-1) ... Processing triggers for man-db (2.10.2-1) ... Processing triggers for libc-bin (2.35-0ubuntu3.4) ... /sbin/ldconfig.real: /usr/local/lib/libtbbmalloc_proxy.so.2 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbbind.so.3 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_5.so.3 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_0.so.3 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc.so.2 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbb.so.12 is not a symbolic link

Processing /content/kohya-trainer Preparing metadata (setup.py) ... done Collecting accelerate==0.15.0 (from -r requirements.txt (line 1)) Downloading accelerate-0.15.0-py3-none-any.whl (191 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 191.5/191.5 kB 5.6 MB/s eta 0:00:00 Collecting transformers==4.26.0 (from -r requirements.txt (line 2)) Downloading transformers-4.26.0-py3-none-any.whl (6.3 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.3/6.3 MB 25.8 MB/s eta 0:00:00 Collecting ftfy==6.1.1 (from -r requirements.txt (line 3)) Downloading ftfy-6.1.1-py3-none-any.whl (53 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.1/53.1 kB 7.1 MB/s eta 0:00:00 Requirement already satisfied: albumentations==1.3.1 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 4)) (1.3.1) Requirement already satisfied: opencv-python==4.8.0.76 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 5)) (4.8.0.76) Collecting einops==0.6.0 (from -r requirements.txt (line 6)) Downloading einops-0.6.0-py3-none-any.whl (41 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.6/41.6 kB 6.2 MB/s eta 0:00:00 Collecting diffusers==0.10.2 (from -r requirements.txt (line 7)) Downloading diffusers-0.10.2-py3-none-any.whl (503 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 503.1/503.1 kB 39.4 MB/s eta 0:00:00 Collecting pytorch-lightning==1.9.0 (from -r requirements.txt (line 8)) Downloading pytorch_lightning-1.9.0-py3-none-any.whl (825 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 825.8/825.8 kB 39.4 MB/s eta 0:00:00 Collecting bitsandbytes==0.41.3.post2 (from -r requirements.txt (line 9)) Downloading bitsandbytes-0.41.3.post2-py3-none-any.whl (92.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 92.6/92.6 MB 9.0 MB/s eta 0:00:00 Collecting tensorflow==2.14.0 (from -r requirements.txt (line 10)) Downloading tensorflow-2.14.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (489.8 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 489.8/489.8 MB 3.4 MB/s eta 0:00:00 Collecting safetensors==0.4.1 (from -r requirements.txt (line 11)) Downloading safetensors-0.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 88.6 MB/s eta 0:00:00 Requirement already satisfied: toml==0.10.2 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 12)) (0.10.2) Collecting voluptuous==0.13.1 (from -r requirements.txt (line 13)) Downloading voluptuous-0.13.1-py3-none-any.whl (29 kB) Collecting lion_pytorch==0.0.6 (from -r requirements.txt (line 14)) Downloading lion_pytorch-0.0.6-py3-none-any.whl (4.2 kB) Collecting dadaptation==3.1 (from -r requirements.txt (line 15)) Downloading dadaptation-3.1.tar.gz (13 kB) Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Collecting prodigyopt==1.0 (from -r requirements.txt (line 16)) Downloading prodigyopt-1.0-py3-none-any.whl (5.5 kB) Collecting lycoris_lora==0.1.4 (from -r requirements.txt (line 17)) Downloading lycoris_lora-0.1.4.tar.gz (37 kB) Preparing metadata (setup.py) ... done Collecting xformers==0.0.22.post7 (from -r requirements.txt (line 18)) Downloading xformers-0.0.22.post7-cp310-cp310-manylinux2014_x86_64.whl (211.8 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 211.8/211.8 MB 6.1 MB/s eta 0:00:00 Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.15.0->-r requirements.txt (line 1)) (1.25.2) Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.15.0->-r requirements.txt (line 1)) (24.0) Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate==0.15.0->-r requirements.txt (line 1)) (5.9.5) Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from accelerate==0.15.0->-r requirements.txt (line 1)) (6.0.1) Requirement already satisfied: torch>=1.4.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.15.0->-r requirements.txt (line 1)) (2.2.1+cu121) Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers==4.26.0->-r requirements.txt (line 2)) (3.13.1) Requirement already satisfied: huggingface-hub<1.0,>=0.11.0 in /usr/local/lib/python3.10/dist-packages (from transformers==4.26.0->-r requirements.txt (line 2)) (0.20.3) Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.26.0->-r requirements.txt (line 2)) (2023.12.25) Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers==4.26.0->-r requirements.txt (line 2)) (2.31.0) Collecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers==4.26.0->-r requirements.txt (line 2)) Downloading tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.8/7.8 MB 98.0 MB/s eta 0:00:00 Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers==4.26.0->-r requirements.txt (line 2)) (4.66.2) Requirement already satisfied: wcwidth>=0.2.5 in /usr/local/lib/python3.10/dist-packages (from ftfy==6.1.1->-r requirements.txt (line 3)) (0.2.13) Requirement already satisfied: scipy>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from albumentations==1.3.1->-r requirements.txt (line 4)) (1.11.4) Requirement already satisfied: scikit-image>=0.16.1 in /usr/local/lib/python3.10/dist-packages (from albumentations==1.3.1->-r requirements.txt (line 4)) (0.19.3) Requirement already satisfied: qudida>=0.0.4 in /usr/local/lib/python3.10/dist-packages (from albumentations==1.3.1->-r requirements.txt (line 4)) (0.0.4) Requirement already satisfied: opencv-python-headless>=4.1.1 in /usr/local/lib/python3.10/dist-packages (from albumentations==1.3.1->-r requirements.txt (line 4)) (4.9.0.80) Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.10/dist-packages (from diffusers==0.10.2->-r requirements.txt (line 7)) (7.0.2) Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from diffusers==0.10.2->-r requirements.txt (line 7)) (9.4.0) Requirement already satisfied: fsspec[http]>2021.06.0 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (2023.6.0) Collecting torchmetrics>=0.7.0 (from pytorch-lightning==1.9.0->-r requirements.txt (line 8)) Downloading torchmetrics-1.3.1-py3-none-any.whl (840 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 840.4/840.4 kB 58.6 MB/s eta 0:00:00 Requirement already satisfied: typing-extensions>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (4.10.0) Collecting lightning-utilities>=0.4.2 (from pytorch-lightning==1.9.0->-r requirements.txt (line 8)) Downloading lightning_utilities-0.10.1-py3-none-any.whl (24 kB) Requirement already satisfied: absl-py>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (1.4.0) Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (1.6.3) Requirement already satisfied: flatbuffers>=23.5.26 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (24.3.7) Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (0.5.4) Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (0.2.0) Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (3.9.0) Requirement already satisfied: libclang>=13.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (16.0.6) Requirement already satisfied: ml-dtypes==0.2.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (0.2.0) Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (3.3.0) Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (3.20.3) Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (67.7.2) Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (1.16.0) Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (2.4.0) Requirement already satisfied: wrapt<1.15,>=1.11.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (1.14.1) Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (0.36.0) Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (1.62.1) Collecting tensorboard<2.15,>=2.14 (from tensorflow==2.14.0->-r requirements.txt (line 10)) Downloading tensorboard-2.14.1-py3-none-any.whl (5.5 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.5/5.5 MB 104.0 MB/s eta 0:00:00 Collecting tensorflow-estimator<2.15,>=2.14.0 (from tensorflow==2.14.0->-r requirements.txt (line 10)) Downloading tensorflow_estimator-2.14.0-py2.py3-none-any.whl (440 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 440.7/440.7 kB 48.5 MB/s eta 0:00:00 Collecting keras<2.15,>=2.14.0 (from tensorflow==2.14.0->-r requirements.txt (line 10)) Downloading keras-2.14.0-py3-none-any.whl (1.7 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 97.1 MB/s eta 0:00:00 Collecting torch>=1.4.0 (from accelerate==0.15.0->-r requirements.txt (line 1)) Downloading torch-2.1.0-cp310-cp310-manylinux1_x86_64.whl (670.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 670.2/670.2 MB 2.5 MB/s eta 0:00:00 Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) (1.12) Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) (3.2.1) Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) (3.1.3) Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23.7/23.7 MB 43.9 MB/s eta 0:00:00 Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 823.6/823.6 kB 58.1 MB/s eta 0:00:00 Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.1/14.1 MB 63.2 MB/s eta 0:00:00 Collecting nvidia-cudnn-cu12==8.9.2.26 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 731.7/731.7 MB 965.1 kB/s eta 0:00:00 Collecting nvidia-cublas-cu12==12.1.3.1 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 410.6/410.6 MB 2.7 MB/s eta 0:00:00 Collecting nvidia-cufft-cu12==11.0.2.54 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.6/121.6 MB 8.2 MB/s eta 0:00:00 Collecting nvidia-curand-cu12==10.3.2.106 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.5/56.5 MB 11.7 MB/s eta 0:00:00 Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 124.2/124.2 MB 8.6 MB/s eta 0:00:00 Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 196.0/196.0 MB 5.9 MB/s eta 0:00:00 Collecting nvidia-nccl-cu12==2.18.1 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading nvidia_nccl_cu12-2.18.1-py3-none-manylinux1_x86_64.whl (209.8 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 209.8/209.8 MB 6.4 MB/s eta 0:00:00 Collecting nvidia-nvtx-cu12==12.1.105 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 99.1/99.1 kB 14.9 MB/s eta 0:00:00 Collecting triton==2.1.0 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading triton-2.1.0-0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (89.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 89.2/89.2 MB 9.2 MB/s eta 0:00:00 Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading nvidia_nvjitlink_cu12-12.4.99-py3-none-manylinux2014_x86_64.whl (21.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.1/21.1 MB 81.8 MB/s eta 0:00:00 Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from astunparse>=1.6.0->tensorflow==2.14.0->-r requirements.txt (line 10)) (0.43.0) Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /usr/local/lib/python3.10/dist-packages (from fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (3.9.3) Requirement already satisfied: scikit-learn>=0.19.1 in /usr/local/lib/python3.10/dist-packages (from qudida>=0.0.4->albumentations==1.3.1->-r requirements.txt (line 4)) (1.2.2) Requirement already satisfied: imageio>=2.4.1 in /usr/local/lib/python3.10/dist-packages (from scikit-image>=0.16.1->albumentations==1.3.1->-r requirements.txt (line 4)) (2.31.6) Requirement already satisfied: tifffile>=2019.7.26 in /usr/local/lib/python3.10/dist-packages (from scikit-image>=0.16.1->albumentations==1.3.1->-r requirements.txt (line 4)) (2024.2.12) Requirement already satisfied: PyWavelets>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-image>=0.16.1->albumentations==1.3.1->-r requirements.txt (line 4)) (1.5.0) Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (2.27.0) Collecting google-auth-oauthlib<1.1,>=0.5 (from tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) Downloading google_auth_oauthlib-1.0.0-py2.py3-none-any.whl (18 kB) Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (3.5.2) Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (0.7.2) Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (3.0.1) Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.26.0->-r requirements.txt (line 2)) (3.3.2) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.26.0->-r requirements.txt (line 2)) (3.6) Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.26.0->-r requirements.txt (line 2)) (2.0.7) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.26.0->-r requirements.txt (line 2)) (2024.2.2) Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata->diffusers==0.10.2->-r requirements.txt (line 7)) (3.17.0) Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (1.3.1) Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (23.2.0) Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (1.4.1) Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (6.0.5) Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (1.9.4) Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (4.0.3) Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (5.3.3) Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (0.3.0) Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (4.9) Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<1.1,>=0.5->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (1.4.0) Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.19.1->qudida>=0.0.4->albumentations==1.3.1->-r requirements.txt (line 4)) (1.3.2) Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.19.1->qudida>=0.0.4->albumentations==1.3.1->-r requirements.txt (line 4)) (3.3.0) Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/dist-packages (from werkzeug>=1.0.1->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (2.1.5) Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) (1.3.0) Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (0.5.1) Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<1.1,>=0.5->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (3.2.2) Building wheels for collected packages: dadaptation, lycoris_lora, library Building wheel for dadaptation (pyproject.toml) ... done Created wheel for dadaptation: filename=dadaptation-3.1-py3-none-any.whl size=22788 sha256=a9524ad975696b22ec6bdbf314da832dba911591bd5a19e8f25c05e9aa44345c Stored in directory: /root/.cache/pip/wheels/34/25/92/b7708738900a69a166f41de56c776039b5a05599b36075c34f Building wheel for lycoris_lora (setup.py) ... done Created wheel for lycoris_lora: filename=lycoris_lora-0.1.4-py3-none-any.whl size=26134 sha256=70b8a7107bbaa864c9f3b9481da03643b9721993c4d1026d6e157075cde791a0 Stored in directory: /root/.cache/pip/wheels/65/d4/14/c7a61ec6ce0398272767c355bd02af1aa309cecd7102b22080 Building wheel for library (setup.py) ... done Created wheel for library: filename=library-0.0.0-py3-none-any.whl size=84081 sha256=fbad74ebde46c99ff106a033b13a15d74d94e931ab4ed94b3d7a869ac9348a40 Stored in directory: /root/.cache/pip/wheels/d8/ef/ac/94db7d75dc3a8e81533bc2d001d79467c693ec69ed02bbee59 Successfully built dadaptation lycoris_lora library Installing collected packages: voluptuous, tokenizers, library, bitsandbytes, triton, tensorflow-estimator, safetensors, prodigyopt, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, lightning-utilities, keras, ftfy, einops, dadaptation, nvidia-cusparse-cu12, nvidia-cudnn-cu12, transformers, nvidia-cusolver-cu12, google-auth-oauthlib, diffusers, torch, tensorboard, xformers, torchmetrics, tensorflow, lycoris_lora, lion_pytorch, accelerate, pytorch-lightning Attempting uninstall: tokenizers Found existing installation: tokenizers 0.15.2 Uninstalling tokenizers-0.15.2: Successfully uninstalled tokenizers-0.15.2 Attempting uninstall: triton Found existing installation: triton 2.2.0 Uninstalling triton-2.2.0: Successfully uninstalled triton-2.2.0 Attempting uninstall: tensorflow-estimator Found existing installation: tensorflow-estimator 2.15.0 Uninstalling tensorflow-estimator-2.15.0: Successfully uninstalled tensorflow-estimator-2.15.0 Attempting uninstall: safetensors Found existing installation: safetensors 0.4.2 Uninstalling safetensors-0.4.2: Successfully uninstalled safetensors-0.4.2 Attempting uninstall: keras Found existing installation: keras 2.15.0 Uninstalling keras-2.15.0: Successfully uninstalled keras-2.15.0 Attempting uninstall: transformers Found existing installation: transformers 4.38.2 Uninstalling transformers-4.38.2: Successfully uninstalled transformers-4.38.2 Attempting uninstall: google-auth-oauthlib Found existing installation: google-auth-oauthlib 1.2.0 Uninstalling google-auth-oauthlib-1.2.0: Successfully uninstalled google-auth-oauthlib-1.2.0 Attempting uninstall: torch Found existing installation: torch 2.2.1+cu121 Uninstalling torch-2.2.1+cu121: Successfully uninstalled torch-2.2.1+cu121 Attempting uninstall: tensorboard Found existing installation: tensorboard 2.15.2 Uninstalling tensorboard-2.15.2: Successfully uninstalled tensorboard-2.15.2 Attempting uninstall: tensorflow Found existing installation: tensorflow 2.15.0 Uninstalling tensorflow-2.15.0: Successfully uninstalled tensorflow-2.15.0 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. tf-keras 2.15.1 requires tensorflow<2.16,>=2.15, but you have tensorflow 2.14.0 which is incompatible. torchaudio 2.2.1+cu121 requires torch==2.2.1, but you have torch 2.1.0 which is incompatible. torchtext 0.17.1 requires torch==2.2.1, but you have torch 2.1.0 which is incompatible. torchvision 0.17.1+cu121 requires torch==2.2.1, but you have torch 2.1.0 which is incompatible. Successfully installed accelerate-0.15.0 bitsandbytes-0.41.3.post2 dadaptation-3.1 diffusers-0.10.2 einops-0.6.0 ftfy-6.1.1 google-auth-oauthlib-1.0.0 keras-2.14.0 library-0.0.0 lightning-utilities-0.10.1 lion_pytorch-0.0.6 lycoris_lora-0.1.4 nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-8.9.2.26 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-nccl-cu12-2.18.1 nvidia-nvjitlink-cu12-12.4.99 nvidia-nvtx-cu12-12.1.105 prodigyopt-1.0 pytorch-lightning-1.9.0 safetensors-0.4.1 tensorboard-2.14.1 tensorflow-2.14.0 tensorflow-estimator-2.14.0 tokenizers-0.13.3 torch-2.1.0 torchmetrics-1.3.1 transformers-4.26.0 triton-2.1.0 voluptuous-0.13.1 xformers-0.0.22.post7

✅ Installation finished in 261 seconds.

🔄 Downloading model...

Download Results: gid |stat|avg speed |path/URI ======+====+===========+======================================================= dcaa59|OK | 167MiB/s|//content/sd-v1-5-pruned-noema-fp16.safetensors

Status Legend: (OK):download completed.

📄 Config saved to /content/drive/MyDrive/Loras/Astro/training_config.toml 📄 Dataset config saved to /content/drive/MyDrive/Loras/Astro/dataset_config.toml

⭐ Starting trainer...

CUDA backend failed to initialize: Found CUDA version 12010, but JAX was built against version 12020, which is newer. The copy of CUDA that is installed must be at least as new as the version against which JAX was built. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.) ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ /content/kohya-trainer/train_network.py:17 in │ │ │ │ 14 from accelerate.utils import set_seed │ │ 15 from diffusers import DDPMScheduler │ │ 16 │ │ ❱ 17 import library.train_util as train_util │ │ 18 from library.train_util import ( │ │ 19 │ DreamBoothDataset, │ │ 20 ) │ │ │ │ /content/kohya-trainer/library/train_util.py:36 in │ │ │ │ 33 import torch │ │ 34 from torch.nn.parallel import DistributedDataParallel as DDP │ │ 35 from torch.optim import Optimizer │ │ ❱ 36 from torchvision import transforms │ │ 37 from transformers import CLIPTokenizer │ │ 38 import transformers │ │ 39 import diffusers │ │ │ │ /usr/local/lib/python3.10/dist-packages/torchvision/init.py:6 in │ │ │ │ 3 from modulefinder import Module │ │ 4 │ │ 5 import torch │ │ ❱ 6 from torchvision import _meta_registrations, datasets, io, models, ops, transforms, util │ │ 7 │ │ 8 from .extension import _HAS_OPS │ │ 9 │ │ │ │ /usr/local/lib/python3.10/dist-packages/torchvision/_meta_registrations.py:164 in │ │ │ │ 161 │ │ 162 │ │ 163 @torch._custom_ops.impl_abstract("torchvision::nms") │ │ ❱ 164 def meta_nms(dets, scores, iou_threshold): │ │ 165 │ torch._check(dets.dim() == 2, lambda: f"boxes should be a 2d tensor, got {dets.dim() │ │ 166 │ torch._check(dets.size(1) == 4, lambda: f"boxes should have 4 elements in dimension │ │ 167 │ torch._check(scores.dim() == 1, lambda: f"scores should be a 1d tensor, got {scores. │ │ │ │ /usr/local/lib/python3.10/dist-packages/torch/_custom_ops.py:253 in inner │ │ │ │ 250 │ """ │ │ 251 │ │ │ 252 │ def inner(func): │ │ ❱ 253 │ │ custom_op = _find_custom_op(qualname, also_check_torch_library=True) │ │ 254 │ │ custom_op.impl_abstract(_stacklevel=3)(func) │ │ 255 │ │ return func │ │ 256 │ │ │ │ /usr/local/lib/python3.10/dist-packages/torch/_custom_op/impl.py:1076 in _find_custom_op │ │ │ │ 1073 │ │ raise RuntimeError( │ │ 1074 │ │ │ f"Could not find custom op \"{qualname}\". Did you register it via " │ │ 1075 │ │ │ f"the torch._custom_ops API?") │ │ ❱ 1076 │ overload = get_op(qualname) │ │ 1077 │ result = custom_op_from_existing(overload) │ │ 1078 │ return result │ │ 1079 │ │ │ │ /usr/local/lib/python3.10/dist-packages/torch/_custom_op/impl.py:1062 in get_op │ │ │ │ 1059 │ │ error_not_found() │ │ 1060 │ opnamespace = getattr(torch.ops, ns) │ │ 1061 │ if not hasattr(opnamespace, name): │ │ ❱ 1062 │ │ error_not_found() │ │ 1063 │ packet = getattr(opnamespace, name) │ │ 1064 │ if not hasattr(packet, 'default'): │ │ 1065 │ │ error_not_found() │ │ │ │ /usr/local/lib/python3.10/dist-packages/torch/_custom_op/impl.py:1052 in error_not_found │ │ │ │ 1049 │ │ 1050 def get_op(qualname): │ │ 1051 │ def error_not_found(): │ │ ❱ 1052 │ │ raise ValueError( │ │ 1053 │ │ │ f"Could not find the operator {qualname}. Please make sure you have " │ │ 1054 │ │ │ f"already registered the operator and (if registered from C++) " │ │ 1055 │ │ │ f"loaded it via torch.ops.load_library.") │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ ValueError: Could not find the operator torchvision::nms. Please make sure you have already registered the operator and (if registered from C++) loaded it via torch.ops.load_library. ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ /usr/local/bin/accelerate:8 in │ │ │ │ 5 from accelerate.commands.accelerate_cli import main │ │ 6 if name == 'main': │ │ 7 │ sys.argv[0] = re.sub(r'(-script.pyw|.exe)?$', '', sys.argv[0]) │ │ ❱ 8 │ sys.exit(main()) │ │ 9 │ │ │ │ /usr/local/lib/python3.10/dist-packages/accelerate/commands/accelerate_cli.py:45 in main │ │ │ │ 42 │ │ exit(1) │ │ 43 │ │ │ 44 │ # Run │ │ ❱ 45 │ args.func(args) │ │ 46 │ │ 47 │ │ 48 if name == "main": │ │ │ │ /usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py:1104 in launch_command │ │ │ │ 1101 │ elif defaults is not None and defaults.compute_environment == ComputeEnvironment.AMA │ │ 1102 │ │ sagemaker_launcher(defaults, args) │ │ 1103 │ else: │ │ ❱ 1104 │ │ simple_launcher(args) │ │ 1105 │ │ 1106 │ │ 1107 def main(): │ │ │ │ /usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py:567 in simple_launcher │ │ │ │ 564 │ process = subprocess.Popen(cmd, env=current_env) │ │ 565 │ process.wait() │ │ 566 │ if process.returncode != 0: │ │ ❱ 567 │ │ raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) │ │ 568 │ │ 569 │ │ 570 def multi_gpu_launcher(args): │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ CalledProcessError: Command '['/usr/bin/python3', 'train_network.py', '--dataset_config=/content/drive/MyDrive/Loras/Astro/dataset_config.toml', '--config_file=/content/drive/MyDrive/Loras/Astro/training_config.toml']' returned non-zero exit status 1.

baicai99 commented 6 months ago

same problem

Snipaste_2024-03-14_15-31-21
Shadow153X153 commented 6 months ago

same problem Screenshot_2

ErichEisner commented 6 months ago

Same here. I made one LoRA yesterday and then shortly afterwards I tried to make a second and got the same errors. Tried again today, but same result.

illlliill0001 commented 6 months ago

Same here.

stadiffs commented 6 months ago

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. tf-keras 2.15.1 requires tensorflow<2.16,>=2.15, but you have tensorflow 2.14.0 which is incompatible. torchaudio 2.2.1+cu121 requires torch==2.2.1, but you have torch 2.1.0 which is incompatible. torchtext 0.17.1 requires torch==2.2.1, but you have torch 2.1.0 which is incompatible. torchvision 0.17.1+cu121 requires torch==2.2.1, but you have torch 2.1.0 which is incompatible.

mintkoko commented 6 months ago

It's the same problem There's a problem today

Installing dependencies...

Cloning into '/content/kohya-trainer'... remote: Enumerating objects: 5921, done. remote: Counting objects: 100% (2786/2786), done. remote: Compressing objects: 100% (359/359), done. remote: Total 5921 (delta 2619), reused 2433 (delta 2427), pack-reused 3135 Receiving objects: 100% (5921/5921), 9.37 MiB | 17.98 MiB/s, done. Resolving deltas: 100% (4207/4207), done. HEAD is now at 9a67e0d Merge pull request #610 from lubobill1990/patch-1 40 packages can be upgraded. Run 'apt list --upgradable' to see them. The following additional packages will be installed: libaria2-0 libc-ares2 The following NEW packages will be installed: aria2 libaria2-0 libc-ares2 0 upgraded, 3 newly installed, 0 to remove and 40 not upgraded. Need to get 1,513 kB of archives. After this operation, 5,441 kB of additional disk space will be used. Selecting previously unselected package libc-ares2:amd64. (Reading database ... 121752 files and directories currently installed.) Preparing to unpack .../libc-ares2_1.18.1-1ubuntu0.22.04.3_amd64.deb ... Unpacking libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.3) ... Selecting previously unselected package libaria2-0:amd64. Preparing to unpack .../libaria2-0_1.36.0-1_amd64.deb ... Unpacking libaria2-0:amd64 (1.36.0-1) ... Selecting previously unselected package aria2. Preparing to unpack .../aria2_1.36.0-1_amd64.deb ... Unpacking aria2 (1.36.0-1) ... Setting up libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.3) ... Setting up libaria2-0:amd64 (1.36.0-1) ... Setting up aria2 (1.36.0-1) ... Processing triggers for man-db (2.10.2-1) ... Processing triggers for libc-bin (2.35-0ubuntu3.4) ... /sbin/ldconfig.real: /usr/local/lib/libtbbmalloc_proxy.so.2 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbbind.so.3 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_5.so.3 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_0.so.3 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc.so.2 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbb.so.12 is not a symbolic link

Processing /content/kohya-trainer Preparing metadata (setup.py) ... done Collecting accelerate==0.15.0 (from -r requirements.txt (line 1)) Downloading accelerate-0.15.0-py3-none-any.whl (191 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 191.5/191.5 kB 3.2 MB/s eta 0:00:00 Collecting transformers==4.26.0 (from -r requirements.txt (line 2)) Downloading transformers-4.26.0-py3-none-any.whl (6.3 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.3/6.3 MB 62.5 MB/s eta 0:00:00 Collecting ftfy==6.1.1 (from -r requirements.txt (line 3)) Downloading ftfy-6.1.1-py3-none-any.whl (53 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.1/53.1 kB 8.0 MB/s eta 0:00:00 Requirement already satisfied: albumentations==1.3.1 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 4)) (1.3.1) Requirement already satisfied: opencv-python==4.8.0.76 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 5)) (4.8.0.76) Collecting einops==0.6.0 (from -r requirements.txt (line 6)) Downloading einops-0.6.0-py3-none-any.whl (41 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.6/41.6 kB 6.1 MB/s eta 0:00:00 Collecting diffusers==0.10.2 (from -r requirements.txt (line 7)) Downloading diffusers-0.10.2-py3-none-any.whl (503 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 503.1/503.1 kB 53.6 MB/s eta 0:00:00 Collecting pytorch-lightning==1.9.0 (from -r requirements.txt (line 8)) Downloading pytorch_lightning-1.9.0-py3-none-any.whl (825 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 825.8/825.8 kB 66.5 MB/s eta 0:00:00 Collecting bitsandbytes==0.41.3.post2 (from -r requirements.txt (line 9)) Downloading bitsandbytes-0.41.3.post2-py3-none-any.whl (92.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 92.6/92.6 MB 8.6 MB/s eta 0:00:00 Collecting tensorflow==2.14.0 (from -r requirements.txt (line 10)) Downloading tensorflow-2.14.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (489.8 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 489.8/489.8 MB 3.3 MB/s eta 0:00:00 Collecting safetensors==0.4.1 (from -r requirements.txt (line 11)) Downloading safetensors-0.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 48.5 MB/s eta 0:00:00 Requirement already satisfied: toml==0.10.2 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 12)) (0.10.2) Collecting voluptuous==0.13.1 (from -r requirements.txt (line 13)) Downloading voluptuous-0.13.1-py3-none-any.whl (29 kB) Collecting lion_pytorch==0.0.6 (from -r requirements.txt (line 14)) Downloading lion_pytorch-0.0.6-py3-none-any.whl (4.2 kB) Collecting dadaptation==3.1 (from -r requirements.txt (line 15)) Downloading dadaptation-3.1.tar.gz (13 kB) Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Collecting prodigyopt==1.0 (from -r requirements.txt (line 16)) Downloading prodigyopt-1.0-py3-none-any.whl (5.5 kB) Collecting lycoris_lora==0.1.4 (from -r requirements.txt (line 17)) Downloading lycoris_lora-0.1.4.tar.gz (37 kB) Preparing metadata (setup.py) ... done Collecting xformers==0.0.22.post7 (from -r requirements.txt (line 18)) Downloading xformers-0.0.22.post7-cp310-cp310-manylinux2014_x86_64.whl (211.8 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 211.8/211.8 MB 5.7 MB/s eta 0:00:00 Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.15.0->-r requirements.txt (line 1)) (1.25.2) Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.15.0->-r requirements.txt (line 1)) (24.0) Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate==0.15.0->-r requirements.txt (line 1)) (5.9.5) Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from accelerate==0.15.0->-r requirements.txt (line 1)) (6.0.1) Requirement already satisfied: torch>=1.4.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.15.0->-r requirements.txt (line 1)) (2.2.1+cu121) Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers==4.26.0->-r requirements.txt (line 2)) (3.13.1) Requirement already satisfied: huggingface-hub<1.0,>=0.11.0 in /usr/local/lib/python3.10/dist-packages (from transformers==4.26.0->-r requirements.txt (line 2)) (0.20.3) Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.26.0->-r requirements.txt (line 2)) (2023.12.25) Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers==4.26.0->-r requirements.txt (line 2)) (2.31.0) Collecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers==4.26.0->-r requirements.txt (line 2)) Downloading tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.8/7.8 MB 98.2 MB/s eta 0:00:00 Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers==4.26.0->-r requirements.txt (line 2)) (4.66.2) Requirement already satisfied: wcwidth>=0.2.5 in /usr/local/lib/python3.10/dist-packages (from ftfy==6.1.1->-r requirements.txt (line 3)) (0.2.13) Requirement already satisfied: scipy>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from albumentations==1.3.1->-r requirements.txt (line 4)) (1.11.4) Requirement already satisfied: scikit-image>=0.16.1 in /usr/local/lib/python3.10/dist-packages (from albumentations==1.3.1->-r requirements.txt (line 4)) (0.19.3) Requirement already satisfied: qudida>=0.0.4 in /usr/local/lib/python3.10/dist-packages (from albumentations==1.3.1->-r requirements.txt (line 4)) (0.0.4) Requirement already satisfied: opencv-python-headless>=4.1.1 in /usr/local/lib/python3.10/dist-packages (from albumentations==1.3.1->-r requirements.txt (line 4)) (4.9.0.80) Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.10/dist-packages (from diffusers==0.10.2->-r requirements.txt (line 7)) (7.0.2) Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from diffusers==0.10.2->-r requirements.txt (line 7)) (9.4.0) Requirement already satisfied: fsspec[http]>2021.06.0 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (2023.6.0) Collecting torchmetrics>=0.7.0 (from pytorch-lightning==1.9.0->-r requirements.txt (line 8)) Downloading torchmetrics-1.3.1-py3-none-any.whl (840 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 840.4/840.4 kB 67.5 MB/s eta 0:00:00 Requirement already satisfied: typing-extensions>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (4.10.0) Collecting lightning-utilities>=0.4.2 (from pytorch-lightning==1.9.0->-r requirements.txt (line 8)) Downloading lightning_utilities-0.10.1-py3-none-any.whl (24 kB) Requirement already satisfied: absl-py>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (1.4.0) Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (1.6.3) Requirement already satisfied: flatbuffers>=23.5.26 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (24.3.7) Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (0.5.4) Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (0.2.0) Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (3.9.0) Requirement already satisfied: libclang>=13.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (16.0.6) Requirement already satisfied: ml-dtypes==0.2.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (0.2.0) Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (3.3.0) Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (3.20.3) Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (67.7.2) Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (1.16.0) Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (2.4.0) Requirement already satisfied: wrapt<1.15,>=1.11.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (1.14.1) Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (0.36.0) Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (1.62.1) Collecting tensorboard<2.15,>=2.14 (from tensorflow==2.14.0->-r requirements.txt (line 10)) Downloading tensorboard-2.14.1-py3-none-any.whl (5.5 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.5/5.5 MB 93.5 MB/s eta 0:00:00 Collecting tensorflow-estimator<2.15,>=2.14.0 (from tensorflow==2.14.0->-r requirements.txt (line 10)) Downloading tensorflow_estimator-2.14.0-py2.py3-none-any.whl (440 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 440.7/440.7 kB 50.7 MB/s eta 0:00:00 Collecting keras<2.15,>=2.14.0 (from tensorflow==2.14.0->-r requirements.txt (line 10)) Downloading keras-2.14.0-py3-none-any.whl (1.7 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 87.7 MB/s eta 0:00:00 Collecting torch>=1.4.0 (from accelerate==0.15.0->-r requirements.txt (line 1)) Downloading torch-2.1.0-cp310-cp310-manylinux1_x86_64.whl (670.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 670.2/670.2 MB 2.5 MB/s eta 0:00:00 Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) (1.12) Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) (3.2.1) Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) (3.1.3) Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23.7/23.7 MB 49.9 MB/s eta 0:00:00 Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 823.6/823.6 kB 74.8 MB/s eta 0:00:00 Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.1/14.1 MB 84.6 MB/s eta 0:00:00 Collecting nvidia-cudnn-cu12==8.9.2.26 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 731.7/731.7 MB 1.7 MB/s eta 0:00:00 Collecting nvidia-cublas-cu12==12.1.3.1 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 410.6/410.6 MB 2.5 MB/s eta 0:00:00 Collecting nvidia-cufft-cu12==11.0.2.54 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.6/121.6 MB 8.3 MB/s eta 0:00:00 Collecting nvidia-curand-cu12==10.3.2.106 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.5/56.5 MB 13.2 MB/s eta 0:00:00 Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 124.2/124.2 MB 9.2 MB/s eta 0:00:00 Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 196.0/196.0 MB 6.8 MB/s eta 0:00:00 Collecting nvidia-nccl-cu12==2.18.1 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading nvidia_nccl_cu12-2.18.1-py3-none-manylinux1_x86_64.whl (209.8 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 209.8/209.8 MB 4.9 MB/s eta 0:00:00 Collecting nvidia-nvtx-cu12==12.1.105 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 99.1/99.1 kB 16.0 MB/s eta 0:00:00 Collecting triton==2.1.0 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading triton-2.1.0-0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (89.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 89.2/89.2 MB 9.3 MB/s eta 0:00:00 Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) Downloading nvidia_nvjitlink_cu12-12.4.99-py3-none-manylinux2014_x86_64.whl (21.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.1/21.1 MB 62.0 MB/s eta 0:00:00 Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from astunparse>=1.6.0->tensorflow==2.14.0->-r requirements.txt (line 10)) (0.43.0) Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /usr/local/lib/python3.10/dist-packages (from fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (3.9.3) Requirement already satisfied: scikit-learn>=0.19.1 in /usr/local/lib/python3.10/dist-packages (from qudida>=0.0.4->albumentations==1.3.1->-r requirements.txt (line 4)) (1.2.2) Requirement already satisfied: imageio>=2.4.1 in /usr/local/lib/python3.10/dist-packages (from scikit-image>=0.16.1->albumentations==1.3.1->-r requirements.txt (line 4)) (2.31.6) Requirement already satisfied: tifffile>=2019.7.26 in /usr/local/lib/python3.10/dist-packages (from scikit-image>=0.16.1->albumentations==1.3.1->-r requirements.txt (line 4)) (2024.2.12) Requirement already satisfied: PyWavelets>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-image>=0.16.1->albumentations==1.3.1->-r requirements.txt (line 4)) (1.5.0) Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (2.27.0) Collecting google-auth-oauthlib<1.1,>=0.5 (from tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) Downloading google_auth_oauthlib-1.0.0-py2.py3-none-any.whl (18 kB) Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (3.5.2) Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (0.7.2) Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (3.0.1) Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.26.0->-r requirements.txt (line 2)) (3.3.2) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.26.0->-r requirements.txt (line 2)) (3.6) Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.26.0->-r requirements.txt (line 2)) (2.0.7) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.26.0->-r requirements.txt (line 2)) (2024.2.2) Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata->diffusers==0.10.2->-r requirements.txt (line 7)) (3.17.0) Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (1.3.1) Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (23.2.0) Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (1.4.1) Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (6.0.5) Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (1.9.4) Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (4.0.3) Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (5.3.3) Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (0.3.0) Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (4.9) Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<1.1,>=0.5->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (1.4.0) Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.19.1->qudida>=0.0.4->albumentations==1.3.1->-r requirements.txt (line 4)) (1.3.2) Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.19.1->qudida>=0.0.4->albumentations==1.3.1->-r requirements.txt (line 4)) (3.3.0) Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/dist-packages (from werkzeug>=1.0.1->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (2.1.5) Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) (1.3.0) Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (0.5.1) Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<1.1,>=0.5->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (3.2.2) Building wheels for collected packages: dadaptation, lycoris_lora, library Building wheel for dadaptation (pyproject.toml) ... done Created wheel for dadaptation: filename=dadaptation-3.1-py3-none-any.whl size=22788 sha256=eb186e00b1caed09479017534003480f7a675e2b47334460182ab109e230b60d Stored in directory: /root/.cache/pip/wheels/34/25/92/b7708738900a69a166f41de56c776039b5a05599b36075c34f Building wheel for lycoris_lora (setup.py) ... done Created wheel for lycoris_lora: filename=lycoris_lora-0.1.4-py3-none-any.whl size=26134 sha256=a9a099a70274c53fcbe4d84e969df03728614e40f7c28e115a49592e25fa9177 Stored in directory: /root/.cache/pip/wheels/65/d4/14/c7a61ec6ce0398272767c355bd02af1aa309cecd7102b22080 Building wheel for library (setup.py) ... done Created wheel for library: filename=library-0.0.0-py3-none-any.whl size=84081 sha256=65750aed1a6b70818f91b7a87ed8a89cbfa56e2d6c00a47e8bdd3180ec12934e Stored in directory: /root/.cache/pip/wheels/d8/ef/ac/94db7d75dc3a8e81533bc2d001d79467c693ec69ed02bbee59 Successfully built dadaptation lycoris_lora library Installing collected packages: voluptuous, tokenizers, library, bitsandbytes, triton, tensorflow-estimator, safetensors, prodigyopt, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, lightning-utilities, keras, ftfy, einops, dadaptation, nvidia-cusparse-cu12, nvidia-cudnn-cu12, transformers, nvidia-cusolver-cu12, google-auth-oauthlib, diffusers, torch, tensorboard, xformers, torchmetrics, tensorflow, lycoris_lora, lion_pytorch, accelerate, pytorch-lightning Attempting uninstall: tokenizers Found existing installation: tokenizers 0.15.2 Uninstalling tokenizers-0.15.2: Successfully uninstalled tokenizers-0.15.2 Attempting uninstall: triton Found existing installation: triton 2.2.0 Uninstalling triton-2.2.0: Successfully uninstalled triton-2.2.0 Attempting uninstall: tensorflow-estimator Found existing installation: tensorflow-estimator 2.15.0 Uninstalling tensorflow-estimator-2.15.0: Successfully uninstalled tensorflow-estimator-2.15.0 Attempting uninstall: safetensors Found existing installation: safetensors 0.4.2 Uninstalling safetensors-0.4.2: Successfully uninstalled safetensors-0.4.2 Attempting uninstall: keras Found existing installation: keras 2.15.0 Uninstalling keras-2.15.0: Successfully uninstalled keras-2.15.0 Attempting uninstall: transformers Found existing installation: transformers 4.38.2 Uninstalling transformers-4.38.2: Successfully uninstalled transformers-4.38.2 Attempting uninstall: google-auth-oauthlib Found existing installation: google-auth-oauthlib 1.2.0 Uninstalling google-auth-oauthlib-1.2.0: Successfully uninstalled google-auth-oauthlib-1.2.0 Attempting uninstall: torch Found existing installation: torch 2.2.1+cu121 Uninstalling torch-2.2.1+cu121: Successfully uninstalled torch-2.2.1+cu121

Hectoris919 commented 6 months ago

I'm having the same issue on google colab - does this happen on a computer as well? I don't have a powerful enough computer to test it out myself

hollowstrawberry commented 6 months ago

I have fixed the dependencies for the XL trainer, but sadly we need a new xformers version for the regular trainer to work properly. Therefore, the XL trainer is working again, but not the regular trainer.

I'll keep an eye on https://github.com/facebookresearch/xformers/issues/991

Hectoris919 commented 6 months ago

I have fixed the dependencies for the XL trainer, but sadly we need a new xformers version for the regular trainer to work properly. Therefore, the XL trainer is working again, but not the regular trainer.

I'll keep an eye on facebookresearch/xformers#991

It also needs tensorflow==2.15.0 instead of tensorflow==2.14.0

stadiffs commented 6 months ago

xformers torchvision torchaudio torchtext

are not comatible with pytorch 2.2.1

is a mess X).

lucaswalkeryoung commented 6 months ago

Yay for madness! I guess this gives me time to focus on building a better dataset until the trainer gets back up and running XD

Yesterday, somehow, I actually found a solution to this. For the life of me I can't find how to repeat it though.

ridhoyp commented 6 months ago

I have fixed the dependencies for the XL trainer, but sadly we need a new xformers version for the regular trainer to work properly. Therefore, the XL trainer is working again, but not the regular trainer.

I'll keep an eye on facebookresearch/xformers#991

Thankyou and apreciate for your hardworks :) hope its solves soon..

alumcreek commented 6 months ago

Should we be watching this page for news of updates on this issue?

hollowstrawberry commented 6 months ago

Should be fixed now.

ridhoyp commented 6 months ago

thankyou, its already run..!

ErichEisner commented 6 months ago

I trained a LoRA with the new version ... but then I get an error message in A111 (NansException) ... happens only with that newly trained LoRA.

ridhoyp commented 6 months ago

I trained a LoRA with the new version ... but then I get an error message in A111 (NansException) ... happens only with that newly trained LoRA.

i just tried also, but still with epoch2 loras trained, its ok, doesnt have any error while doing txt2img

ErichEisner commented 6 months ago

Hm ... no idea then what my problem is. I trained a large number of LoRAs with this trainer / dataset maker before the recent problems ... and they all work in Stable Diffusion A111. Only the new one does not (and I created it a second time just to be sure - same result). Sorry if I misunderstand, but I'm using 10 Epochs ...

stadiffs commented 6 months ago

You're amazing, it looks like is working again, Thank you.

dazaibsd commented 6 months ago

You're amazing, it looks like is working again, Thank you.

Can you tell me what setup you use? For example, I use 10 epochs, 20 repeats, dim/alpha 32, and it works well for me, but now the Loras are broken.

ErichEisner commented 6 months ago

So it's not just me? 10 Epochs, 10 repeats, 2e-4 and 1e-4 ... Trainer completes training, but using LoRA results in error message about NansException. Using commandline --no-half, I get txt2img to run, but it's incredibly slow and the picture resembles more modern art than a photo of a woman for example.

ArtyomSosyrev commented 6 months ago

You're amazing, it looks like is working again, Thank you.

Can you tell me what setup you use? For example, I use 10 epochs, 20 repeats, dim/alpha 32, and it works well for me, but now the Loras are broken.

The same thing. However, I did not change the parameters at all.