search

Deci-AI / super-gradients

Easily train or fine-tune SOTA computer vision models with one open source training library. The home of Yolo-NAS.
https://www.supergradients.com
Apache License 2.0
4.59k stars 510 forks source link

Error installing super-gradients (or super_gradients) and training on colab. #1585

Closed johnomage closed 1 year ago

johnomage commented 1 year ago

🐛 Describe the bug

Error installing super-gradients (or super_gradients) on colab.

I got these errors/warnings while pip-installing SuperGradients library:

!pip install super_gradients==3.2.0

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.

lida 0.0.10 requires fastapi, which is not installed.

lida 0.0.10 requires kaleido, which is not installed.

lida 0.0.10 requires python-multipart, which is not installed.

lida 0.0.10 requires uvicorn, which is not installed.

tensorflow 2.14.0 requires numpy>=1.23.5, but you have numpy 1.23.0 which is incompatible.

Successfully installed Deprecated-1.2.14 antlr4-python3-runtime-4.9.3 boto3-1.28.73 botocore-1.31.73 coloredlogs-15.0.1 coverage-5.3.1 docutils-0.17.1 einops-0.3.2 humanfriendly-10.0 hydra-core-1.3.2 jmespath-1.0.1 json-tricks-3.16.1 numpy-1.23.0 omegaconf-2.3.0 onnx-1.13.0 onnx-simplifier-0.4.35 onnxruntime-1.13.1 pyDeprecate-0.3.2 pycocotools-2.0.6 pyparsing-2.4.5 rapidfuzz-3.4.0 s3transfer-0.7.0 sphinx-4.0.3 sphinx-rtd-theme-1.3.0 sphinxcontrib-applehelp-1.0.4 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-2.0.1 sphinxcontrib-jquery-4.1 sphinxcontrib-qthelp-1.0.3 sphinxcontrib-serializinghtml-1.1.5 stringcase-1.2.0 super_gradients-3.2.0 termcolor-1.1.0 torchmetrics-0.8.0 treelib-1.6.1

WARNING: The following packages were previously imported in this runtime:

[numpy,pydevd_plugins,pyparsing,sphinxcontrib]

You must restart the runtime in order to use newly installed versions.

[2023-10-29 02:25:13] INFO - crash_tips_setup.py - Crash tips is enabled. You can set your environment variable to CRASH_HANDLER=FALSE to disable it

The console stream is logged into /root/sg_logs/console.log

[2023-10-29 02:25:15] WARNING - init.py - Failed to import pytorch_quantization

[2023-10-29 02:25:17] INFO - utils.py - NumExpr defaulting to 2 threads.

/usr/local/lib/python3.10/dist-packages/_distutils_hack/init.py:33: UserWarning: Setuptools is replacing distutils.

warnings.warn("Setuptools is replacing distutils.")

[2023-10-29 02:25:19] WARNING - calibrator.py - Failed to import pytorch_quantization

[2023-10-29 02:25:19] WARNING - export.py - Failed to import pytorch_quantization

[2023-10-29 02:25:19] WARNING - selective_quantization_utils.py - Failed to import pytorch_quantization

[2023-10-29 02:25:19] WARNING - env_sanity_check.py - Failed to verify installed packages: boto3 required but not found

[2023-10-29 02:25:19] WARNING - env_sanity_check.py - Failed to verify installed packages: deprecated required but not found

[2023-10-29 02:25:19] WARNING - env_sanity_check.py - Failed to verify installed packages: coverage required but not found

[2023-10-29 02:25:19] WARNING - env_sanity_check.py - Failed to verify installed packages: sphinx-rtd-theme required but not found

[2023-10-29 02:25:19] WARNING - env_sanity_check.py - Failed to verify installed packages: torchmetrics required but not found

[2023-10-29 02:25:19] WARNING - env_sanity_check.py - Failed to verify installed packages: hydra-core required but not found

[2023-10-29 02:25:19] WARNING - env_sanity_check.py - Failed to verify installed packages: omegaconf required but not found

[2023-10-29 02:25:19] WARNING - env_sanity_check.py - Failed to verify installed packages: onnxruntime required but not found

[2023-10-29 02:25:19] WARNING - env_sanity_check.py - Failed to verify installed packages: onnx required but not found

[2023-10-29 02:25:19] WARNING - env_sanity_check.py - Failed to verify installed packages: einops required but not found

[2023-10-29 02:25:19] WARNING - env_sanity_check.py - Failed to verify installed packages: treelib required but not found

[2023-10-29 02:25:19] WARNING - env_sanity_check.py - Failed to verify installed packages: stringcase required but not found

[2023-10-29 02:25:19] WARNING - env_sanity_check.py - Failed to verify installed packages: rapidfuzz required but not found

[2023-10-29 02:25:19] WARNING - env_sanity_check.py - Failed to verify installed packages: json-tricks required but not found

[2023-10-29 02:25:19] WARNING - env_sanity_check.py - Failed to verify installed packages: onnx-simplifier required but not found.

Also for:

train_params = {"initial_lr": config.INITIAL_LR*1000,
                "max_epochs": config.EPOCHS//10,
                "loss": config.LOSS,
                "optimizer": config.OPTIMIZER,
                "train_metrics_list": ["Accuracy", "Top5"],
                "val_metrics_list": ["Accuracy", "Top5"],
                "metric_to_watch": "Accuracy",

                # optimizers
                "optimizers_params": {},
                "criterion_params": {"smooth_eps": 0.1},
                "warmup_initial_lr": 3e-4,
                "warmup_mode": "linear_epoch_step",
                "lr_warmup_epochs": 3,
                "lr_warmup_steps": 2,
                "lr_cooldown_epochs": 1
                }

trainer.train(model=model,
                   training_params=train_params,
                   train_loader=train_loader,
                  valid_loader=val_loader)

i got: ValueError: 'accuracy' is not in list.

Anyone else got these?

Versions

--2023-10-29 02:44:04-- https://raw.githubusercontent.com/pytorch/pytorch/main/torch/utils/collect_env.py Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ... Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 21737 (21K) [text/plain] Saving to: ‘collect_env.py’

collect_env.py 100%[===================>] 21.23K --.-KB/s in 0s

2023-10-29 02:44:04 (67.3 MB/s) - ‘collect_env.py’ saved [21737/21737]

Collecting environment information... PyTorch version: 2.1.0+cu118 Is debug build: False CUDA used to build PyTorch: 11.8 ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.2 LTS (x86_64) GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version: 14.0.0-1ubuntu1.1 CMake version: version 3.27.7 Libc version: glibc-2.35

Python version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime) Python platform: Linux-5.15.120+-x86_64-with-glibc2.35 Is CUDA available: True CUDA runtime version: 11.8.89 CUDA_MODULE_LOADING set to: LAZY GPU models and configuration: GPU 0: Tesla T4 Nvidia driver version: 525.105.17 cuDNN version: Probably one of the following: /usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.0 /usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.0 /usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.0 /usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.0 /usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.0 /usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.0 /usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.0 HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True

CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 2 On-line CPU(s) list: 0,1 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) CPU @ 2.20GHz CPU family: 6 Model: 79 Thread(s) per core: 2 Core(s) per socket: 1 Socket(s): 1 Stepping: 0 BogoMIPS: 4399.99 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat md_clear arch_capabilities Hypervisor vendor: KVM Virtualization type: full L1d cache: 32 KiB (1 instance) L1i cache: 32 KiB (1 instance) L2 cache: 256 KiB (1 instance) L3 cache: 55 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0,1 Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Mitigation; PTE Inversion Vulnerability Mds: Vulnerable; SMT Host state unknown Vulnerability Meltdown: Vulnerable Vulnerability Mmio stale data: Vulnerable Vulnerability Retbleed: Vulnerable Vulnerability Spec store bypass: Vulnerable Vulnerability Spectre v1: Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers Vulnerability Spectre v2: Vulnerable, IBPB: disabled, STIBP: disabled, PBRSB-eIBRS: Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Vulnerable

Versions of relevant libraries: [pip3] numpy==1.23.0 [pip3] onnx==1.13.0 [pip3] onnx-simplifier==0.4.35 [pip3] onnxruntime==1.13.1 [pip3] torch==2.1.0+cu118 [pip3] torchaudio==2.1.0+cu118 [pip3] torchdata==0.7.0 [pip3] torchmetrics==0.8.0 [pip3] torchsummary==1.5.1 [pip3] torchtext==0.16.0 [pip3] torchvision==0.16.0+cu118 [pip3] triton==2.1.0 [conda] Could not collect

BloodAxe commented 1 year ago

Can you please share the link of this notebook (If it's a public one)? Also, try latest 3.3.1 release, maybe you will have better luck with this version

johnomage commented 1 year ago

Can you please share the link of this notebook (If it's a public one)? Also, try latest 3.3.1 release, maybe you will have better luck with this version

Hi Thanks for helping.

Here's the link to the notebook

ikranergiz commented 1 year ago

I encountered the same error when I ran !pip install super-gradients==3.3.1.

To resolve this, I installed the packages emerged in the error message.

!pip install pytube --upgrade
!pip install fastapi
!pip install kaleido
!pip install python-multipart
!pip install uvicorn
!pip install pyparsing==2.4.7
!pip install numpy==1.23.5

When I ran code below,

from super_gradients.training import Trainer

CHECKPOINT_DIR = 'CHECKPOINT-PATH'
trainer = Trainer(experiment_name='yolonas_run', ckpt_root_dir=CHECKPOINT_DIR)

I also got an error and installed packages in the error messages as in the previous step. 

!pip install boto3
!pip install deprecated
!pip install coverage 
!pip install sphinx-rtd-theme
!pip install torchmetrics 
!pip install hydra-core
!pip install omegaconf 
!pip install onnxruntime 
!pip install onnx 
!pip install einops 
!pip install treelib 
!pip install stringcase 
!pip install rapidfuzz 
!pip install json-tricks
!pip install onnx-simplifier
!pip install data-gradients

And it works for me! I hope it works for you too.

If you'd like to take a look my notebook, here's the link yolonas-dataverse - 1

johnomage commented 1 year ago

I encountered the same error when I ran !pip install super-gradients==3.3.1.

To resolve this, I installed the packages emerged in the error message.

!pip install pytube --upgrade
!pip install fastapi
!pip install kaleido
!pip install python-multipart
!pip install uvicorn
!pip install pyparsing==2.4.7
!pip install numpy==1.23.5

When I ran code below,

from super_gradients.training import Trainer

CHECKPOINT_DIR = 'CHECKPOINT-PATH'
trainer = Trainer(experiment_name='yolonas_run', ckpt_root_dir=CHECKPOINT_DIR)

I also got an error and installed packages in the error messages as in the previous step.

!pip install boto3
!pip install deprecated
!pip install coverage 
!pip install sphinx-rtd-theme
!pip install torchmetrics 
!pip install hydra-core
!pip install omegaconf 
!pip install onnxruntime 
!pip install onnx 
!pip install einops 
!pip install treelib 
!pip install stringcase 
!pip install rapidfuzz 
!pip install json-tricks
!pip install onnx-simplifier
!pip install data-gradients

And it works for me! I hope it works for you too.

If you'd like to take a look my notebook, here's the link yolonas-dataverse - 1

Thanks mate. This worked.

.train() is running smoothly.

ikranergiz commented 1 year ago

I've discovered a new and simple solution. After downloading the super-gradients package, it's necessary to restart the notebook. This is mentioned in the YOLO-NAS notebook on Kaggle in the installation process. If you'd like to take a look, here's the link: Intro to SuperGradients + YOLONAS Starter Notebook

Hamamdk commented 1 year ago

Hi sir can you help me ,DreamBooth_Stable_Diffusion.ipynb can't working

Hamamdk commented 1 year ago

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. lida 0.0.10 requires kaleido, which is not installed. llmx 0.0.15a0 requires cohere, which is not installed. llmx 0.0.15a0 requires openai, which is not installed. llmx 0.0.15a0 requires tiktoken, which is not installed. tensorflow-probability 0.22.0 requires typing-extensions<4.6.0, but you have typing-extensions 4.8.0 which is incompatible.

Hamamdk commented 1 year ago

WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for: PyTorch 2.1.0+cu121 with CUDA 1201 (you have 2.1.0+cu118) Python 3.10.13 (you have 3.10.12) Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers) Memory-efficient attention, SwiGLU, sparse and more won't be available. Set XFORMERS_MORE_DETAILS=1 for more details 2023-11-06 05:57:58.909288: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2023-11-06 05:57:58.909344: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2023-11-06 05:57:58.909391: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2023-11-06 05:58:00.548575: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT /usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_hf_folder.py:98: UserWarning: A token has been found in /root/.huggingface/token. This is the old path where tokens were stored. The new location is /root/.cache/huggingface/token which is configurable using HF_HOME environment variable. Your token has been copied to this new location. You can now safely delete the old token file manually or use huggingface-cli logout. warnings.warn( Downloading (…)lve/main/config.json: 100% 547/547 [00:00<00:00, 2.91MB/s] Downloading (…)ch_model.safetensors: 100% 335M/335M [00:04<00:00, 72.4MB/s] Downloading (…)p16/model_index.json: 100% 543/543 [00:00<00:00, 3.32MB/s] text_encoder/model.safetensors not found Fetching 15 files: 0% 0/15 [00:00<?, ?it/s] Downloading pytorch_model.bin: 0% 0.00/246M [00:00<?, ?B/s]

Downloading (…)_encoder/config.json: 100% 636/636 [00:00<00:00, 3.80MB/s]

Downloading (…)tokenizer/merges.txt: 0% 0.00/525k [00:00<?, ?B/s]

Downloading (…)rocessor_config.json: 100% 342/342 [00:00<00:00, 2.08MB/s] Fetching 15 files: 7% 1/15 [00:00<00:11, 1.20it/s]

Downloading (…)_checker/config.json: 100% 4.70k/4.70k [00:00<00:00, 25.3MB/s]

Downloading (…)cheduler_config.json: 100% 307/307 [00:00<00:00, 2.06MB/s]

Downloading (…)cial_tokens_map.json: 100% 472/472 [00:00<00:00, 3.31MB/s]

Downloading pytorch_model.bin: 0% 0.00/608M [00:00<?, ?B/s]

Downloading pytorch_model.bin: 2% 10.5M/608M [00:00<00:05, 99.9MB/s]

Downloading (…)okenizer_config.json: 100% 822/822 [00:00<00:00, 3.81MB/s]

Downloading (…)tokenizer/merges.txt: 100% 525k/525k [00:00<00:00, 2.58MB/s]

Downloading pytorch_model.bin: 4% 10.5M/246M [00:00<00:05, 43.2MB/s]

Downloading pytorch_model.bin: 5% 31.5M/608M [00:00<00:03, 150MB/s]

Downloading (…)on_pytorch_model.bin: 0% 0.00/1.72G [00:00<?, ?B/s]

Downloading (…)tokenizer/vocab.json: 0% 0.00/1.06M [00:00<?, ?B/s]

Downloading (…)cc2/unet/config.json: 100% 806/806 [00:00<00:00, 3.36MB/s]

Downloading pytorch_model.bin: 13% 31.5M/246M [00:00<00:02, 89.2MB/s]

Downloading pytorch_model.bin: 10% 62.9M/608M [00:00<00:02, 202MB/s]

Downloading (…)tokenizer/vocab.json: 100% 1.06M/1.06M [00:00<00:00, 7.69MB/s]

Downloading (…)9cc2/vae/config.json: 100% 609/609 [00:00<00:00, 2.78MB/s]

Downloading (…)on_pytorch_model.bin: 1% 10.5M/1.72G [00:00<00:38, 44.7MB/s] Downloading pytorch_model.bin: 21% 52.4M/246M [00:00<00:01, 97.9MB/s]

Downloading pytorch_model.bin: 14% 83.9M/608M [00:00<00:03, 150MB/s]

Downloading (…)on_pytorch_model.bin: 1% 21.0M/1.72G [00:00<00:26, 62.9MB/s] Downloading pytorch_model.bin: 30% 73.4M/246M [00:00<00:01, 104MB/s]

Downloading (…)on_pytorch_model.bin: 0% 0.00/167M [00:00<?, ?B/s]

Downloading pytorch_model.bin: 17% 105M/608M [00:00<00:03, 134MB/s]

Downloading (…)on_pytorch_model.bin: 2% 41.9M/1.72G [00:00<00:19, 84.7MB/s] Downloading pytorch_model.bin: 38% 94.4M/246M [00:00<00:01, 116MB/s]

Downloading pytorch_model.bin: 21% 126M/608M [00:00<00:03, 136MB/s]

Downloading (…)on_pytorch_model.bin: 4% 62.9M/1.72G [00:00<00:16, 103MB/s]

Downloading (…)on_pytorch_model.bin: 6% 10.5M/167M [00:00<00:04, 39.2MB/s] Downloading pytorch_model.bin: 47% 115M/246M [00:01<00:01, 110MB/s]

Downloading (…)on_pytorch_model.bin: 13% 21.0M/167M [00:00<00:02, 57.9MB/s]

Downloading pytorch_model.bin: 24% 147M/608M [00:01<00:03, 119MB/s]

Downloading (…)on_pytorch_model.bin: 5% 83.9M/1.72G [00:00<00:15, 107MB/s] Downloading pytorch_model.bin: 55% 136M/246M [00:01<00:01, 96.9MB/s]

Downloading (…)on_pytorch_model.bin: 25% 41.9M/167M [00:00<00:01, 69.9MB/s]

Downloading (…)on_pytorch_model.bin: 6% 105M/1.72G [00:01<00:17, 93.8MB/s]

Downloading pytorch_model.bin: 28% 168M/608M [00:01<00:04, 95.1MB/s] Downloading pytorch_model.bin: 60% 147M/246M [00:01<00:01, 89.8MB/s]

Downloading (…)on_pytorch_model.bin: 31% 52.4M/167M [00:00<00:01, 65.3MB/s]

Downloading (…)on_pytorch_model.bin: 7% 115M/1.72G [00:01<00:19, 80.7MB/s] Downloading pytorch_model.bin: 64% 157M/246M [00:01<00:01, 84.5MB/s]

Downloading (…)on_pytorch_model.bin: 38% 62.9M/167M [00:00<00:01, 65.0MB/s]

Downloading (…)on_pytorch_model.bin: 7% 126M/1.72G [00:01<00:19, 80.0MB/s]

Downloading pytorch_model.bin: 31% 189M/608M [00:01<00:05, 78.6MB/s] Downloading pytorch_model.bin: 68% 168M/246M [00:01<00:01, 75.5MB/s]

Downloading (…)on_pytorch_model.bin: 8% 136M/1.72G [00:01<00:21, 74.0MB/s]

Downloading (…)on_pytorch_model.bin: 44% 73.4M/167M [00:01<00:01, 59.8MB/s]

Downloading pytorch_model.bin: 33% 199M/608M [00:01<00:05, 76.2MB/s] Downloading pytorch_model.bin: 72% 178M/246M [00:02<00:00, 69.7MB/s]

Downloading (…)on_pytorch_model.bin: 50% 83.9M/167M [00:01<00:01, 62.1MB/s]

Downloading (…)on_pytorch_model.bin: 9% 147M/1.72G [00:01<00:22, 70.4MB/s]

Downloading pytorch_model.bin: 34% 210M/608M [00:02<00:05, 72.9MB/s] Downloading pytorch_model.bin: 77% 189M/246M [00:02<00:00, 68.6MB/s]

Downloading (…)on_pytorch_model.bin: 56% 94.4M/167M [00:01<00:01, 63.6MB/s]

Downloading (…)on_pytorch_model.bin: 9% 157M/1.72G [00:01<00:22, 70.7MB/s]

Downloading pytorch_model.bin: 36% 220M/608M [00:02<00:05, 70.6MB/s] Downloading pytorch_model.bin: 81% 199M/246M [00:02<00:00, 67.1MB/s]

Downloading (…)on_pytorch_model.bin: 10% 168M/1.72G [00:02<00:21, 70.7MB/s]

Downloading (…)on_pytorch_model.bin: 63% 105M/167M [00:01<00:00, 63.9MB/s]

Downloading pytorch_model.bin: 38% 231M/608M [00:02<00:06, 61.8MB/s] Downloading pytorch_model.bin: 85% 210M/246M [00:02<00:00, 45.6MB/s]

Downloading (…)on_pytorch_model.bin: 10% 178M/1.72G [00:02<00:32, 47.9MB/s]

Downloading pytorch_model.bin: 40% 241M/608M [00:02<00:06, 53.1MB/s]

Downloading (…)on_pytorch_model.bin: 69% 115M/167M [00:02<00:01, 43.4MB/s]

Downloading (…)on_pytorch_model.bin: 11% 189M/1.72G [00:02<00:28, 54.1MB/s]

Downloading pytorch_model.bin: 41% 252M/608M [00:03<00:06, 53.3MB/s]

Downloading (…)on_pytorch_model.bin: 75% 126M/167M [00:02<00:00, 45.6MB/s] Downloading pytorch_model.bin: 89% 220M/246M [00:03<00:00, 42.1MB/s]

Downloading (…)on_pytorch_model.bin: 12% 199M/1.72G [00:02<00:25, 58.8MB/s]

Downloading (…)on_pytorch_model.bin: 12% 210M/1.72G [00:02<00:22, 67.5MB/s] Downloading pytorch_model.bin: 94% 231M/246M [00:03<00:00, 46.0MB/s]

Downloading pytorch_model.bin: 43% 262M/608M [00:03<00:06, 50.7MB/s]

Downloading (…)on_pytorch_model.bin: 13% 220M/1.72G [00:03<00:20, 71.7MB/s]

Downloading pytorch_model.bin: 45% 273M/608M [00:03<00:06, 49.3MB/s]

Downloading (…)on_pytorch_model.bin: 88% 147M/167M [00:02<00:00, 45.6MB/s] Downloading pytorch_model.bin: 98% 241M/246M [00:03<00:00, 44.4MB/s]

Downloading (…)on_pytorch_model.bin: 13% 231M/1.72G [00:03<00:24, 61.0MB/s] Downloading pytorch_model.bin: 100% 246M/246M [00:03<00:00, 43.6MB/s]

Downloading pytorch_model.bin: 100% 246M/246M [00:03<00:00, 66.4MB/s]

Downloading (…)on_pytorch_model.bin: 94% 157M/167M [00:02<00:00, 49.0MB/s]

Downloading (…)on_pytorch_model.bin: 14% 241M/1.72G [00:03<00:23, 62.3MB/s]

Downloading pytorch_model.bin: 48% 294M/608M [00:03<00:05, 60.8MB/s]

Downloading (…)on_pytorch_model.bin: 100% 167M/167M [00:03<00:00, 55.8MB/s]

Downloading (…)on_pytorch_model.bin: 100% 167M/167M [00:03<00:00, 54.4MB/s]

Downloading (…)on_pytorch_model.bin: 16% 273M/1.72G [00:03<00:15, 93.3MB/s]

Downloading pytorch_model.bin: 52% 315M/608M [00:03<00:03, 77.5MB/s]

Downloading (…)on_pytorch_model.bin: 17% 294M/1.72G [00:03<00:13, 109MB/s]

Downloading pytorch_model.bin: 55% 336M/608M [00:04<00:02, 93.0MB/s]

Downloading (…)on_pytorch_model.bin: 18% 315M/1.72G [00:03<00:12, 117MB/s]

Downloading pytorch_model.bin: 59% 357M/608M [00:04<00:02, 107MB/s]

Downloading pytorch_model.bin: 62% 377M/608M [00:04<00:01, 116MB/s]

Downloading (…)on_pytorch_model.bin: 20% 336M/1.72G [00:04<00:11, 122MB/s]

Downloading (…)on_pytorch_model.bin: 21% 357M/1.72G [00:04<00:10, 132MB/s]

Downloading pytorch_model.bin: 66% 398M/608M [00:04<00:01, 122MB/s]

Downloading (…)on_pytorch_model.bin: 22% 377M/1.72G [00:04<00:09, 136MB/s]

Downloading pytorch_model.bin: 69% 419M/608M [00:04<00:01, 128MB/s]

Downloading (…)on_pytorch_model.bin: 23% 398M/1.72G [00:04<00:09, 138MB/s]

Downloading pytorch_model.bin: 72% 440M/608M [00:04<00:01, 135MB/s]

Downloading (…)on_pytorch_model.bin: 24% 419M/1.72G [00:04<00:09, 136MB/s]

Downloading pytorch_model.bin: 76% 461M/608M [00:04<00:01, 132MB/s]

Downloading (…)on_pytorch_model.bin: 26% 440M/1.72G [00:04<00:08, 143MB/s]

Downloading pytorch_model.bin: 79% 482M/608M [00:05<00:00, 129MB/s]

Downloading (…)on_pytorch_model.bin: 27% 461M/1.72G [00:05<00:08, 141MB/s]

Downloading pytorch_model.bin: 83% 503M/608M [00:05<00:00, 137MB/s]

Downloading (…)on_pytorch_model.bin: 28% 482M/1.72G [00:05<00:08, 144MB/s]

Downloading pytorch_model.bin: 86% 524M/608M [00:05<00:00, 140MB/s]

Downloading (…)on_pytorch_model.bin: 29% 503M/1.72G [00:05<00:08, 144MB/s]

Downloading pytorch_model.bin: 90% 545M/608M [00:05<00:00, 144MB/s]

Downloading (…)on_pytorch_model.bin: 30% 524M/1.72G [00:05<00:08, 139MB/s]

Downloading pytorch_model.bin: 93% 566M/608M [00:05<00:00, 123MB/s]

Downloading pytorch_model.bin: 97% 587M/608M [00:05<00:00, 134MB/s]

Downloading (…)on_pytorch_model.bin: 32% 545M/1.72G [00:05<00:09, 120MB/s]

Downloading pytorch_model.bin: 100% 608M/608M [00:06<00:00, 141MB/s]

Downloading pytorch_model.bin: 100% 608M/608M [00:06<00:00, 100MB/s] Fetching 15 files: 27% 4/15 [00:06<00:19, 1.81s/it]

Downloading (…)on_pytorch_model.bin: 34% 587M/1.72G [00:05<00:07, 143MB/s]

Downloading (…)on_pytorch_model.bin: 36% 619M/1.72G [00:06<00:06, 165MB/s]

Downloading (…)on_pytorch_model.bin: 38% 650M/1.72G [00:06<00:05, 182MB/s]

Downloading (…)on_pytorch_model.bin: 40% 682M/1.72G [00:06<00:05, 191MB/s]

Downloading (…)on_pytorch_model.bin: 41% 703M/1.72G [00:06<00:05, 173MB/s]

Downloading (…)on_pytorch_model.bin: 42% 724M/1.72G [00:06<00:05, 173MB/s]

Downloading (…)on_pytorch_model.bin: 44% 755M/1.72G [00:06<00:04, 195MB/s]

Downloading (…)on_pytorch_model.bin: 46% 786M/1.72G [00:06<00:04, 208MB/s]

Downloading (…)on_pytorch_model.bin: 48% 818M/1.72G [00:07<00:04, 220MB/s]

Downloading (…)on_pytorch_model.bin: 49% 849M/1.72G [00:07<00:03, 222MB/s]

Downloading (…)on_pytorch_model.bin: 51% 881M/1.72G [00:07<00:03, 225MB/s]

Downloading (…)on_pytorch_model.bin: 53% 912M/1.72G [00:07<00:03, 225MB/s]

Downloading (…)on_pytorch_model.bin: 55% 944M/1.72G [00:07<00:03, 227MB/s]

Downloading (…)on_pytorch_model.bin: 57% 975M/1.72G [00:07<00:03, 233MB/s]

Downloading (…)on_pytorch_model.bin: 59% 1.01G/1.72G [00:07<00:03, 233MB/s]

Downloading (…)on_pytorch_model.bin: 60% 1.04G/1.72G [00:07<00:02, 228MB/s]

Downloading (…)on_pytorch_model.bin: 62% 1.07G/1.72G [00:08<00:02, 238MB/s]

Downloading (…)on_pytorch_model.bin: 64% 1.10G/1.72G [00:08<00:02, 241MB/s]

Downloading (…)on_pytorch_model.bin: 66% 1.13G/1.72G [00:08<00:02, 249MB/s]

Downloading (…)on_pytorch_model.bin: 68% 1.16G/1.72G [00:08<00:02, 249MB/s]

Downloading (…)on_pytorch_model.bin: 70% 1.20G/1.72G [00:08<00:02, 242MB/s]

Downloading (…)on_pytorch_model.bin: 71% 1.23G/1.72G [00:08<00:02, 218MB/s]

Downloading (…)on_pytorch_model.bin: 73% 1.26G/1.72G [00:08<00:02, 215MB/s]

Downloading (…)on_pytorch_model.bin: 75% 1.29G/1.72G [00:09<00:01, 217MB/s]

Downloading (…)on_pytorch_model.bin: 77% 1.32G/1.72G [00:09<00:01, 222MB/s]

Downloading (…)on_pytorch_model.bin: 79% 1.35G/1.72G [00:09<00:01, 227MB/s]

Downloading (…)on_pytorch_model.bin: 81% 1.38G/1.72G [00:09<00:01, 222MB/s]

Downloading (…)on_pytorch_model.bin: 82% 1.42G/1.72G [00:09<00:01, 228MB/s]

Downloading (…)on_pytorch_model.bin: 84% 1.45G/1.72G [00:09<00:01, 235MB/s]

Downloading (…)on_pytorch_model.bin: 86% 1.48G/1.72G [00:09<00:01, 235MB/s]

Downloading (…)on_pytorch_model.bin: 88% 1.51G/1.72G [00:09<00:00, 243MB/s]

Downloading (…)on_pytorch_model.bin: 90% 1.54G/1.72G [00:10<00:00, 254MB/s]

Downloading (…)on_pytorch_model.bin: 91% 1.57G/1.72G [00:10<00:00, 267MB/s]

Downloading (…)on_pytorch_model.bin: 93% 1.60G/1.72G [00:10<00:00, 271MB/s]

Downloading (…)on_pytorch_model.bin: 95% 1.64G/1.72G [00:10<00:00, 266MB/s]

Downloading (…)on_pytorch_model.bin: 97% 1.67G/1.72G [00:10<00:00, 273MB/s]

Downloading (…)on_pytorch_model.bin: 100% 1.72G/1.72G [00:10<00:00, 160MB/s] Fetching 15 files: 100% 15/15 [00:11<00:00, 1.26it/s] /usr/local/lib/python3.10/dist-packages/transformers/models/clip/feature_extraction_clip.py:28: FutureWarning: The class CLIPFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use CLIPImageProcessor instead. warnings.warn( You have disabled the safety checker for <class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline'> by passing safety_checker=None. Ensure that you abide to the conditions of the Stable Diffusion license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances, disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 . Traceback (most recent call last): File "/content/train_dreambooth.py", line 869, in main(args) File "/content/train_dreambooth.py", line 487, in main pipeline.enable_xformers_memory_efficient_attention() File "/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/pipeline_utils.py", line 1322, in enable_xformers_memory_efficient_attention self.set_use_memory_efficient_attention_xformers(True, attention_op) File "/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/pipeline_utils.py", line 1347, in set_use_memory_efficient_attention_xformers fn_recursive_set_mem_eff(module) File "/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/pipeline_utils.py", line 1338, in fn_recursive_set_mem_eff module.set_use_memory_efficient_attention_xformers(valid, attention_op) File "/usr/local/lib/python3.10/dist-packages/diffusers/models/modeling_utils.py", line 219, in set_use_memory_efficient_attention_xformers fn_recursive_set_mem_eff(module) File "/usr/local/lib/python3.10/dist-packages/diffusers/models/modeling_utils.py", line 215, in fn_recursive_set_mem_eff fn_recursive_set_mem_eff(child) File "/usr/local/lib/python3.10/dist-packages/diffusers/models/modeling_utils.py", line 215, in fn_recursive_set_mem_eff fn_recursive_set_mem_eff(child) File "/usr/local/lib/python3.10/dist-packages/diffusers/models/modeling_utils.py", line 215, in fn_recursive_set_mem_eff fn_recursive_set_mem_eff(child) File "/usr/local/lib/python3.10/dist-packages/diffusers/models/modeling_utils.py", line 212, in fn_recursive_set_mem_eff module.set_use_memory_efficient_attention_xformers(valid, attention_op) File "/usr/local/lib/python3.10/dist-packages/diffusers/models/modeling_utils.py", line 219, in set_use_memory_efficient_attention_xformers fn_recursive_set_mem_eff(module) File "/usr/local/lib/python3.10/dist-packages/diffusers/models/modeling_utils.py", line 215, in fn_recursive_set_mem_eff fn_recursive_set_mem_eff(child) File "/usr/local/lib/python3.10/dist-packages/diffusers/models/modeling_utils.py", line 215, in fn_recursive_set_mem_eff fn_recursive_set_mem_eff(child) File "/usr/local/lib/python3.10/dist-packages/diffusers/models/modeling_utils.py", line 212, in fn_recursive_set_mem_eff module.set_use_memory_efficient_attention_xformers(valid, attention_op) File "/usr/local/lib/python3.10/dist-packages/diffusers/models/attention_processor.py", line 151, in set_use_memory_efficient_attention_xformers raise e File "/usr/local/lib/python3.10/dist-packages/diffusers/models/attention_processor.py", line 145, in set_use_memory_efficient_attentionxformers = xformers.ops.memory_efficient_attention( File "/usr/local/lib/python3.10/dist-packages/xformers/ops/fmha/init.py", line 223, in memory_efficient_attention return _memory_efficient_attention( File "/usr/local/lib/python3.10/dist-packages/xformers/ops/fmha/init.py", line 321, in _memory_efficient_attention return _memory_efficient_attention_forward( File "/usr/local/lib/python3.10/dist-packages/xformers/ops/fmha/init.py", line 337, in _memory_efficient_attention_forward op = _dispatch_fw(inp, False) File "/usr/local/lib/python3.10/dist-packages/xformers/ops/fmha/dispatch.py", line 120, in _dispatch_fw return _run_priority_list( File "/usr/local/lib/python3.10/dist-packages/xformers/ops/fmha/dispatch.py", line 63, in _run_priority_list raise NotImplementedError(msg) NotImplementedError: No operator found for memory_efficient_attention_forward with inputs: query : shape=(1, 2, 1, 40) (torch.float32) key : shape=(1, 2, 1, 40) (torch.float32) value : shape=(1, 2, 1, 40) (torch.float32) attn_bias : <class 'NoneType'> p : 0.0 decoderF is not supported because: xFormers wasn't build with CUDA support attn_bias type is <class 'NoneType'> operator wasn't built - see python -m xformers.info for more info flshattF@0.0.0 is not supported because: xFormers wasn't build with CUDA support requires device with capability > (8, 0) but your GPU has capability (7, 5) (too old) dtype=torch.float32 (supported: {torch.float16, torch.bfloat16}) operator wasn't built - see python -m xformers.info for more info tritonflashattF is not supported because: xFormers wasn't build with CUDA support requires device with capability > (8, 0) but your GPU has capability (7, 5) (too old) dtype=torch.float32 (supported: {torch.float16, torch.bfloat16}) operator wasn't built - see python -m xformers.info for more info triton is not available requires GPU with sm80 minimum compute capacity, e.g., A100/H100/L4 Only work on pre-MLIR triton for now cutlassF is not supported because: xFormers wasn't build with CUDA support operator wasn't built - see python -m xformers.info for more info smallkF is not supported because: max(query.shape[-1] != value.shape[-1]) > 32 xFormers wasn't build with CUDA support operator wasn't built - see python -m xformers.info for more info unsupported embed per head: 40