Linaqruf
commented
1 year ago no, it runs well, nothing wrong with tensorflow notification, there is a bug in colab where it can't find tensorrt path but it doesn't affect the training. also bitsandbytes notification, it automatically find the path itself. Detected Cuda Version 11.2 means you're running on gpu not cpu
and why it's longer? because you set train_batch_size to 4 and max_train_steps to 5000, it's equal to 20k max train steps. You can try lower your train step or train batch size.
Also try to not set max train step too higher if your images is only 4, your model will overfitted easily
DarkAlchy
2023-01-26 19:54:55.306109: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2023-01-26 19:54:56.013115: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia 2023-01-26 19:54:56.013226: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia 2023-01-26 19:54:56.013246: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. found 0 images. no metadata / メタデータファイルがありません: /content/drive/MyDrive/fine_tune/meta_clean.json
/content/drive/MyDrive/kohya-trainer/finetune 2023-01-26 20:01:45.449924: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2023-01-26 20:01:46.157549: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia 2023-01-26 20:01:46.157670: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia 2023-01-26 20:01:46.157697: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. load images from /content/drive/MyDrive/Aliens found 4 images. loading BLIP caption: https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_large_caption.pth Downloading (…)solve/main/vocab.txt: 100% 232k/232k [00:00<00:00, 263kB/s] Downloading (…)okenizer_config.json: 100% 28.0/28.0 [00:00<00:00, 10.9kB/s] Downloading (…)lve/main/config.json: 100% 570/570 [00:00<00:00, 212kB/s] 100% 1.66G/1.66G [01:14<00:00, 24.0MB/s] load checkpoint from https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_large_caption.pth BLIP loaded 0% 0/4 [00:00<?, ?it/s]convert image mode RGBA to RGB: /content/drive/MyDrive/Aliens/maxresdefault.png 25% 1/4 [00:00<00:02, 1.03it/s]convert image mode RGBA to RGB: /content/drive/MyDrive/Aliens/newsgeek.png 50% 2/4 [00:01<00:01, 1.03it/s]convert image mode RGBA to RGB: /content/drive/MyDrive/Aliens/FGddysgUYAMQ-sJ.png 100% 4/4 [00:03<00:00, 1.06it/s] done! 2023-01-26 20:03:42.032881: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2023-01-26 20:03:42.958994: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.8/dist-packages/cv2/../../lib64:/usr/lib64-nvidia 2023-01-26 20:03:42.959110: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.8/dist-packages/cv2/../../lib64:/usr/lib64-nvidia 2023-01-26 20:03:42.959128: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. downloading wd14 tagger model from hf_hub /usr/local/lib/python3.8/dist-packages/huggingface_hub/file_download.py:1020: FutureWarning: The
force_filenameparameter is deprecated as a new caching system, which keeps the filenames as they are on the Hub, is now in place. warnings.warn( Downloading (…)"keras_metadata.pb";: 100% 328k/328k [00:00<00:00, 25.9MB/s] Downloading (…)"saved_model.pb";: 100% 3.81M/3.81M [00:00<00:00, 134MB/s] Downloading (…)in/selected_tags.csv: 100% 174k/174k [00:00<00:00, 259kB/s] Downloading (…)ata-00000-of-00001";: 100% 365M/365M [00:01<00:00, 280MB/s] Downloading (…)"variables.index";: 100% 13.8k/13.8k [00:00<00:00, 6.07MB/s] found 4 images. loading model and labels 2023-01-26 20:03:53.629195: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:42] Overriding orig_value setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0. WARNING:tensorflow:No training configuration found in save file, so the model was not compiled. Compile it manually. WARNING:tensorflow:No training configuration found in save file, so the model was not compiled. Compile it manually. 100% 4/4 [00:00<00:00, 23.32it/s] done!Went on ahead and tried to train to get these:
Looks like something is wrong and possibly using the CPU because this is longer than even a full on Dreambooth training.