[Errno 36] File name too long: Generating class images fails when many tags in [filewords]

[art-phony]

Kindly read this ENTIRE form and fill it out with the requested information. It may seem tedious, but it's even more tedious asking basic questions to ensure I'm not trying to troubleshoot a bug I've already fixed. Help me help you, answer all the questions and follow the steps below before posting.

Please find the following lines in the console and paste them below. If you do not provide this information, your issue will be automatically closed.

Python 3.9.13 (main, May 23 2022, 22:01:06) 
[GCC 9.4.0]
Commit hash: 4b3c5bc24bffdf429c463a465763b3077fe55eb8
Installing requirements for Web UI
Checking Dreambooth requirements...
Dreambooth revision is 1d7d34110005e7930b589546ff5d82ce16cdcadd
[+] bitsandbytes version 0.35.0 installed.
[+] diffusers version 0.8.1 installed.
[+] transformers version 4.21.0 installed.
[ ] torch version 1.12.1+cu116 installed.
[ ] torchvision version 0.13.1+cu116 installed.
[ ] xformers version 0.0.15.dev0+103e863.d20221206 installed.

Have you read the Readme? Yes

Have you completely restarted the stable-diffusion-webUI, not just reloaded the UI? Yes

Have you updated Dreambooth to the latest revision? Yes

commit 88cc50ace470580842b519115545b9a08ebe115e (HEAD -> main, origin/main, origin/HEAD)
Merge: f696c73 252573b
Author: d8ahazard <d8ahazard@gmail.com>
Date:   Tue Dec 6 13:26:01 2022 -0600

    Merge pull request #427 from Janca/patch-2

    Disabled safety checker

Have you updated the Stable-Diffusion-WebUI to the latest version? Yes

No, really. Please save us both some trouble and update the SD-WebUI and Extension and restart before posting this. Reply 'OK' Below to acknowledge that you did this. OK

Describe the bug

In the initial stage of starting Dreambooth training, an error occurs in the part where the class image is generated, saying "Exception training model: [Errno 36] File name too long". As a result, Dreambooth training cannot be started.

Provide logs

If a crash has occurred, please provide the entire stack trace from the log, including the last few log messages before the crash occurred.

/notebooks/stable-diffusion-webui
Dreambooth API layer loaded
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Loading weights [38c1ebe3] from /notebooks/stable-diffusion-webui/models/Stable-diffusion/Anything-V3.0-pruned-fp16.ckpt
Loading VAE weights from: /notebooks/stable-diffusion-webui/models/VAE/Anything-V3.0.vae.pt
Applying xformers cross attention optimization.
Model loaded.
Loaded a total of 0 textual inversion embeddings.
Embeddings: 
Running on local URL:  http://127.0.0.1:7860/
Running on public URL: https://xxxxxxxxxxxxxxxx.gradio.app/

This share link expires in 72 hours. For free permanent hosting and GPU upgrades (NEW!), check out Spaces: https://huggingface.co/spaces
Returning [0.9, 0.999, 1e-08, 0.01, 'xformers', False, '', 1, True, True, True, 1.72e-07, 'constant', 0, 1, 75, 16000, 'fp16', True, 1, True, '', 1, 768, 1, False, 3000, 3000, False, 1, True, True, False, False, False, '', 7.5, 40, '', '[filewords]', 'girl', 'Description', '/notebooks/training_data/dreambooth_training/2_MVスクショ_768', '[filewords]', 'yoisakikanade', -1, 1, 20000, -1, 7.5, 40, 'longbody, lowres, bad anatomy, bad hands, missing fingers, pubic hair,extra digit, fewer digits, missing fingers, bad hands, text error, cropped, worst quality, low quality, standard quality, watermark, signature, username, jpeg artifacts, blurry, grayscale,', '', '', '', 7.5, 60, '', '', '', 'Description', '', '', '', -1, 1, 0, -1, 7.5, 60, '', '', '', '', 7.5, 60, '', '', '', 'Description', '', '', '', -1, 1, 0, -1, 7.5, 60, '', '', '', 'Loaded config.']
Concept 0 class dir is /notebooks/stable-diffusion-webui/models/dreambooth/yoisakikanadeCatTshirtMV/classifiers_0
Starting Dreambooth training...
 Cleanup completed. 
 Allocated: 0.0GB 
 Reserved: 0.0GB 

 Allocated 0.0/2.0GB 
 Reserved: 0.0/2.0GB 

Initializing dreambooth training...
Patching transformers to fix kwargs errors.
Replace CrossAttention.forward to use xformers
Checking concept: {'max_steps': -1, 'instance_data_dir': '/notebooks/training_data/dreambooth_training/2_MVスクショ_768', 'class_data_dir': '', 'file_prompt_contents': 'Description', 'instance_prompt': '[filewords]', 'class_prompt': '[filewords]', 'save_sample_prompt': '', 'save_sample_template': '', 'instance_token': 'yoisakikanade', 'class_token': 'girl', 'num_class_images': 20000, 'class_negative_prompt': '', 'class_guidance_scale': 7.5, 'class_infer_steps': 40, 'save_sample_negative_prompt': 'longbody, lowres, bad anatomy, bad hands, missing fingers, pubic hair,extra digit, fewer digits, missing fingers, bad hands, text error, cropped, worst quality, low quality, standard quality, watermark, signature, username, jpeg artifacts, blurry, grayscale,', 'n_save_sample': 1, 'sample_seed': -1, 'save_guidance_scale': 7.5, 'save_infer_steps': 40}
Concept requires 20000 images.
Class image dir is not set, defaulting to /notebooks/stable-diffusion-webui/models/dreambooth/yoisakikanadeCatTshirtMV/classifiers_0
Class dir /notebooks/stable-diffusion-webui/models/dreambooth/yoisakikanadeCatTshirtMV/classifiers_0 has 18 images.
  0%|                                      | 2/19982 [00:14<40:05:58,  7.23s/it]
 Training completed, reloading SD Model. 
 Allocated: 0.0GB 
 Reserved: 0.0GB 

Memory output: {}
 Restored system models. 
 Allocated: 2.0GB 
 Reserved: 2.0GB 

Returning result: Exception training model: [Errno 36] File name too long: '/notebooks/stable-diffusion-webui/models/dreambooth/yoisakikanadeCatTshirtMV/classifiers_0/20-headphones cat ear headphones 1girl headset choker fake animal ears blue eyes long hair open mouth solo blurry animal ears hair between eyes blurry background shirt black shirt collarbone outstretched arm cat ears earmuffs necklace short sleeves depth of field hair ornament silver hair android.jpg'

Environment

# cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

# python -c 'import torch; print(torch.version.cuda)'
11.6
# nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_Mar__8_18:18:20_PST_2022
Cuda compilation tools, release 11.6, V11.6.124
Build cuda_11.6.r11.6/compiler.31057947_0
# nvidia-smi
Sat Nov 19 13:49:28 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.73.05    Driver Version: 510.73.05    CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA A100-SXM...  Off  | 00000000:00:05.0 Off |                    0 |
| N/A   38C    P0    62W / 400W |   1139MiB / 81920MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+

This problem confirmed in A100-80G and A6000.

Screenshots/Config If the issue is specific to an error while training, please provide a screenshot of training parameters or the db_config.json file from /models/dreambooth/MODELNAME/db_config.json

{
    "adam_beta1": 0.9,
    "adam_beta2": 0.999,
    "adam_epsilon": 1e-08,
    "adam_weight_decay": 0.01,
    "attention": "xformers",
    "center_crop": false,
    "concepts_path": "",
    "gradient_accumulation_steps": 1,
    "gradient_checkpointing": true,
    "half_model": true,
    "hflip": true,
    "learning_rate": 1.72e-07,
    "lr_scheduler": "constant",
    "lr_warmup_steps": 0,
    "max_grad_norm": 1,
    "max_token_length": 75,
    "max_train_steps": 16000,
    "mixed_precision": "fp16",
    "model_dir": "/notebooks/stable-diffusion-webui/models/dreambooth/yoisakikanadeCatTshirtMV",
    "model_name": "yoisakikanadeCatTshirtMV",
    "not_cache_latents": true,
    "num_train_epochs": 1,
    "pad_tokens": true,
    "pretrained_model_name_or_path": "/notebooks/stable-diffusion-webui/models/dreambooth/yoisakikanadeCatTshirtMV/working",
    "pretrained_vae_name_or_path": "",
    "prior_loss_weight": 1,
    "resolution": 768,
    "revision": 0,
    "sample_batch_size": 1,
    "save_class_txt": false,
    "save_embedding_every": 3000,
    "save_preview_every": 3000,
    "scale_lr": false,
    "src": "/notebooks/stable-diffusion-webui/models/Stable-diffusion/Anything-V3.0-pruned-fp16.ckpt",
    "train_batch_size": 1,
    "train_text_encoder": true,
    "use_8bit_adam": true,
    "use_concepts": false,
    "use_cpu": false,
    "use_ema": false,
    "scheduler": "ddim",
    "v2": false,
    "has_ema": "True",
    "concepts_list": [
        {
            "max_steps": -1,
            "instance_data_dir": "/notebooks/training_data/dreambooth_training/2_MV\u30b9\u30af\u30b7\u30e7_768",
            "class_data_dir": "",
            "file_prompt_contents": "Description",
            "instance_prompt": "[filewords]",
            "class_prompt": "[filewords]",
            "save_sample_prompt": "",
            "save_sample_template": "",
            "instance_token": "yoisakikanade",
            "class_token": "girl",
            "num_class_images": 20000,
            "class_negative_prompt": "",
            "class_guidance_scale": 7.5,
            "class_infer_steps": 40,
            "save_sample_negative_prompt": "longbody, lowres, bad anatomy, bad hands, missing fingers, pubic hair,extra digit, fewer digits, missing fingers, bad hands, text error, cropped, worst quality, low quality, standard quality, watermark, signature, username, jpeg artifacts, blurry, grayscale,",
            "n_save_sample": 1,
            "sample_seed": -1,
            "save_guidance_scale": 7.5,
            "save_infer_steps": 40
        }
    ],
    "lifetime_revision": 0
}

[art-phony]

I took few samples OK/NG filenames. It seems Linux's limitation of "filenames must be 255 bytes or less".

OK filenames (nums of chars)

25-headphones cat ear headphones headset earmuffs 1girl hair ornament long hair solo choker shirt hair between eyes short sleeves robot ears headphones around neck hairclip black shirt closed mouth purple eyes.jpg (213)
26-1girl solo hair ornament long hair headphones bangs cat ear headphones looking at viewer simple background hair between eyes eyebrows visible through hair fake animal ears virtual youtuber hairclip animal ears green hair.jpg (227)
27-1girl long hair solo headphones headset choker hair ornament short sleeves black shirt hairclip blue eyes cat ear headphones shirt upper body bangs looking at viewer.jpg (172)
28-headphones 1girl cat ear headphones long hair hair ornament solo choker closed eyes headset shirt open mouth bangs animal ears facing viewer hairclip short sleeves.jpg (170)
29-1girl solo hair ornament cross x hair ornament collarbone green background looking at viewer bangs choker long hair hairclip tattoo portrait hat open mouth cross hair ornament.jpg (182)
30-headphones 1girl cat ear headphones choker long hair headset solo hair ornament shirt blue eyes earmuffs open mouth black shirt silver hair star symbol short sleeves animal ears hair between eyes black choker cat ears facepaint fake animal ears.jpg (251)
31-plaid long hair plaid skirt headphones 1girl solo very long hair bag skirt building skyscraper.jpg (101)

NG filenames

28-choker headphones blue eyes solo 1girl blurry blurry background headset long hair black choker depth of field heart choker collarbone bangs blue hair earmuffs cardigan long sleeves cat ear headphones open clothes jacket looking at viewer open mouth facial mark tattoo.jpg (274)
20-headphones cat ear headphones 1girl headset choker fake animal ears blue eyes long hair open mouth solo blurry animal ears hair between eyes blurry background shirt black shirt collarbone outstretched arm cat ears earmuffs necklace short sleeves depth of field hair ornament silver hair android.jpg (301)
28-headphones 1girl hair ornament long hair solo cat ear headphones black shirt blue hair open mouth headset curtains blue eyes cross hair between eyes shirt upper body necklace choker vertical stripes hairclip striped short sleeves looking at viewer window.jpg (261)
28-headphones 1girl cat ear headphones headset solo closed eyes long hair hair ornament open mouth cat ears gloves shirt animal ears simple background short sleeves white background choker cross fake animal ears collarbone silver hair necklace bangs white hair earmuffs black shirt.jpg (285)
32-headphones cat ear headphones 1girl headset long hair red background hair ornament red sky solo hair between eyes headphones around neck choker claw pose smile hairclip silver hair very long hair looking at viewer earmuffs hands up cat ears upper body shirt.jpg (265)

[art-phony]

I found that setting the variable args.save_class_txt on L500 of train_dreambooth.py to True will solve this problem. However, I am not sure how to set the args.save_class_txt variable to True.

https://github.com/d8ahazard/sd_dreambooth_extension/blob/88cc50ace470580842b519115545b9a08ebe115e/dreambooth/train_dreambooth.py#L500

[art-phony]

Note: using concepts_list.json won't solve this problem.

[leppie]

This was fixed recently. Your version is about 20 commits behind.

[art-phony]

Thank you for your comment. As you mentioned, somehow I forgot to update WebUI itself. But after updated WebUI to latest version (44c46f), still same error occurs. I believe this error is related to sd_dreambooth_extension script itself, not WebUI related.

[d8ahazard]

FWIW, I'm removing the args.save_class bit entirely. I thought it'd be neat to just have the class images with the names in the filename, but there's a very good reason why we just write them to text, and this is why.

This will be fixed entirely once I merge the forthcoming v21 branch.

[d8ahazard]

This is now resolved.