[X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui
Is EasyPhoto the latest version?
[X] I have updated EasyPhoto to the latest version and the bug still exists.
What happened?
Loading weights [59ffe2243a] from /home/gaol/codes/temp/stable-diffusion-webui/models/Stable-diffusion/Chilloutmix-Ni-pruned-fp16-fix.safetensors
2024-03-16 15:22:02,890 - modelscope - INFO - PyTorch version 2.0.1+cu118 Found.
2024-03-16 15:22:02,891 - modelscope - INFO - TensorFlow version 2.13.0 Found.
2024-03-16 15:22:02,891 - modelscope - INFO - Loading ast index from /home/gaol/.cache/modelscope/ast_indexer
2024-03-16 15:22:02,947 - modelscope - INFO - Loading done! Current index file version is 1.9.3, with md5 985d60ab3829178ada728d5649a2ffda and a total number of 943 components indexed
2024-03-16 15:22:03,980 - modelscope - INFO - Use user-specified model revision: v1.0.3
2024-03-16 15:22:04,303 - modelscope - WARNING - ('PIPELINES', 'face_recognition', 'face_recognition') not found in ast index file
2024-03-16 15:22:04,303 - modelscope - INFO - initiate model from /home/gaol/.cache/modelscope/hub/bubbliiiing/cv_retinafce_recognition
2024-03-16 15:22:04,303 - modelscope - INFO - initiate model from location /home/gaol/.cache/modelscope/hub/bubbliiiing/cv_retinafce_recognition.
2024-03-16 15:22:04,304 - modelscope - INFO - initialize model from /home/gaol/.cache/modelscope/hub/bubbliiiing/cv_retinafce_recognition
2024-03-16 15:22:04,306 - modelscope - WARNING - ('MODELS', 'face_recognition', 'face_recognition') not found in ast index file
2024-03-16 15:22:04,850 - modelscope - INFO - Model revision not specified, use revision: v2.0.2
2024-03-16 15:22:06,525 - modelscope - INFO - initiate model from /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface
2024-03-16 15:22:06,525 - modelscope - INFO - initiate model from location /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface.
2024-03-16 15:22:06,526 - modelscope - WARNING - No preprocessor field found in cfg.
2024-03-16 15:22:06,526 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2024-03-16 15:22:06,526 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface'}. trying to build by task and model information.
2024-03-16 15:22:06,526 - modelscope - WARNING - Find task: face-detection, model type: None. Insufficient information to build preprocessor, skip building preprocessor
2024-03-16 15:22:06,526 - modelscope - INFO - loading model from /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface/pytorch_model.pt
/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
warnings.warn(
/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing weights=None.
warnings.warn(msg)
2024-03-16 15:22:06,914 - modelscope - INFO - load model done
2024-03-16 15:22:07,571 - modelscope - INFO - load facefusion models done
2024-03-16 15:22:07,571 - modelscope - INFO - init done
2024-03-16 15:22:07,572 - modelscope - WARNING - No preprocessor field found in cfg.
2024-03-16 15:22:07,572 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2024-03-16 15:22:07,572 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/home/gaol/.cache/modelscope/hub/bubbliiiing/cv_retinafce_recognition'}. trying to build by task and model information.
2024-03-16 15:22:07,572 - modelscope - WARNING - No preprocessor key ('face_recognition', 'face_recognition') found in PREPROCESSOR_MAP, skip building preprocessor.
2024-03-16 15:22:07,572 - modelscope - INFO - image face recognition model init done
2024-03-16 15:22:08,121 - modelscope - INFO - Use user-specified model revision: v2.0.2
2024-03-16 15:22:09,744 - modelscope - INFO - initiate model from /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface
2024-03-16 15:22:09,744 - modelscope - INFO - initiate model from location /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface.
2024-03-16 15:22:09,746 - modelscope - WARNING - No preprocessor field found in cfg.
2024-03-16 15:22:09,746 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2024-03-16 15:22:09,746 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface'}. trying to build by task and model information.
2024-03-16 15:22:09,746 - modelscope - WARNING - Find task: face-detection, model type: None. Insufficient information to build preprocessor, skip building preprocessor
2024-03-16 15:22:09,746 - modelscope - INFO - loading model from /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface/pytorch_model.pt
2024-03-16 15:22:10,014 - modelscope - INFO - load model done
2024-03-16 15:22:10,645 - modelscope - INFO - Use user-specified model revision: v1.0.0
2024-03-16 15:22:10,832 - modelscope - INFO - initiate model from /home/gaol/.cache/modelscope/hub/damo/cv_u2net_salient-detection
2024-03-16 15:22:10,832 - modelscope - INFO - initiate model from location /home/gaol/.cache/modelscope/hub/damo/cv_u2net_salient-detection.
2024-03-16 15:22:10,834 - modelscope - INFO - initialize model from /home/gaol/.cache/modelscope/hub/damo/cv_u2net_salient-detection
2024-03-16 15:22:11,091 - modelscope - WARNING - No preprocessor field found in cfg.
2024-03-16 15:22:11,091 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2024-03-16 15:22:11,091 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/home/gaol/.cache/modelscope/hub/damo/cv_u2net_salient-detection'}. trying to build by task and model information.
2024-03-16 15:22:11,091 - modelscope - WARNING - No preprocessor key ('detection', 'semantic-segmentation') found in PREPROCESSOR_MAP, skip building preprocessor.
2024-03-16 15:22:11,691 - modelscope - INFO - Use user-specified model revision: v1.0.2
2024-03-16 15:22:11,943 - modelscope - WARNING - ('PIPELINES', 'skin-retouching-torch', 'skin-retouching-torch') not found in ast index file
2024-03-16 15:22:11,943 - modelscope - INFO - initiate model from /home/gaol/.cache/modelscope/hub/damo/cv_unet_skin_retouching_torch
2024-03-16 15:22:11,943 - modelscope - INFO - initiate model from location /home/gaol/.cache/modelscope/hub/damo/cv_unet_skin_retouching_torch.
2024-03-16 15:22:11,944 - modelscope - WARNING - No preprocessor field found in cfg.
2024-03-16 15:22:11,944 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2024-03-16 15:22:11,944 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/home/gaol/.cache/modelscope/hub/damo/cv_unet_skin_retouching_torch'}. trying to build by task and model information.
2024-03-16 15:22:11,944 - modelscope - WARNING - Find task: skin-retouching-torch, model type: None. Insufficient information to build preprocessor, skip building preprocessor
2024-03-16 15:22:12,637 - modelscope - INFO - Model revision not specified, use revision: v2.0.2
2024-03-16 15:22:14,524 - modelscope - INFO - initiate model from /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface
2024-03-16 15:22:14,524 - modelscope - INFO - initiate model from location /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface.
2024-03-16 15:22:14,526 - modelscope - WARNING - No preprocessor field found in cfg.
2024-03-16 15:22:14,526 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2024-03-16 15:22:14,526 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface'}. trying to build by task and model information.
2024-03-16 15:22:14,526 - modelscope - WARNING - Find task: face-detection, model type: None. Insufficient information to build preprocessor, skip building preprocessor
2024-03-16 15:22:14,526 - modelscope - INFO - loading model from /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface/pytorch_model.pt
2024-03-16 15:22:14,797 - modelscope - INFO - load model done
2024-03-16 15:22:15.412807243 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'Sub1664:0'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.412822062 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'Shape1662:0'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.412831040 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_41660'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525554751 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2649'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525574428 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2657'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525577167 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2644'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525579922 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2594'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525583236 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2596'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525586891 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2653'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525589904 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2624'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525592868 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2652'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525596530 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2645'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525599328 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2643'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525602181 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2648'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525604868 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2647'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525607502 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2641'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525611251 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2633'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525614155 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2632'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525617785 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2614'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525620526 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2613'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525623573 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2658'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525626884 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2606'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525630091 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt__2598'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:16,225 - modelscope - INFO - Use user-specified model revision: v1.0.0
2024-03-16 15:22:16,454 - modelscope - INFO - initiate model from /home/gaol/.cache/modelscope/hub/damo/cv_gpen_image-portrait-enhancement
2024-03-16 15:22:16,455 - modelscope - INFO - initiate model from location /home/gaol/.cache/modelscope/hub/damo/cv_gpen_image-portrait-enhancement.
2024-03-16 15:22:16,456 - modelscope - INFO - initialize model from /home/gaol/.cache/modelscope/hub/damo/cv_gpen_image-portrait-enhancement
Loading ResNet ArcFace
2024-03-16 15:22:18,087 - modelscope - INFO - load face enhancer model done
2024-03-16 15:22:18,337 - modelscope - INFO - load face detector model done
2024-03-16 15:22:18,583 - modelscope - INFO - load sr model done
2024-03-16 15:22:19,307 - modelscope - INFO - load fqa model done
0%| | 0/15 [00:00<?, ?it/s]2024-03-16 15:22:19,761 - modelscope - WARNING - task skin-retouching-torch input definition is missing
2024-03-16 15:22:21,407 - modelscope - WARNING - task skin-retouching-torch output keys are missing
2024-03-16 15:22:21,411 - modelscope - WARNING - task face_recognition input definition is missing
2024-03-16 15:22:21,555 - modelscope - INFO - model inference done
2024-03-16 15:22:21,555 - modelscope - WARNING - task face_recognition output keys are missing
7%|?????????????? | 1/15 [00:02<00:31, 2.22s/it]2024-03-16 15:22:22,273 - modelscope - INFO - model inference done
13%|???????????????????????????? | 2/15 [00:02<00:17, 1.34s/it]2024-03-16 15:22:22,961 - modelscope - INFO - model inference done
20%|????????????????????????????????????????? | 3/15 [00:03<00:12, 1.04s/it]2024-03-16 15:22:23,690 - modelscope - INFO - model inference done
27%|??????????????????????????????????????????????????????? | 4/15 [00:04<00:10, 1.09it/s]2024-03-16 15:22:24,654 - modelscope - INFO - model inference done
33%|????????????????????????????????????????????????????????????????????? | 5/15 [00:05<00:09, 1.07it/s]2024-03-16 15:22:25,290 - modelscope - INFO - model inference done
40%|?????????????????????????????????????????????????????????????????????????????????? | 6/15 [00:05<00:07, 1.20it/s]2024-03-16 15:22:26,010 - modelscope - INFO - model inference done
47%|???????????????????????????????????????????????????????????????????????????????????????????????? | 7/15 [00:06<00:06, 1.26it/s]2024-03-16 15:22:27,181 - modelscope - INFO - model inference done
53%|?????????????????????????????????????????????????????????????????????????????????????????????????????????????? | 8/15 [00:07<00:06, 1.09it/s]2024-03-16 15:22:28,236 - modelscope - INFO - model inference done
60%|??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? | 9/15 [00:08<00:05, 1.04it/s]2024-03-16 15:22:29,217 - modelscope - INFO - model inference done
67%|???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? | 10/15 [00:09<00:04, 1.04it/s]2024-03-16 15:22:30,171 - modelscope - INFO - model inference done
73%|?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? | 11/15 [00:10<00:03, 1.04it/s]2024-03-16 15:22:31,393 - modelscope - INFO - model inference done
80%|???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? | 12/15 [00:12<00:03, 1.04s/it]2024-03-16 15:22:32,414 - modelscope - INFO - model inference done
87%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? | 13/15 [00:13<00:02, 1.03s/it]2024-03-16 15:22:33,103 - modelscope - INFO - model inference done
93%|??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? | 14/15 [00:13<00:00, 1.07it/s]2024-03-16 15:22:33,818 - modelscope - INFO - model inference done
100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 15/15 [00:14<00:00, 1.04it/s]
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/0.jpg total scores: 0.5047546609308565 face angles 0.9206686066500163
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/12.jpg total scores: 0.49227943258374113 face angles 0.9208677161732743
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/6.jpg total scores: 0.4752780163014266 face angles 0.9655220357517296
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/2.jpg total scores: 0.4714630612945019 face angles 0.9158910610739464
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/13.jpg total scores: 0.46400164259168414 face angles 0.9542378038928844
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/4.jpg total scores: 0.43687105673798315 face angles 0.8650432225418873
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/7.jpg total scores: 0.4353354856725917 face angles 0.9829790242269123
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/11.jpg total scores: 0.4166387907586268 face angles 0.7875557185843958
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/8.jpg total scores: 0.4151525819786572 face angles 0.9470527983614994
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/5.jpg total scores: 0.3955914799303482 face angles 0.9771394596698413
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/14.jpg total scores: 0.38413634652262346 face angles 0.9993536825291083
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/10.jpg total scores: 0.34892923676821336 face angles 0.7398419459302334
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/9.jpg total scores: 0.3327070567244012 face angles 0.8863536608311919
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/1.jpg total scores: 0.3145806877875612 face angles 0.6169503507383685
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/3.jpg total scores: 0.3134642199143675 face angles 0.8505665005567431
jpg: 0.jpg face_id_scores 0.5047546609308565
jpg: 12.jpg face_id_scores 0.49227943258374113
jpg: 11.jpg face_id_scores 0.4166387907586268
jpg: 2.jpg face_id_scores 0.4714630612945019
jpg: 1.jpg face_id_scores 0.3145806877875612
jpg: 4.jpg face_id_scores 0.43687105673798315
jpg: 6.jpg face_id_scores 0.4752780163014266
jpg: 13.jpg face_id_scores 0.46400164259168414
jpg: 10.jpg face_id_scores 0.34892923676821336
jpg: 7.jpg face_id_scores 0.4353354856725917
jpg: 8.jpg face_id_scores 0.4151525819786572
jpg: 5.jpg face_id_scores 0.3955914799303482
jpg: 14.jpg face_id_scores 0.38413634652262346
jpg: 9.jpg face_id_scores 0.3327070567244012
jpg: 3.jpg face_id_scores 0.3134642199143675
15it [00:13, 1.12it/s]
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/0.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/1.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/2.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/3.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/4.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/5.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/6.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/7.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/8.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/9.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/10.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/11.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/12.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/13.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/14.jpg
2024-03-16 15:22:47,750 - EasyPhoto - train_file_path : /home/gaol/codes/temp/stable-diffusion-webui/extensions/sd-webui-EasyPhoto/scripts/train_kohya/train_lora.py
2024-03-16 15:22:47,750 - EasyPhoto - cache_log_file_path: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-tmp/train_kohya_log.txt
2024-03-16 15:22:49.493300: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-16 15:22:52.402091: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-16 15:22:53,191 - modelscope - INFO - PyTorch version 2.0.1+cu118 Found.
2024-03-16 15:22:53,192 - modelscope - INFO - TensorFlow version 2.13.0 Found.
2024-03-16 15:22:53,192 - modelscope - INFO - Loading ast index from /home/gaol/.cache/modelscope/ast_indexer
2024-03-16 15:22:53,209 - modelscope - INFO - Loading done! Current index file version is 1.9.3, with md5 985d60ab3829178ada728d5649a2ffda and a total number of 943 components indexed
03/16/2024 15:22:53 - INFO - main - Distributed environment: MULTI_GPU Backend: nccl
Num processes: 1
Process index: 0
Local process index: 0
Device: cuda:0
Mixed precision type: fp16
{'sample_max_value', 'variance_type', 'thresholding', 'clip_sample_range', 'dynamic_thresholding_ratio', 'timestep_spacing', 'prediction_type'} was not found in config. Values will be initialized to default values.
UNet2DConditionModel: 64, 8, 768, False, False
loading u-net:
loading vae:
loading text encoder:
create LoRA network. base dim (rank): 128, alpha: 64
neuron dropout: p=None, rank dropout: p=None, module dropout: p=None
create LoRA for Text Encoder:
create LoRA for Text Encoder: 72 modules.
create LoRA for U-Net: 192 modules.
enable LoRA for text encoder
enable LoRA for U-Net
Resolving data files: 100%|??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 31/31 [00:00<00:00, 436320.21it/s]
Downloading and preparing dataset imagefolder/default to /home/gaol/.cache/huggingface/datasets/imagefolder/default-cd3ed4aa3d015dcd/0.0.0/37fbb85cc714a338bea574ac6c7d0b5be5aff46c1862c1989b20e0771199e93f...
Downloading data files: 100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 16/16 [00:00<00:00, 106017.16it/s]
Downloading data files: 100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 15/15 [00:00<00:00, 116293.09it/s]
Extracting data files: 100%|???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 15/15 [00:00<00:00, 6948.05it/s]
Dataset imagefolder downloaded and prepared to /home/gaol/.cache/huggingface/datasets/imagefolder/default-cd3ed4aa3d015dcd/0.0.0/37fbb85cc714a338bea574ac6c7d0b5be5aff46c1862c1989b20e0771199e93f. Subsequent calls will reuse this data.
100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 1/1 [00:00<00:00, 1410.32it/s]
03/16/2024 15:23:01 - INFO - main - Running training
03/16/2024 15:23:01 - INFO - main - Num examples = 15
03/16/2024 15:23:01 - INFO - main - Num Epochs = 3000
03/16/2024 15:23:01 - INFO - main - Instantaneous batch size per device = 8
03/16/2024 15:23:01 - INFO - main - Total train batch size (w. parallel, distributed & accumulation) = 64
03/16/2024 15:23:01 - INFO - main - Gradient Accumulation steps = 8
03/16/2024 15:23:01 - INFO - main - Total optimization steps = 3000
Steps: 0%| | 0/3000 [00:00<?, ?it/s]2024-03-16 15:23:02,628 - modelscope - INFO - Model revision not specified, use revision: v2.0.2
2024-03-16 15:23:04,325 - modelscope - INFO - initiate model from /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface
2024-03-16 15:23:04,325 - modelscope - INFO - initiate model from location /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface.
2024-03-16 15:23:04,326 - modelscope - WARNING - No preprocessor field found in cfg.
2024-03-16 15:23:04,326 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2024-03-16 15:23:04,326 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface'}. trying to build by task and model information.
2024-03-16 15:23:04,326 - modelscope - WARNING - Find task: face-detection, model type: None. Insufficient information to build preprocessor, skip building preprocessor
2024-03-16 15:23:04,326 - modelscope - INFO - loading model from /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface/pytorch_model.pt
/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
warnings.warn(
/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing weights=None.
warnings.warn(msg)
2024-03-16 15:23:04,575 - modelscope - INFO - load model done
/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/conv.py:459: UserWarning: Applied workaround for CuDNN issue, install nvrtc.so (Triggered internally at ../aten/src/ATen/native/cudnn/Conv_v8.cpp:80.)
return F.conv2d(input, weight, bias, self.stride,
[2024-03-16 15:23:09,607] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2024-03-16 15:23:09,809] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing call
[2024-03-16 15:23:09,812] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2024-03-16 15:23:11,013] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2024-03-16 15:23:11,029] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function debug_wrapper
Traceback (most recent call last):
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 670, in call_user_compiler
compiled_fn = compiler_fn(gm, self.fake_example_inputs())
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 1055, in debug_wrapper
compiled_gm = compiler_fn(gm, example_inputs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/init.py", line 1390, in call
return compilefx(model, inputs_, config_patches=self.config)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 455, in compile_fx
return aot_autograd(
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 48, in compiler_fn
cg = aot_module_simplified(gm, example_inputs, kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2822, in aot_module_simplified
compiled_fn = create_aot_dispatcher_function(
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
r = func(*args, kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2515, in create_aot_dispatcher_function
compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1676, in aot_wrapper_dedupe
fw_metadata, _out = run_functionalized_fw_and_collect_metadata(
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 607, in inner
flat_f_outs = f(flat_f_args)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2793, in functional_call
out = Interpreter(mod).run(args[params_len:], kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/fx/interpreter.py", line 136, in run
self.env[node] = self.run_node(node)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/fx/interpreter.py", line 177, in run_node
return getattr(self, n.op)(n.target, args, kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/fx/interpreter.py", line 294, in call_module
return submod(*args, kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/extensions/sd-webui-EasyPhoto/scripts/train_kohya/utils/lora_utils.py", line 140, in forward
lx = self.lora_down(x)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, *kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_inductor/overrides.py", line 38, in __torch_function__
return func(args, kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/utils/_stats.py", line 20, in wrapper
return fn(*args, *kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 987, in __torch_dispatch__
return self.dispatch(func, types, args, kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1066, in dispatch
args, kwargs = self.validate_and_convert_non_fake_tensors(
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1220, in validate_and_convert_non_fake_tensors
return tree_map_only(
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/utils/_pytree.py", line 266, in tree_map_only
return tree_map(map_only(ty)(fn), pytree)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/utils/_pytree.py", line 196, in tree_map
return tree_unflatten([fn(i) for i in flat_args], spec)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/utils/_pytree.py", line 196, in
return tree_unflatten([fn(i) for i in flat_args], spec)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/utils/_pytree.py", line 247, in inner
return f(x)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1212, in validate
raise Exception(
Exception: Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'. Found in aten._to_copy.default((Parameter containing:
tensor([[-0.0176, -0.0284, 0.0112, ..., 0.0230, 0.0277, 0.0246],
[ 0.0259, 0.0047, 0.0272, ..., 0.0275, 0.0020, 0.0336],
[-0.0033, -0.0350, 0.0030, ..., -0.0004, 0.0218, 0.0237],
...,
[ 0.0084, -0.0230, -0.0020, ..., 0.0119, 0.0042, 0.0045],
[-0.0068, -0.0207, -0.0300, ..., -0.0241, 0.0173, 0.0291],
[-0.0163, -0.0165, 0.0014, ..., -0.0115, 0.0128, 0.0221]],
device='cuda:0', requires_grad=True),), **{'dtype': torch.float16})
While executing %self_text_model_encoder_layers_0_self_attn_q_proj : [#users=1] = call_module[target=self_text_model_encoder_layers_0_self_attn_q_proj](args = (%self_text_model_encoder_layers_0_layer_norm1,), kwargs = {})
Original traceback:
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 272, in forward
query_states = self.q_proj(hidden_states) * self.scale
| File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 383, in forward
hidden_states, attn_weights = self.self_attn(
| File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 654, in forward
layer_outputs = encoder_layer(
| File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 740, in forward
encoder_outputs = self.encoder(
| File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 822, in forward
return self.text_model(
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/gaol/codes/temp/stable-diffusion-webui/extensions/sd-webui-EasyPhoto/scripts/train_kohya/train_lora.py", line 1397, in
main()
File "/home/gaol/codes/temp/stable-diffusion-webui/extensions/sd-webui-EasyPhoto/scripts/train_kohya/utils/gpu_info.py", line 190, in wrapper
result = func(*args, kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/extensions/sd-webui-EasyPhoto/scripts/train_kohya/train_lora.py", line 1132, in main
encoder_hidden_states = text_encoder(batch["input_ids"])[0]
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, *kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 82, in forward
return self.dynamo_ctx(self._orig_mod.forward)(args, kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 209, in _fn
return fn(*args, kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/utils/operations.py", line 581, in forward
return model_forward(*args, *kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/utils/operations.py", line 569, in call
return convert_to_fp32(self.model_forward(args, kwargs))
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
return func(*args, kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 337, in catch_errors
return callback(frame, cache_size, hooks)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 404, in _convert_frame
result = inner_convert(frame, cache_size, hooks)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 104, in _fn
return fn(*args, *kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 262, in _convert_frame_assert
return _compile(
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
r = func(args, kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 324, in _compile
out_code = transform_code_object(code, transform)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 445, in transform_code_object
transformations(instructions, code_options)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 311, in transform
tracer.run()
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1726, in run
super().run()
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 576, in run
and self.step()
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 540, in step
getattr(self, inst.opname)(inst)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1792, in RETURN_VALUE
self.output.compile_subgraph(
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 541, in compile_subgraph
self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 588, in compile_and_call_fx_graph
compiled_fn = self.call_user_compiler(gm)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
r = func(*args, *kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 675, in call_user_compiler
raise BackendCompilerFailed(self.compiler_fn, e) from e
torch._dynamo.exc.BackendCompilerFailed: debug_wrapper raised Exception: Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'. Found in aten._to_copy.default((Parameter containing:
tensor([[-0.0176, -0.0284, 0.0112, ..., 0.0230, 0.0277, 0.0246],
[ 0.0259, 0.0047, 0.0272, ..., 0.0275, 0.0020, 0.0336],
[-0.0033, -0.0350, 0.0030, ..., -0.0004, 0.0218, 0.0237],
...,
[ 0.0084, -0.0230, -0.0020, ..., 0.0119, 0.0042, 0.0045],
[-0.0068, -0.0207, -0.0300, ..., -0.0241, 0.0173, 0.0291],
[-0.0163, -0.0165, 0.0014, ..., -0.0115, 0.0128, 0.0221]],
device='cuda:0', requires_grad=True),), **{'dtype': torch.float16})
While executing %self_text_model_encoder_layers_0_self_attn_q_proj : [#users=1] = call_module[target=self_text_model_encoder_layers_0_self_attn_q_proj](args = (%self_text_model_encoder_layers_0_layer_norm1,), kwargs = {})
Original traceback:
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 272, in forward
query_states = self.q_proj(hidden_states) * self.scale
| File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 383, in forward
hidden_states, attn_weights = self.self_attn(
| File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 654, in forward
layer_outputs = encoder_layer(
| File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 740, in forward
encoder_outputs = self.encoder(
| File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 822, in forward
return self.text_model(
Set torch._dynamo.config.verbose=True for more information
You can suppress this exception and fall back to eager by setting:
torch._dynamo.config.suppress_errors = True
Steps: 0%| | 0/3000 [00:11<?, ?it/s]
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 2387) of binary: /home/gaol/codes/temp/stable-diffusion-webui/venv/bin/python3
Traceback (most recent call last):
File "/home/gaol/miniforge3/envs/stable/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/gaol/miniforge3/envs/stable/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 989, in
main()
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 985, in main
launch_command(args)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 970, in launch_command
multi_gpu_launcher(args)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 646, in multi_gpu_launcher
distrib_run.run(args)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
Is there an existing issue for this?
Is EasyPhoto the latest version?
What happened?
Loading weights [59ffe2243a] from /home/gaol/codes/temp/stable-diffusion-webui/models/Stable-diffusion/Chilloutmix-Ni-pruned-fp16-fix.safetensors 2024-03-16 15:22:02,890 - modelscope - INFO - PyTorch version 2.0.1+cu118 Found. 2024-03-16 15:22:02,891 - modelscope - INFO - TensorFlow version 2.13.0 Found. 2024-03-16 15:22:02,891 - modelscope - INFO - Loading ast index from /home/gaol/.cache/modelscope/ast_indexer 2024-03-16 15:22:02,947 - modelscope - INFO - Loading done! Current index file version is 1.9.3, with md5 985d60ab3829178ada728d5649a2ffda and a total number of 943 components indexed 2024-03-16 15:22:03,980 - modelscope - INFO - Use user-specified model revision: v1.0.3 2024-03-16 15:22:04,303 - modelscope - WARNING - ('PIPELINES', 'face_recognition', 'face_recognition') not found in ast index file 2024-03-16 15:22:04,303 - modelscope - INFO - initiate model from /home/gaol/.cache/modelscope/hub/bubbliiiing/cv_retinafce_recognition 2024-03-16 15:22:04,303 - modelscope - INFO - initiate model from location /home/gaol/.cache/modelscope/hub/bubbliiiing/cv_retinafce_recognition. 2024-03-16 15:22:04,304 - modelscope - INFO - initialize model from /home/gaol/.cache/modelscope/hub/bubbliiiing/cv_retinafce_recognition 2024-03-16 15:22:04,306 - modelscope - WARNING - ('MODELS', 'face_recognition', 'face_recognition') not found in ast index file 2024-03-16 15:22:04,850 - modelscope - INFO - Model revision not specified, use revision: v2.0.2 2024-03-16 15:22:06,525 - modelscope - INFO - initiate model from /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface 2024-03-16 15:22:06,525 - modelscope - INFO - initiate model from location /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface. 2024-03-16 15:22:06,526 - modelscope - WARNING - No preprocessor field found in cfg. 2024-03-16 15:22:06,526 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file. 2024-03-16 15:22:06,526 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface'}. trying to build by task and model information. 2024-03-16 15:22:06,526 - modelscope - WARNING - Find task: face-detection, model type: None. Insufficient information to build preprocessor, skip building preprocessor 2024-03-16 15:22:06,526 - modelscope - INFO - loading model from /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface/pytorch_model.pt /home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead. warnings.warn( /home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or
Nonefor 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passingweights=None. warnings.warn(msg) 2024-03-16 15:22:06,914 - modelscope - INFO - load model done 2024-03-16 15:22:07,571 - modelscope - INFO - load facefusion models done 2024-03-16 15:22:07,571 - modelscope - INFO - init done 2024-03-16 15:22:07,572 - modelscope - WARNING - No preprocessor field found in cfg. 2024-03-16 15:22:07,572 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file. 2024-03-16 15:22:07,572 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/home/gaol/.cache/modelscope/hub/bubbliiiing/cv_retinafce_recognition'}. trying to build by task and model information. 2024-03-16 15:22:07,572 - modelscope - WARNING - No preprocessor key ('face_recognition', 'face_recognition') found in PREPROCESSOR_MAP, skip building preprocessor. 2024-03-16 15:22:07,572 - modelscope - INFO - image face recognition model init done 2024-03-16 15:22:08,121 - modelscope - INFO - Use user-specified model revision: v2.0.2 2024-03-16 15:22:09,744 - modelscope - INFO - initiate model from /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface 2024-03-16 15:22:09,744 - modelscope - INFO - initiate model from location /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface. 2024-03-16 15:22:09,746 - modelscope - WARNING - No preprocessor field found in cfg. 2024-03-16 15:22:09,746 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file. 2024-03-16 15:22:09,746 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface'}. trying to build by task and model information. 2024-03-16 15:22:09,746 - modelscope - WARNING - Find task: face-detection, model type: None. Insufficient information to build preprocessor, skip building preprocessor 2024-03-16 15:22:09,746 - modelscope - INFO - loading model from /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface/pytorch_model.pt 2024-03-16 15:22:10,014 - modelscope - INFO - load model done 2024-03-16 15:22:10,645 - modelscope - INFO - Use user-specified model revision: v1.0.0 2024-03-16 15:22:10,832 - modelscope - INFO - initiate model from /home/gaol/.cache/modelscope/hub/damo/cv_u2net_salient-detection 2024-03-16 15:22:10,832 - modelscope - INFO - initiate model from location /home/gaol/.cache/modelscope/hub/damo/cv_u2net_salient-detection. 2024-03-16 15:22:10,834 - modelscope - INFO - initialize model from /home/gaol/.cache/modelscope/hub/damo/cv_u2net_salient-detection 2024-03-16 15:22:11,091 - modelscope - WARNING - No preprocessor field found in cfg. 2024-03-16 15:22:11,091 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file. 2024-03-16 15:22:11,091 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/home/gaol/.cache/modelscope/hub/damo/cv_u2net_salient-detection'}. trying to build by task and model information. 2024-03-16 15:22:11,091 - modelscope - WARNING - No preprocessor key ('detection', 'semantic-segmentation') found in PREPROCESSOR_MAP, skip building preprocessor. 2024-03-16 15:22:11,691 - modelscope - INFO - Use user-specified model revision: v1.0.2 2024-03-16 15:22:11,943 - modelscope - WARNING - ('PIPELINES', 'skin-retouching-torch', 'skin-retouching-torch') not found in ast index file 2024-03-16 15:22:11,943 - modelscope - INFO - initiate model from /home/gaol/.cache/modelscope/hub/damo/cv_unet_skin_retouching_torch 2024-03-16 15:22:11,943 - modelscope - INFO - initiate model from location /home/gaol/.cache/modelscope/hub/damo/cv_unet_skin_retouching_torch. 2024-03-16 15:22:11,944 - modelscope - WARNING - No preprocessor field found in cfg. 2024-03-16 15:22:11,944 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file. 2024-03-16 15:22:11,944 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/home/gaol/.cache/modelscope/hub/damo/cv_unet_skin_retouching_torch'}. trying to build by task and model information. 2024-03-16 15:22:11,944 - modelscope - WARNING - Find task: skin-retouching-torch, model type: None. Insufficient information to build preprocessor, skip building preprocessor 2024-03-16 15:22:12,637 - modelscope - INFO - Model revision not specified, use revision: v2.0.2 2024-03-16 15:22:14,524 - modelscope - INFO - initiate model from /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface 2024-03-16 15:22:14,524 - modelscope - INFO - initiate model from location /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface. 2024-03-16 15:22:14,526 - modelscope - WARNING - No preprocessor field found in cfg. 2024-03-16 15:22:14,526 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file. 2024-03-16 15:22:14,526 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface'}. trying to build by task and model information. 2024-03-16 15:22:14,526 - modelscope - WARNING - Find task: face-detection, model type: None. Insufficient information to build preprocessor, skip building preprocessor 2024-03-16 15:22:14,526 - modelscope - INFO - loading model from /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface/pytorch_model.pt 2024-03-16 15:22:14,797 - modelscope - INFO - load model done 2024-03-16 15:22:15.412807243 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'Sub1664:0'. It is not used by any node and should be removed from the model. 2024-03-16 15:22:15.412822062 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'Shape1662:0'. It is not used by any node and should be removed from the model. 2024-03-16 15:22:15.412831040 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_41660'. It is not used by any node and should be removed from the model. 2024-03-16 15:22:15.525554751 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2649'. It is not used by any node and should be removed from the model. 2024-03-16 15:22:15.525574428 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2657'. It is not used by any node and should be removed from the model. 2024-03-16 15:22:15.525577167 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2644'. It is not used by any node and should be removed from the model. 2024-03-16 15:22:15.525579922 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2594'. It is not used by any node and should be removed from the model. 2024-03-16 15:22:15.525583236 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2596'. It is not used by any node and should be removed from the model. 2024-03-16 15:22:15.525586891 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2653'. It is not used by any node and should be removed from the model. 2024-03-16 15:22:15.525589904 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2624'. It is not used by any node and should be removed from the model. 2024-03-16 15:22:15.525592868 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2652'. It is not used by any node and should be removed from the model. 2024-03-16 15:22:15.525596530 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2645'. It is not used by any node and should be removed from the model. 2024-03-16 15:22:15.525599328 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2643'. It is not used by any node and should be removed from the model. 2024-03-16 15:22:15.525602181 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2648'. It is not used by any node and should be removed from the model. 2024-03-16 15:22:15.525604868 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2647'. It is not used by any node and should be removed from the model. 2024-03-16 15:22:15.525607502 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2641'. It is not used by any node and should be removed from the model. 2024-03-16 15:22:15.525611251 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2633'. It is not used by any node and should be removed from the model. 2024-03-16 15:22:15.525614155 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2632'. It is not used by any node and should be removed from the model. 2024-03-16 15:22:15.525617785 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2614'. It is not used by any node and should be removed from the model. 2024-03-16 15:22:15.525620526 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2613'. It is not used by any node and should be removed from the model. 2024-03-16 15:22:15.525623573 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2658'. It is not used by any node and should be removed from the model. 2024-03-16 15:22:15.525626884 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt2606'. It is not used by any node and should be removed from the model. 2024-03-16 15:22:15.525630091 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt__2598'. It is not used by any node and should be removed from the model. 2024-03-16 15:22:16,225 - modelscope - INFO - Use user-specified model revision: v1.0.0 2024-03-16 15:22:16,454 - modelscope - INFO - initiate model from /home/gaol/.cache/modelscope/hub/damo/cv_gpen_image-portrait-enhancement 2024-03-16 15:22:16,455 - modelscope - INFO - initiate model from location /home/gaol/.cache/modelscope/hub/damo/cv_gpen_image-portrait-enhancement. 2024-03-16 15:22:16,456 - modelscope - INFO - initialize model from /home/gaol/.cache/modelscope/hub/damo/cv_gpen_image-portrait-enhancement Loading ResNet ArcFace 2024-03-16 15:22:18,087 - modelscope - INFO - load face enhancer model done 2024-03-16 15:22:18,337 - modelscope - INFO - load face detector model done 2024-03-16 15:22:18,583 - modelscope - INFO - load sr model done 2024-03-16 15:22:19,307 - modelscope - INFO - load fqa model done 0%| | 0/15 [00:00<?, ?it/s]2024-03-16 15:22:19,761 - modelscope - WARNING - task skin-retouching-torch input definition is missing 2024-03-16 15:22:21,407 - modelscope - WARNING - task skin-retouching-torch output keys are missing 2024-03-16 15:22:21,411 - modelscope - WARNING - task face_recognition input definition is missing 2024-03-16 15:22:21,555 - modelscope - INFO - model inference done 2024-03-16 15:22:21,555 - modelscope - WARNING - task face_recognition output keys are missing 7%|?????????????? | 1/15 [00:02<00:31, 2.22s/it]2024-03-16 15:22:22,273 - modelscope - INFO - model inference done 13%|???????????????????????????? | 2/15 [00:02<00:17, 1.34s/it]2024-03-16 15:22:22,961 - modelscope - INFO - model inference done 20%|????????????????????????????????????????? | 3/15 [00:03<00:12, 1.04s/it]2024-03-16 15:22:23,690 - modelscope - INFO - model inference done 27%|??????????????????????????????????????????????????????? | 4/15 [00:04<00:10, 1.09it/s]2024-03-16 15:22:24,654 - modelscope - INFO - model inference done 33%|????????????????????????????????????????????????????????????????????? | 5/15 [00:05<00:09, 1.07it/s]2024-03-16 15:22:25,290 - modelscope - INFO - model inference done 40%|?????????????????????????????????????????????????????????????????????????????????? | 6/15 [00:05<00:07, 1.20it/s]2024-03-16 15:22:26,010 - modelscope - INFO - model inference done 47%|???????????????????????????????????????????????????????????????????????????????????????????????? | 7/15 [00:06<00:06, 1.26it/s]2024-03-16 15:22:27,181 - modelscope - INFO - model inference done 53%|?????????????????????????????????????????????????????????????????????????????????????????????????????????????? | 8/15 [00:07<00:06, 1.09it/s]2024-03-16 15:22:28,236 - modelscope - INFO - model inference done 60%|??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? | 9/15 [00:08<00:05, 1.04it/s]2024-03-16 15:22:29,217 - modelscope - INFO - model inference done 67%|???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? | 10/15 [00:09<00:04, 1.04it/s]2024-03-16 15:22:30,171 - modelscope - INFO - model inference done 73%|?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? | 11/15 [00:10<00:03, 1.04it/s]2024-03-16 15:22:31,393 - modelscope - INFO - model inference done 80%|???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? | 12/15 [00:12<00:03, 1.04s/it]2024-03-16 15:22:32,414 - modelscope - INFO - model inference done 87%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? | 13/15 [00:13<00:02, 1.03s/it]2024-03-16 15:22:33,103 - modelscope - INFO - model inference done 93%|??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? | 14/15 [00:13<00:00, 1.07it/s]2024-03-16 15:22:33,818 - modelscope - INFO - model inference done 100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 15/15 [00:14<00:00, 1.04it/s] selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/0.jpg total scores: 0.5047546609308565 face angles 0.9206686066500163 selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/12.jpg total scores: 0.49227943258374113 face angles 0.9208677161732743 selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/6.jpg total scores: 0.4752780163014266 face angles 0.9655220357517296 selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/2.jpg total scores: 0.4714630612945019 face angles 0.9158910610739464 selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/13.jpg total scores: 0.46400164259168414 face angles 0.9542378038928844 selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/4.jpg total scores: 0.43687105673798315 face angles 0.8650432225418873 selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/7.jpg total scores: 0.4353354856725917 face angles 0.9829790242269123 selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/11.jpg total scores: 0.4166387907586268 face angles 0.7875557185843958 selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/8.jpg total scores: 0.4151525819786572 face angles 0.9470527983614994 selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/5.jpg total scores: 0.3955914799303482 face angles 0.9771394596698413 selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/14.jpg total scores: 0.38413634652262346 face angles 0.9993536825291083 selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/10.jpg total scores: 0.34892923676821336 face angles 0.7398419459302334 selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/9.jpg total scores: 0.3327070567244012 face angles 0.8863536608311919 selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/1.jpg total scores: 0.3145806877875612 face angles 0.6169503507383685 selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/3.jpg total scores: 0.3134642199143675 face angles 0.8505665005567431 jpg: 0.jpg face_id_scores 0.5047546609308565 jpg: 12.jpg face_id_scores 0.49227943258374113 jpg: 11.jpg face_id_scores 0.4166387907586268 jpg: 2.jpg face_id_scores 0.4714630612945019 jpg: 1.jpg face_id_scores 0.3145806877875612 jpg: 4.jpg face_id_scores 0.43687105673798315 jpg: 6.jpg face_id_scores 0.4752780163014266 jpg: 13.jpg face_id_scores 0.46400164259168414 jpg: 10.jpg face_id_scores 0.34892923676821336 jpg: 7.jpg face_id_scores 0.4353354856725917 jpg: 8.jpg face_id_scores 0.4151525819786572 jpg: 5.jpg face_id_scores 0.3955914799303482 jpg: 14.jpg face_id_scores 0.38413634652262346 jpg: 9.jpg face_id_scores 0.3327070567244012 jpg: 3.jpg face_id_scores 0.3134642199143675 15it [00:13, 1.12it/s] save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/0.jpg save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/1.jpg save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/2.jpg save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/3.jpg save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/4.jpg save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/5.jpg save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/6.jpg save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/7.jpg save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/8.jpg save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/9.jpg save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/10.jpg save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/11.jpg save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/12.jpg save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/13.jpg save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/14.jpg 2024-03-16 15:22:47,750 - EasyPhoto - train_file_path : /home/gaol/codes/temp/stable-diffusion-webui/extensions/sd-webui-EasyPhoto/scripts/train_kohya/train_lora.py 2024-03-16 15:22:47,750 - EasyPhoto - cache_log_file_path: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-tmp/train_kohya_log.txt 2024-03-16 15:22:49.493300: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT 2024-03-16 15:22:52.402091: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT 2024-03-16 15:22:53,191 - modelscope - INFO - PyTorch version 2.0.1+cu118 Found. 2024-03-16 15:22:53,192 - modelscope - INFO - TensorFlow version 2.13.0 Found. 2024-03-16 15:22:53,192 - modelscope - INFO - Loading ast index from /home/gaol/.cache/modelscope/ast_indexer 2024-03-16 15:22:53,209 - modelscope - INFO - Loading done! Current index file version is 1.9.3, with md5 985d60ab3829178ada728d5649a2ffda and a total number of 943 components indexed 03/16/2024 15:22:53 - INFO - main - Distributed environment: MULTI_GPU Backend: nccl Num processes: 1 Process index: 0 Local process index: 0 Device: cuda:0Mixed precision type: fp16
{'sample_max_value', 'variance_type', 'thresholding', 'clip_sample_range', 'dynamic_thresholding_ratio', 'timestep_spacing', 'prediction_type'} was not found in config. Values will be initialized to default values. UNet2DConditionModel: 64, 8, 768, False, False loading u-net:
loading vae:
loading text encoder:
create LoRA network. base dim (rank): 128, alpha: 64
neuron dropout: p=None, rank dropout: p=None, module dropout: p=None
create LoRA for Text Encoder:
create LoRA for Text Encoder: 72 modules.
create LoRA for U-Net: 192 modules.
enable LoRA for text encoder
enable LoRA for U-Net
Resolving data files: 100%|??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 31/31 [00:00<00:00, 436320.21it/s]
Downloading and preparing dataset imagefolder/default to /home/gaol/.cache/huggingface/datasets/imagefolder/default-cd3ed4aa3d015dcd/0.0.0/37fbb85cc714a338bea574ac6c7d0b5be5aff46c1862c1989b20e0771199e93f...
Downloading data files: 100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 16/16 [00:00<00:00, 106017.16it/s]
Downloading data files: 100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 15/15 [00:00<00:00, 116293.09it/s]
Extracting data files: 100%|???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 15/15 [00:00<00:00, 6948.05it/s]
Dataset imagefolder downloaded and prepared to /home/gaol/.cache/huggingface/datasets/imagefolder/default-cd3ed4aa3d015dcd/0.0.0/37fbb85cc714a338bea574ac6c7d0b5be5aff46c1862c1989b20e0771199e93f. Subsequent calls will reuse this data.
100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 1/1 [00:00<00:00, 1410.32it/s]
03/16/2024 15:23:01 - INFO - main - Running training
03/16/2024 15:23:01 - INFO - main - Num examples = 15
03/16/2024 15:23:01 - INFO - main - Num Epochs = 3000
03/16/2024 15:23:01 - INFO - main - Instantaneous batch size per device = 8
03/16/2024 15:23:01 - INFO - main - Total train batch size (w. parallel, distributed & accumulation) = 64
03/16/2024 15:23:01 - INFO - main - Gradient Accumulation steps = 8
03/16/2024 15:23:01 - INFO - main - Total optimization steps = 3000
Steps: 0%| | 0/3000 [00:00<?, ?it/s]2024-03-16 15:23:02,628 - modelscope - INFO - Model revision not specified, use revision: v2.0.2
2024-03-16 15:23:04,325 - modelscope - INFO - initiate model from /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface
2024-03-16 15:23:04,325 - modelscope - INFO - initiate model from location /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface.
2024-03-16 15:23:04,326 - modelscope - WARNING - No preprocessor field found in cfg.
2024-03-16 15:23:04,326 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2024-03-16 15:23:04,326 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface'}. trying to build by task and model information.
2024-03-16 15:23:04,326 - modelscope - WARNING - Find task: face-detection, model type: None. Insufficient information to build preprocessor, skip building preprocessor
2024-03-16 15:23:04,326 - modelscope - INFO - loading model from /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface/pytorch_model.pt
/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
warnings.warn(
/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or
return tree_unflatten([fn(i) for i in flat_args], spec)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/utils/_pytree.py", line 247, in inner
return f(x)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1212, in validate
raise Exception(
Exception: Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'. Found in aten._to_copy.default( (Parameter containing:
tensor([[-0.0176, -0.0284, 0.0112, ..., 0.0230, 0.0277, 0.0246],
[ 0.0259, 0.0047, 0.0272, ..., 0.0275, 0.0020, 0.0336],
[-0.0033, -0.0350, 0.0030, ..., -0.0004, 0.0218, 0.0237],
...,
[ 0.0084, -0.0230, -0.0020, ..., 0.0119, 0.0042, 0.0045],
[-0.0068, -0.0207, -0.0300, ..., -0.0241, 0.0173, 0.0291],
[-0.0163, -0.0165, 0.0014, ..., -0.0115, 0.0128, 0.0221]],
device='cuda:0', requires_grad=True),), **{'dtype': torch.float16})
Nonefor 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passingweights=None. warnings.warn(msg) 2024-03-16 15:23:04,575 - modelscope - INFO - load model done /home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/conv.py:459: UserWarning: Applied workaround for CuDNN issue, install nvrtc.so (Triggered internally at ../aten/src/ATen/native/cudnn/Conv_v8.cpp:80.) return F.conv2d(input, weight, bias, self.stride, [2024-03-16 15:23:09,607] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward [2024-03-16 15:23:09,809] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing call [2024-03-16 15:23:09,812] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward [2024-03-16 15:23:11,013] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE) [2024-03-16 15:23:11,029] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function debug_wrapper Traceback (most recent call last): File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 670, in call_user_compiler compiled_fn = compiler_fn(gm, self.fake_example_inputs()) File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 1055, in debug_wrapper compiled_gm = compiler_fn(gm, example_inputs) File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/init.py", line 1390, in call return compilefx(model, inputs_, config_patches=self.config) File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 455, in compile_fx return aot_autograd( File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 48, in compiler_fn cg = aot_module_simplified(gm, example_inputs, kwargs) File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2822, in aot_module_simplified compiled_fn = create_aot_dispatcher_function( File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper r = func(*args, kwargs) File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2515, in create_aot_dispatcher_function compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config) File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1676, in aot_wrapper_dedupe fw_metadata, _out = run_functionalized_fw_and_collect_metadata( File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 607, in inner flat_f_outs = f(flat_f_args) File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2793, in functional_call out = Interpreter(mod).run(args[params_len:], kwargs) File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/fx/interpreter.py", line 136, in run self.env[node] = self.run_node(node) File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/fx/interpreter.py", line 177, in run_node return getattr(self, n.op)(n.target, args, kwargs) File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/fx/interpreter.py", line 294, in call_module return submod(*args, kwargs) File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "/home/gaol/codes/temp/stable-diffusion-webui/extensions/sd-webui-EasyPhoto/scripts/train_kohya/utils/lora_utils.py", line 140, in forward lx = self.lora_down(x) File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward return F.linear(input, self.weight, self.bias) File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_inductor/overrides.py", line 38, in __torch_function__ return func(args, kwargs) File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/utils/_stats.py", line 20, in wrapper return fn(*args, *kwargs) File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 987, in __torch_dispatch__ return self.dispatch(func, types, args, kwargs) File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1066, in dispatch args, kwargs = self.validate_and_convert_non_fake_tensors( File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1220, in validate_and_convert_non_fake_tensors return tree_map_only( File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/utils/_pytree.py", line 266, in tree_map_only return tree_map(map_only(ty)(fn), pytree) File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/utils/_pytree.py", line 196, in tree_map return tree_unflatten([fn(i) for i in flat_args], spec) File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/utils/_pytree.py", line 196, inWhile executing %self_text_model_encoder_layers_0_self_attn_q_proj : [#users=1] = call_module[target=self_text_model_encoder_layers_0_self_attn_q_proj](args = (%self_text_model_encoder_layers_0_layer_norm1,), kwargs = {}) Original traceback: File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 272, in forward query_states = self.q_proj(hidden_states) * self.scale | File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 383, in forward hidden_states, attn_weights = self.self_attn( | File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 654, in forward layer_outputs = encoder_layer( | File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 740, in forward encoder_outputs = self.encoder( | File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 822, in forward return self.text_model(
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/home/gaol/codes/temp/stable-diffusion-webui/extensions/sd-webui-EasyPhoto/scripts/train_kohya/train_lora.py", line 1397, in
main()
File "/home/gaol/codes/temp/stable-diffusion-webui/extensions/sd-webui-EasyPhoto/scripts/train_kohya/utils/gpu_info.py", line 190, in wrapper
result = func(*args, kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/extensions/sd-webui-EasyPhoto/scripts/train_kohya/train_lora.py", line 1132, in main
encoder_hidden_states = text_encoder(batch["input_ids"])[0]
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, *kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 82, in forward
return self.dynamo_ctx(self._orig_mod.forward)(args, kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 209, in _fn
return fn(*args, kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/utils/operations.py", line 581, in forward
return model_forward(*args, *kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/utils/operations.py", line 569, in call
return convert_to_fp32(self.model_forward(args, kwargs))
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
return func(*args, kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 337, in catch_errors
return callback(frame, cache_size, hooks)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 404, in _convert_frame
result = inner_convert(frame, cache_size, hooks)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 104, in _fn
return fn(*args, *kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 262, in _convert_frame_assert
return _compile(
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
r = func(args, kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 324, in _compile
out_code = transform_code_object(code, transform)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 445, in transform_code_object
transformations(instructions, code_options)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 311, in transform
tracer.run()
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1726, in run
super().run()
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 576, in run
and self.step()
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 540, in step
getattr(self, inst.opname)(inst)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1792, in RETURN_VALUE
self.output.compile_subgraph(
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 541, in compile_subgraph
self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 588, in compile_and_call_fx_graph
compiled_fn = self.call_user_compiler(gm)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
r = func(*args, *kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 675, in call_user_compiler
raise BackendCompilerFailed(self.compiler_fn, e) from e
torch._dynamo.exc.BackendCompilerFailed: debug_wrapper raised Exception: Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'. Found in aten._to_copy.default((Parameter containing:
tensor([[-0.0176, -0.0284, 0.0112, ..., 0.0230, 0.0277, 0.0246],
[ 0.0259, 0.0047, 0.0272, ..., 0.0275, 0.0020, 0.0336],
[-0.0033, -0.0350, 0.0030, ..., -0.0004, 0.0218, 0.0237],
...,
[ 0.0084, -0.0230, -0.0020, ..., 0.0119, 0.0042, 0.0045],
[-0.0068, -0.0207, -0.0300, ..., -0.0241, 0.0173, 0.0291],
[-0.0163, -0.0165, 0.0014, ..., -0.0115, 0.0128, 0.0221]],
device='cuda:0', requires_grad=True),), **{'dtype': torch.float16})
While executing %self_text_model_encoder_layers_0_self_attn_q_proj : [#users=1] = call_module[target=self_text_model_encoder_layers_0_self_attn_q_proj](args = (%self_text_model_encoder_layers_0_layer_norm1,), kwargs = {}) Original traceback: File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 272, in forward query_states = self.q_proj(hidden_states) * self.scale | File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 383, in forward hidden_states, attn_weights = self.self_attn( | File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 654, in forward layer_outputs = encoder_layer( | File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 740, in forward encoder_outputs = self.encoder( | File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 822, in forward return self.text_model(
Set torch._dynamo.config.verbose=True for more information
You can suppress this exception and fall back to eager by setting: torch._dynamo.config.suppress_errors = True
Steps: 0%| | 0/3000 [00:11<?, ?it/s] ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 2387) of binary: /home/gaol/codes/temp/stable-diffusion-webui/venv/bin/python3 Traceback (most recent call last): File "/home/gaol/miniforge3/envs/stable/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/gaol/miniforge3/envs/stable/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 989, in
main()
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 985, in main
launch_command(args)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 970, in launch_command
multi_gpu_launcher(args)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 646, in multi_gpu_launcher
distrib_run.run(args)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
/home/gaol/codes/temp/stable-diffusion-webui/extensions/sd-webui-EasyPhoto/scripts/train_kohya/train_lora.py FAILED
Failures: