Unable to run training or evaluation

[Metal079]

Hello, I ran the anaconda instructions but when I try to use accelerate launch main.py --config config/align_prop.py:hps or accelerate launch main.py --config config/align_prop.py:evaluate I receive the following error

(alignprop) C:\Users\metal\OneDrive\Documents\programming\AlignProp>accelerate launch main.py --config config/align_prop.py:hps Traceback (most recent call last): File "C:\Users\metal\miniconda3\envs\alignprop\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\metal\miniconda3\envs\alignprop\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Users\metal\miniconda3\envs\alignprop\Scripts\accelerate.exe\__main__.py", line 7, in <module> File "C:\Users\metal\miniconda3\envs\alignprop\lib\site-packages\accelerate\commands\accelerate_cli.py", line 45, in main args.func(args) File "C:\Users\metal\miniconda3\envs\alignprop\lib\site-packages\accelerate\commands\launch.py", line 892, in launch_command args, defaults, mp_from_config_flag = _validate_launch_command(args) File "C:\Users\metal\miniconda3\envs\alignprop\lib\site-packages\accelerate\commands\launch.py", line 777, in _validate_launch_command defaults = load_config_from_file(args.config_file) File "C:\Users\metal\miniconda3\envs\alignprop\lib\site-packages\accelerate\commands\config\config_args.py", line 72, in load_config_from_file return config_class.from_yaml_file(yaml_file=config_file) File "C:\Users\metal\miniconda3\envs\alignprop\lib\site-packages\accelerate\commands\config\config_args.py", line 131, in from_yaml_file return cls(**config_dict) TypeError: ClusterConfig.__init__() got an unexpected keyword argument 'debug'

I am on windows 11 and using a 4090.

[mihirp1998]

I haven't tested the code on windows.

maybe can you try this solution? https://github.com/bmaltais/kohya_ss/issues/564

If this doesn't work maybe try getting rid of the debug variable from config/align_prop.py and main.py ?

let me know if this works.

[Metal079]

I haven't tested the code on windows.

maybe can you try this solution? bmaltais/kohya_ss#564

If this doesn't work maybe try getting rid of the debug variable from config/align_prop.py and main.py ?

let me know if this works.

Thank you, the linked issue solved my problem and I was able to proceed but now there seems to be a second issue.

(alignprop) C:\Users\metal\OneDrive\Documents\programming\AlignProp>accelerate launch main.py --config config/align_prop.py:hps The following values were not passed to accelerate launch and had defaults used instead: --dynamo_backend was set to a value of 'no' To avoid this warning pass in values for each of the problematic parameters or run accelerate config. wandb: Currently logged in as: metal. Use wandb login --relogin to force relogin wandb: Tracking run with wandb version 0.15.12 wandb: Run data is saved locally in C:\Users\metal\OneDrive\Documents\programming\AlignProp\wandb\run-20231009_112348-vjaohu9v wandb: Run wandb offline to turn off syncing. wandb: Syncing run lilac-butterfly-7 wandb: View project at https://wandb.ai/metal/align-prop wandb: View run at https://wandb.ai/metal/align-prop/runs/vjaohu9v I1009 11:23:52.246612 7632 logging.py:47] aesthetic_target: 10 allow_tf32: true debug: false eval_prompt_fn: eval_hps_v2_all grad_checkpoint: true grad_scale: 1 logdir: logs max_vis_images: 2 mixed_precision: fp16 num_checkpoint_limit: 10 num_epochs: 200 only_eval: false per_prompt_stat_tracking: buffer_size: 32 min_count: 16 pretrained: model: runwayml/stable-diffusion-v1-5 revision: main prompt_fn: hps_v2_all resume_from: '' resume_from_2: '' reward_fn: hps run_name: 2023.10.09_11.23.46 same_evaluation: true save_freq: 4 sd_guidance_scale: 7.5 seed: 42 soup_inference: false steps: 50 train: adam_beta1: 0.9 adam_beta2: 0.999 adam_epsilon: 1.0e-08 adam_weight_decay: 0.1 batch_size_per_gpu: 128 batch_size_per_gpu_available: 1 data_loader_iterations: 256 gradient_accumulation_steps: 128 learning_rate: 0.001 loss_coeff: 0.01 max_grad_norm: 5.0 num_gpus: 1 samples_per_epoch_per_gpu: 256 total_batch_size: 128 total_samples_per_epoch: 256 use_8bit_adam: false trunc_backprop_timestep: 40 truncated_backprop: true truncated_backprop_minmax: !!python/tuple

• 0
• 50 truncated_backprop_rand: true vis_freq: 1 visualize_eval: true visualize_train: false

unet\diffusion_pytorch_model.safetensors not found text_config_dict is provided which will be used to initialize CLIPTextConfig. The value text_config["id2label"] will be overriden. Traceback (most recent call last): File "C:\Users\metal\OneDrive\Documents\programming\AlignProp\main.py", line 610, in app.run(main) File "C:\Users\metal\miniconda3\envs\alignprop\lib\site-packages\absl\app.py", line 308, in run _run_main(main, args) File "C:\Users\metal\miniconda3\envs\alignprop\lib\site-packages\absl\app.py", line 254, in _run_main sys.exit(main(argv)) File "C:\Users\metal\OneDrive\Documents\programming\AlignProp\main.py", line 350, in main neg_prompt_embed = pipeline.text_encoder( File "C:\Users\metal\miniconda3\envs\alignprop\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "C:\Users\metal\miniconda3\envs\alignprop\lib\site-packages\transformers\models\clip\modeling_clip.py", line 822, in forward return self.text_model( File "C:\Users\metal\miniconda3\envs\alignprop\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "C:\Users\metal\miniconda3\envs\alignprop\lib\site-packages\transformers\models\clip\modeling_clip.py", line 740, in forward encoder_outputs = self.encoder( File "C:\Users\metal\miniconda3\envs\alignprop\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "C:\Users\metal\miniconda3\envs\alignprop\lib\site-packages\transformers\models\clip\modeling_clip.py", line 654, in forward layer_outputs = encoder_layer( File "C:\Users\metal\miniconda3\envs\alignprop\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "C:\Users\metal\miniconda3\envs\alignprop\lib\site-packages\transformers\models\clip\modeling_clip.py", line 382, in forward hidden_states = self.layer_norm1(hidden_states) File "C:\Users\metal\miniconda3\envs\alignprop\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(args, **kwargs) File "C:\Users\metal\miniconda3\envs\alignprop\lib\site-packages\torch\nn\modules\normalization.py", line 190, in forward return F.layer_norm( File "C:\Users\metal\miniconda3\envs\alignprop\lib\site-packages\torch\nn\functional.py", line 2515, in layer_norm return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled) RuntimeError: "LayerNormKernelImpl" not implemented for 'Half' wandb: Waiting for W&B process to finish... (failed 1). Press Ctrl-C to abort syncing. wandb: View run lilac-butterfly-7 at: https://wandb.ai/metal/align-prop/runs/vjaohu9v wandb: View job at https://wandb.ai/metal/align-prop/jobs/QXJ0aWZhY3RDb2xsZWN0aW9uOjEwNTIwNTg1NA==/version_details/v4 wandb: Synced 6 W&B file(s), 0 media file(s), 2 artifact file(s) and 0 other file(s) wandb: Find logs at: .\wandb\run-20231009_112348-vjaohu9v\logs Traceback (most recent call last): File "C:\Users\metal\miniconda3\envs\alignprop\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\metal\miniconda3\envs\alignprop\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Users\metal\miniconda3\envs\alignprop\Scripts\accelerate.exe__main__.py", line 7, in File "C:\Users\metal\miniconda3\envs\alignprop\lib\site-packages\accelerate\commands\accelerate_cli.py", line 45, in main args.func(args) File "C:\Users\metal\miniconda3\envs\alignprop\lib\site-packages\accelerate\commands\launch.py", line 915, in launch_command simple_launcher(args) File "C:\Users\metal\miniconda3\envs\alignprop\lib\site-packages\accelerate\commands\launch.py", line 578, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['C:\Users\metal\miniconda3\envs\alignprop\python.exe', 'main.py', '--config', 'config/align_prop.py:hps']' returned non-zero exit status 1.

[Metal079]

Nevermind I figured it out! I manually set the device to "cuda" and upgraded pytorch to 2.1 with cuda 12.1 and it works now!