search

XLabs-AI / x-flux

Apache License 2.0
1.65k stars 118 forks source link

RuntimeError: Given groups=1, weight of size [128, 3, 3, 3], expected input[1, 4, 512, 512] to have 3 channels, but got 4 channels instead #66

Closed rbur0425 closed 3 months ago

rbur0425 commented 3 months ago

I have an A6000 GPU 48 GB memory. When I run x-flux i get the follow error - 


[2024-08-21 18:30:09,982] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
[2024-08-21 18:30:09,985] [INFO] [logging.py:96:log_dist] [Rank 0] Using client Optimizer as basic optimizer
[2024-08-21 18:30:09,985] [INFO] [logging.py:96:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer
[2024-08-21 18:30:10,000] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Basic Optimizer = AdamW
[2024-08-21 18:30:10,000] [INFO] [utils.py:56:is_zero_supported_optimizer] Checking ZeRO support for optimizer=AdamW type=<class 'torch.optim.adamw.AdamW'>
[2024-08-21 18:30:10,000] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 2 optimizer
[2024-08-21 18:30:10,000] [INFO] [stage_1_and_2.py:148:__init__] Reduce bucket size 500,000,000
[2024-08-21 18:30:10,000] [INFO] [stage_1_and_2.py:149:__init__] Allgather bucket size 500,000,000
[2024-08-21 18:30:10,000] [INFO] [stage_1_and_2.py:150:__init__] CPU Offload: False
[2024-08-21 18:30:10,000] [INFO] [stage_1_and_2.py:151:__init__] Round robin gradient partitioning: False
[2024-08-21 18:30:10,358] [INFO] [utils.py:781:see_memory_usage] Before initializing optimizer states
[2024-08-21 18:30:10,359] [INFO] [utils.py:782:see_memory_usage] MA 31.65 GB         Max_MA 31.67 GB         CA 31.74 GB         Max_CA 32 GB
[2024-08-21 18:30:10,359] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 11.78 GB, percent = 3.7%
[2024-08-21 18:30:10,539] [INFO] [utils.py:781:see_memory_usage] After initializing optimizer states
[2024-08-21 18:30:10,539] [INFO] [utils.py:782:see_memory_usage] MA 31.65 GB         Max_MA 31.7 GB         CA 31.78 GB         Max_CA 32 GB
[2024-08-21 18:30:10,540] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 11.78 GB, percent = 3.7%
[2024-08-21 18:30:10,540] [INFO] [stage_1_and_2.py:543:__init__] optimizer state initialized
[2024-08-21 18:30:10,699] [INFO] [utils.py:781:see_memory_usage] After initializing ZeRO optimizer
[2024-08-21 18:30:10,700] [INFO] [utils.py:782:see_memory_usage] MA 31.65 GB         Max_MA 31.65 GB         CA 31.78 GB         Max_CA 32 GB
[2024-08-21 18:30:10,700] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 11.78 GB, percent = 3.7%
[2024-08-21 18:30:10,705] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Final Optimizer = DeepSpeedZeroOptimizer
[2024-08-21 18:30:10,705] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed using client LR scheduler
[2024-08-21 18:30:10,705] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed LR Scheduler = None
[2024-08-21 18:30:10,705] [INFO] [logging.py:96:log_dist] [Rank 0] step=0, skipped=0, lr=[1e-05], mom=[(0.9, 0.999)]
[2024-08-21 18:30:10,707] [INFO] [config.py:997:print] DeepSpeedEngine configuration:
[2024-08-21 18:30:10,707] [INFO] [config.py:1001:print]   activation_checkpointing_config  {
    "partition_activations": false,
    "contiguous_memory_optimization": false,
    "cpu_checkpointing": false,
    "number_checkpoints": null,
    "synchronize_checkpoint_boundary": false,
    "profile": false
}
[2024-08-21 18:30:10,708] [INFO] [config.py:1001:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True}
[2024-08-21 18:30:10,708] [INFO] [config.py:1001:print]   amp_enabled .................. False
[2024-08-21 18:30:10,708] [INFO] [config.py:1001:print]   amp_params ................... False
[2024-08-21 18:30:10,708] [INFO] [config.py:1001:print]   autotuning_config ............ {
    "enabled": false,
    "start_step": null,
    "end_step": null,
    "metric_path": null,
    "arg_mappings": null,
    "metric": "throughput",
    "model_info": null,
    "results_dir": "autotuning_results",
    "exps_dir": "autotuning_exps",
    "overwrite": true,
    "fast": true,
    "start_profile_step": 3,
    "end_profile_step": 5,
    "tuner_type": "gridsearch",
    "tuner_early_stopping": 5,
    "tuner_num_trials": 50,
    "model_info_path": null,
    "mp_size": 1,
    "max_train_batch_size": null,
    "min_train_batch_size": 1,
    "max_train_micro_batch_size_per_gpu": 1.024000e+03,
    "min_train_micro_batch_size_per_gpu": 1,
    "num_tuning_micro_batch_sizes": 3
}
[2024-08-21 18:30:10,708] [INFO] [config.py:1001:print]   bfloat16_enabled ............. True
[2024-08-21 18:30:10,708] [INFO] [config.py:1001:print]   bfloat16_immediate_grad_update  False
[2024-08-21 18:30:10,708] [INFO] [config.py:1001:print]   checkpoint_parallel_write_pipeline  False
[2024-08-21 18:30:10,708] [INFO] [config.py:1001:print]   checkpoint_tag_validation_enabled  True
[2024-08-21 18:30:10,708] [INFO] [config.py:1001:print]   checkpoint_tag_validation_fail  False
[2024-08-21 18:30:10,708] [INFO] [config.py:1001:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x76cfa81dbeb0>
[2024-08-21 18:30:10,708] [INFO] [config.py:1001:print]   communication_data_type ...... None
[2024-08-21 18:30:10,708] [INFO] [config.py:1001:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2024-08-21 18:30:10,708] [INFO] [config.py:1001:print]   curriculum_enabled_legacy .... False
[2024-08-21 18:30:10,708] [INFO] [config.py:1001:print]   curriculum_params_legacy ..... False
[2024-08-21 18:30:10,708] [INFO] [config.py:1001:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2024-08-21 18:30:10,708] [INFO] [config.py:1001:print]   data_efficiency_enabled ...... False
[2024-08-21 18:30:10,708] [INFO] [config.py:1001:print]   dataloader_drop_last ......... False
[2024-08-21 18:30:10,708] [INFO] [config.py:1001:print]   disable_allgather ............ False
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   dump_state ................... False
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   dynamic_loss_scale_args ...... None
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   eigenvalue_enabled ........... False
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   eigenvalue_gas_boundary_resolution  1
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   eigenvalue_layer_name ........ bert.encoder.layer
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   eigenvalue_layer_num ......... 0
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   eigenvalue_max_iter .......... 100
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   eigenvalue_stability ......... 1e-06
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   eigenvalue_tol ............... 0.01
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   eigenvalue_verbose ........... False
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   elasticity_enabled ........... False
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   flops_profiler_config ........ {
    "enabled": false,
    "recompute_fwd_factor": 0.0,
    "profile_step": 1,
    "module_depth": -1,
    "top_modules": 1,
    "detailed": true,
    "output_file": null
}
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   fp16_auto_cast ............... None
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   fp16_enabled ................. False
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   fp16_master_weights_and_gradients  False
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   global_rank .................. 0
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   grad_accum_dtype ............. None
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   gradient_accumulation_steps .. 2
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   gradient_clipping ............ 1.0
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   gradient_predivide_factor .... 1.0
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   graph_harvesting ............. False
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   initial_dynamic_scale ........ 1
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   load_universal_checkpoint .... False
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   loss_scale ................... 1.0
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   memory_breakdown ............. False
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   mics_hierarchial_params_gather  False
[2024-08-21 18:30:10,709] [INFO] [config.py:1001:print]   mics_shard_size .............. -1
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') comet=CometConfig(enabled=False, samples_log_interval=100, project=None, workspace=None, api_key=None, experiment_name=None, experiment_key=None, online=None, mode=None) wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=False
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   nebula_config ................ {
    "enabled": false,
    "persistent_storage_path": null,
    "persistent_time_interval": 100,
    "num_of_version_in_retention": 2,
    "enable_nebula_load": true,
    "load_path": null
}
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   optimizer_legacy_fusion ...... False
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   optimizer_name ............... None
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   optimizer_params ............. None
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True}
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   pld_enabled .................. False
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   pld_params ................... False
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   prescale_gradients ........... False
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   scheduler_name ............... None
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   scheduler_params ............. None
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   seq_parallel_communication_data_type  torch.float32
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   sparse_attention ............. None
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   sparse_gradients_enabled ..... False
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   steps_per_print .............. inf
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   timers_config ................ enabled=True synchronized=True
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   train_batch_size ............. 2
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   train_micro_batch_size_per_gpu  1
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   use_data_before_expert_parallel_  False
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   use_node_local_storage ....... False
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   wall_clock_breakdown ......... False
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   weight_quantization_config ... None
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   world_size ................... 1
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   zero_allow_untested_optimizer  True
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   zero_config .................. stage=2 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=500,000,000 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=500,000,000 overlap_comm=False load_from_fp32_weights=True elastic_checkpoint=False offload_param=DeepSpeedZeroOffloadParamConfig(device='none', nvme_path=None, buffer_count=5, buffer_size=100,000,000, max_in_cpu=1,000,000,000, pin_memory=False) offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='none', nvme_path=None, buffer_count=4, pin_memory=False, pipeline=False, pipeline_read=False, pipeline_write=False, fast_init=False, ratio=1.0) sub_group_size=1,000,000,000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50,000,000 param_persistence_threshold=100,000 model_persistence_threshold=sys.maxsize max_live_parameters=1,000,000,000 max_reuse_distance=1,000,000,000 gather_16bit_weights_on_model_save=False use_all_reduce_for_fetch_params=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   zero_enabled ................. True
[2024-08-21 18:30:10,710] [INFO] [config.py:1001:print]   zero_force_ds_cpu_optimizer .. True
[2024-08-21 18:30:10,711] [INFO] [config.py:1001:print]   zero_optimization_stage ...... 2
[2024-08-21 18:30:10,711] [INFO] [config.py:987:print_user_config]   json = {
    "train_batch_size": 2,
    "train_micro_batch_size_per_gpu": 1,
    "gradient_accumulation_steps": 2,
    "zero_optimization": {
        "stage": 2,
        "offload_optimizer": {
            "device": "none",
            "nvme_path": null
        },
        "offload_param": {
            "device": "none",
            "nvme_path": null
        },
        "stage3_gather_16bit_weights_on_model_save": false
    },
    "gradient_clipping": 1.0,
    "steps_per_print": inf,
    "bf16": {
        "enabled": true
    },
    "fp16": {
        "enabled": false
    },
    "zero_allow_untested_optimizer": true
}
08/21/2024 18:30:10 - INFO - __main__ - ***** Running training *****
08/21/2024 18:30:10 - INFO - __main__ -   Num Epochs = 167
08/21/2024 18:30:10 - INFO - __main__ -   Instantaneous batch size per device = 1
08/21/2024 18:30:10 - INFO - __main__ -   Total train batch size (w. parallel, distributed & accumulation) = 2
08/21/2024 18:30:10 - INFO - __main__ -   Gradient Accumulation steps = 2
08/21/2024 18:30:10 - INFO - __main__ -   Total optimization steps = 2000
Checkpoint 'latest' does not exist. Starting a new training run.
Steps:   0%|                                                                                                                 | 0/2000 [00:00<?, ?it/s][rank0]: Traceback (most recent call last):
[rank0]:   File "/home/roland/Sites/x-flux/train_flux_lora_deepspeed.py", line 301, in <module>
[rank0]:     main()
[rank0]:   File "/home/roland/Sites/x-flux/train_flux_lora_deepspeed.py", line 218, in main
[rank0]:     x_1 = vae.encode(img.to(accelerator.device).to(torch.float32))
[rank0]:   File "/home/roland/Sites/x-flux/src/flux/modules/autoencoder.py", line 303, in encode
[rank0]:     z = self.reg(self.encoder(x))
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/home/roland/Sites/x-flux/src/flux/modules/autoencoder.py", line 161, in forward
[rank0]:     hs = [self.conv_in(x)]
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py", line 458, in forward
[rank0]:     return self._conv_forward(input, self.weight, self.bias)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py", line 454, in _conv_forward
[rank0]:     return F.conv2d(input, weight, bias, self.stride,
[rank0]: RuntimeError: Given groups=1, weight of size [128, 3, 3, 3], expected input[1, 4, 512, 512] to have 3 channels, but got 4 channels instead
Steps:   0%|                                                                                                                 | 0/2000 [00:01<?, ?it/s]
[rank0]:[W821 18:30:11.816114849 ProcessGroupNCCL.cpp:1168] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())
E0821 18:30:12.954000 126076669222912 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 0 (pid: 1082) of binary: /usr/bin/python3
Traceback (most recent call last):
  File "/usr/local/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/accelerate_cli.py", line 46, in main
    args.func(args)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 1067, in launch_command
    deepspeed_launcher(args)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 771, in deepspeed_launcher
    distrib_run.run(args)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/run.py", line 892, in run
    elastic_launch(
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 133, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
train_flux_lora_deepspeed.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-08-21_18:30:12
  host      : 347754e4bf53
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 1082)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================```
arrowonstr commented 3 months ago

https://github.com/XLabs-AI/x-flux/issues/49#issuecomment-2293143354 This may help u

rbur0425 commented 3 months ago

@arrowonstr - you are a life saver. i was pulling my hair out. made sure the images had 3 channels and everything is working fine!