pydantic.errors.PydanticUserError: If you use `@root_validator` with pre=False (the default) you MUST specify `skip_on_failure=True`. Note that `@root_validator` is deprecated and should be replaced with `@model_validator`.

[lvsh2012]

/mnt/conda/envs/kolors/lib/python3.8/site-packages/pydantic/_internal/_config.py:341: UserWarning: Valid config keys have changed in V2:

• 'allow_population_by_field_name' has been renamed to 'populate_by_name'
• 'validate_all' has been renamed to 'validate_default' warnings.warn(message, UserWarning) /mnt/conda/envs/kolors/lib/python3.8/site-packages/pydantic/_internal/_fields.py:132: UserWarning: Field "model_persistencethreshold" in DeepSpeedZeroConfig has conflict with protected namespace "model".

You may be able to resolve this warning by setting model_config['protected_namespaces'] = (). warnings.warn( /mnt/conda/envs/kolors/lib/python3.8/site-packages/pydantic/_internal/_config.py:341: UserWarning: Valid config keys have changed in V2:

• 'validate_all' has been renamed to 'validate_default' warnings.warn(message, UserWarning) Traceback (most recent call last): File "dreambooth/train_dreambooth_lora.py", line 34, in import deepspeed File "/mnt/conda/envs/kolors/lib/python3.8/site-packages/deepspeed/init.py", line 15, in from . import module_inject File "/mnt/conda/envs/kolors/lib/python3.8/site-packages/deepspeed/module_inject/init.py", line 1, in from .replace_module import replace_transformer_layer, revert_transformer_layer, ReplaceWithTensorSlicing, GroupQuantizer, generic_injection File "/mnt/conda/envs/kolors/lib/python3.8/site-packages/deepspeed/module_inject/replace_module.py", line 801, in from ..pipe import PipelineModule File "/mnt/conda/envs/kolors/lib/python3.8/site-packages/deepspeed/pipe/init.py", line 1, in from ..runtime.pipe import PipelineModule, LayerSpec, TiedLayerSpec File "/mnt/conda/envs/kolors/lib/python3.8/site-packages/deepspeed/runtime/pipe/init.py", line 1, in from .module import PipelineModule, LayerSpec, TiedLayerSpec File "/mnt/conda/envs/kolors/lib/python3.8/site-packages/deepspeed/runtime/pipe/module.py", line 14, in from ..activation_checkpointing import checkpointing File "/mnt/conda/envs/kolors/lib/python3.8/site-packages/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 24, in from deepspeed.runtime.config import DeepSpeedConfig File "/mnt/conda/envs/kolors/lib/python3.8/site-packages/deepspeed/runtime/config.py", line 29, in from ..monitor.config import get_monitor_config File "/mnt/conda/envs/kolors/lib/python3.8/site-packages/deepspeed/monitor/config.py", line 69, in class DeepSpeedMonitorConfig(DeepSpeedConfigModel): File "/mnt/conda/envs/kolors/lib/python3.8/site-packages/deepspeed/monitor/config.py", line 81, in DeepSpeedMonitorConfig def check_enabled(cls, values): File "/mnt/conda/envs/kolors/lib/python3.8/site-packages/pydantic/deprecated/class_validators.py", line 234, in root_validator return root_validator()(*__args) # type: ignore File "/mnt/conda/envs/kolors/lib/python3.8/site-packages/pydantic/deprecated/class_validators.py", line 240, in root_validator raise PydanticUserError( pydantic.errors.PydanticUserError: If you use @root_validator with pre=False (the default) you MUST specify skip_on_failure=True. Note that @root_validator is deprecated and should be replaced with @model_validator.

For further information visit https://errors.pydantic.dev/2.9/u/root-validator-pre-skip ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 87663) of binary: /mnt/conda/envs/kolors/bin/python Traceback (most recent call last): File "/mnt/conda/envs/kolors/bin/accelerate", line 8, in sys.exit(main()) File "/mnt/conda/envs/kolors/lib/python3.8/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main args.func(args) File "/mnt/conda/envs/kolors/lib/python3.8/site-packages/accelerate/commands/launch.py", line 1008, in launch_command deepspeed_launcher(args) File "/mnt/conda/envs/kolors/lib/python3.8/site-packages/accelerate/commands/launch.py", line 724, in deepspeed_launcher distrib_run.run(args) File "/mnt/conda/envs/kolors/lib/python3.8/site-packages/torch/distributed/run.py", line 753, in run elastic_launch( File "/mnt/conda/envs/kolors/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 132, in call return launch_agent(self._config, self._entrypoint, list(args)) File "/mnt/conda/envs/kolors/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

dreambooth/train_dreambooth_lora.py FAILED

Failures:

------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-09-26_14:30:29 host : dl rank : 0 (local_rank: 0) exitcode : 1 (pid: 87663) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================

[Deng-Xian-Sheng]

update deepspeed

pip install --uxxxx deepspeed