YutackPark
commented
2 months ago That is strange. Could you copy paste whole error backtrace?
켬 2024. 9. 4. 위치 오전 4:23, turbosonics @.***> 작성:
I performed fine-tune training based on MP pretrained model using 2 GPUs for a week (our local cluster has 1 week wall time limit). Following is the input file parameters for the fine-tune training.
model: chemical_species: 'Auto' cutoff: 5.0 channel: 128 is_parity: False lmax: 2 num_convolution_layer: 5 irreps_manual: - "128x0e" - "128x0e+64x1e+32x2e" - "128x0e+64x1e+32x2e" - "128x0e+64x1e+32x2e" - "128x0e+64x1e+32x2e" - "128x0e" weight_nn_hidden_neurons: [64, 64] radial_basis: radial_basis_name: 'bessel' bessel_basis_num: 8 cutoff_function: cutoff_function_name: 'XPLOR' cutoff_on: 4.5 act_gate: {'e': 'silu', 'o': 'tanh'} act_scalar: {'e': 'silu', 'o': 'tanh'} conv_denominator: "avg_num_neigh" train_shift_scale: True train_denominator: True self_connection_type: 'linear' train: train_shuffle: True random_seed: 123 is_train_stress: True epoch: 5000 loss: 'Huber' loss_param: delta: 0.01 optimizer: 'adam' optim_param: lr: 0.002 scheduler: 'ReduceLROnPlateau' scheduler_param: factor: 0.5 patience: 100 # best_metric: TotalLoss force_loss_weight: 2.0 stress_loss_weight: 0.1 per_epoch: 10 error_record: - ['Energy', 'RMSE'] - ['Force', 'RMSE'] - ['Stress', 'RMSE'] - ['Energy', 'MAE'] - ['Force', 'MAE'] - ['Stress', 'MAE'] - ['Energy', 'Loss'] - ['Force', 'Loss'] - ['Stress', 'Loss'] - ['TotalLoss', 'None'] continue: reset_optimizer: True reset_scheduler: True reset_epoch: True checkpoint: '${PTMODEL_DIR}/${PTMODEL_NAME}' # Set True to use shift, scale, and avg_num_neigh from checkpoint (highly recommended) use_statistic_values_of_checkpoint: True data: data_shuffle: True batch_size: 6 data_divide_ratio: 0.2 data_format: 'ase' load_dataset_path: ['${GEOMETRY_DIR}/${GEOMETRY_TRAIN_NAME}']
In this case, the MP pretrained model for "checkpoint" option is {SevenNet directory}/sevenn/pretrained_potentials/SevenNet_0__11July2024/checkpoint_sevennet_0.pth file.
Now, the training job has been killed by the server due to 1 week wall-time limit, so I tried to restart (or continue) this training work by following input script:
model: chemical_species: 'Auto' cutoff: 5.0 channel: 128 is_parity: False lmax: 2 num_convolution_layer: 5 irreps_manual: - "128x0e" - "128x0e+64x1e+32x2e" - "128x0e+64x1e+32x2e" - "128x0e+64x1e+32x2e" - "128x0e+64x1e+32x2e" - "128x0e" weight_nn_hidden_neurons: [64, 64] radial_basis: radial_basis_name: 'bessel' bessel_basis_num: 8 cutoff_function: cutoff_function_name: 'XPLOR' cutoff_on: 4.5 act_gate: {'e': 'silu', 'o': 'tanh'} act_scalar: {'e': 'silu', 'o': 'tanh'} conv_denominator: "avg_num_neigh" train_shift_scale: True train_denominator: True self_connection_type: 'linear' train: train_shuffle: True random_seed: 123 is_train_stress: True epoch: 5000 loss: 'Huber' loss_param: delta: 0.01 optimizer: 'adam' optim_param: lr: 0.002 scheduler: 'ReduceLROnPlateau' scheduler_param: factor: 0.5 patience: 100 # best_metric: TotalLoss force_loss_weight: 2.0 stress_loss_weight: 0.1 per_epoch: 10 error_record: - ['Energy', 'RMSE'] - ['Force', 'RMSE'] - ['Stress', 'RMSE'] - ['Energy', 'MAE'] - ['Force', 'MAE'] - ['Stress', 'MAE'] - ['Energy', 'Loss'] - ['Force', 'Loss'] - ['Stress', 'Loss'] - ['TotalLoss', 'None'] continue: reset_optimizer: False reset_scheduler: False reset_epoch: False checkpoint: '${PTMODEL_DIR}/${PTMODEL_NAME}' # Set True to use shift, scale, and avg_num_neigh from checkpoint (highly recommended) use_statistic_values_of_checkpoint: True data: data_shuffle: True batch_size: 6 data_divide_ratio: 0.2 data_format: 'ase' load_dataset_path: ['${GEOMETRY_DIR}/${GEOMETRY_TRAIN_NAME}']
In this case, the geometry files are the same, but I used {First_fine_tune_directory}/checkpoint_best.pth for the "checkpoint" option. Then I changed three options, reset_optimizer, reset_scheduler, and reset_epoch from True to False. Then I submitted the job using the exact same Slurm condition.
Then, I faced "torch.distributed.elastic.multiprocessing.errors.ChildFailedError"
I can't understand this. The first training job to fine-tune the MP pretrained model workred with 2 GPUs. But then, how and why restart/continue job fails with almost same input parameter setting and exactly the same Slrum setting? Am I doing something wrong during restart/continuation of fine-tune training job?
I compiled SevenNet with Cuda11.8 and prebuilt versrion of Pytorch 2.3.0 (which we can download from https://pytorch.org/get-started/locally/) and python 3.9 to virtual environment.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>
turbosonics
I performed fine-tune training based on MP pretrained model using 2 GPUs for a week (our local cluster has 1 week wall time limit). Following is the input file parameters for the fine-tune training.
In this case, the MP pretrained model for "checkpoint" option is {SevenNet_directory}/sevenn/pretrained_potentials/SevenNet_0__11July2024/checkpoint_sevennet_0.pth file.
Now, the training job has been killed by the server due to 1 week wall-time limit, so I tried to restart (or continue) this training work by following input script:
In this case, the geometry files is the same one with the original first fine-tuning training, but I used {First_fine_tune_directory}/checkpoint_best.pth for the "checkpoint" option this time because it is continuation of the first fine-tuning training.
Then I changed three options, reset_optimizer, reset_scheduler, and reset_epoch from True to False. Then I submitted the job using the exact same Slurm condition.
Command to execute the SevenNet is the same for original first fine-tuning job and continuing fine-tuning job:
srun torchrun --nnodes 2 --nproc_per_node 1 --rdzv_endpoint=$head_node_ip --rdzv_id=$RANDOM --rdzv_backend=c10d --no_python sevenn input.yaml -dI also loaded same modules and performed from the same virtual environment for the first fine-tuning training job and the continuing fine-tuning training job:
I used the same commands to escape rendezvous crash for two jobs:
Then, I faced "torch.distributed.elastic.multiprocessing.errors.ChildFailedError" Actual error file is too long to copy here...
I can't understand this. The first training job to fine-tune the MP pretrained model workred with 2 GPUs. But then, how and why restart/continue job fails with almost same input parameter setting and exactly the same Slrum setting? Am I doing something wrong to the input file for this restart/continuation of fine-tune training job?
CPU memory cannot be an issue because I requested 496GB (whole memory of node) for two GPU nodes. Since the original first fine-tune training operated normally, this shouldn't be an issue of GPU memory as well.
Should I need to use the same GPU node that I used for the first fine-tune training for the continuing fine-tune training? But I just tried the same GPU nodes with the first fine tuning training, still printed out the same Child Faild error crash.
I compiled SevenNet with Cuda11.8 and prebuilt versrion of Pytorch 2.3.0 (which we can download from https://pytorch.org/get-started/locally/) and python 3.9 to virtual environment.