search

PKU-Alignment / omnisafe

JMLR: OmniSafe is an infrastructural framework for accelerating SafeRL research.
https://www.omnisafe.ai
Apache License 2.0
939 stars 132 forks source link

BrokenProcessPool #338

Closed Eureka725 closed 5 months ago

Eureka725 commented 5 months ago

Required prerequisites

Questions

Thank you for your work! I encountered this issue during the training process. image This problem has not been encountered before, it seems that there is an issue with the code section that saves the results. How should I solve it?

Gaiejj commented 5 months ago

This error is a superficial error that occurs only during multi-process execution. Could you provide more detailed error information and the specific configuration of run_experiment_grid.py so that we can pinpoint the issue more accurately?

I run python run_experiment_grid.py with default configuration and it seems run well.

$ python run_experiment_grid.py
Save with config of experiment grid in grid_config.json
================================================================================
ExperimentGrid [Benchmark_Safety_Velocity] runs over parameters:

 env_id                                  [env] 

        "SafetyAntVelocity-v1"
        "SafetyHopperVelocity-v1"
        "SafetyHumanoidVelocity-v1"
        "SafetyWalker2dVelocity-v1"
        "SafetyHalfCheetahVelocity-v1"
        "SafetySwimmerVelocity-v1"

 algo                                    [alg] 

        "CPO"
        "PCPO"

 logger_cfgs:use_wandb                   [log-use] 

        false

 train_cfgs:vector_env_nums              [tra-vec] 

        4

 train_cfgs:torch_threads                [tra-tor] 

        1

 algo_cfgs:steps_per_epoch               [alg-ste] 

        20000

 train_cfgs:total_steps                  [tra-tot] 

        100000

 seed                                    [see] 

        0

 Variants, counting seeds:               12
 Variants, not counting seeds:           12

================================================================================
value {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'CPO', 'env_id': 'SafetyAntVelocity-v1'} key env_id
value {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'CPO', 'env_id': 'SafetyAntVelocity-v1'} key algo
value {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'PCPO', 'env_id': 'SafetyAntVelocity-v1'} key env_id
value {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'PCPO', 'env_id': 'SafetyAntVelocity-v1'} key algo
value {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'CPO', 'env_id': 'SafetyHopperVelocity-v1'} key env_id
value {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'CPO', 'env_id': 'SafetyHopperVelocity-v1'} key algo
value {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'PCPO', 'env_id': 'SafetyHopperVelocity-v1'} key env_id
value {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'PCPO', 'env_id': 'SafetyHopperVelocity-v1'} key algo
value {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'CPO', 'env_id': 'SafetyHumanoidVelocity-v1'} key env_id
value {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'CPO', 'env_id': 'SafetyHumanoidVelocity-v1'} key algo
value {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'PCPO', 'env_id': 'SafetyHumanoidVelocity-v1'} key env_id
value {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'PCPO', 'env_id': 'SafetyHumanoidVelocity-v1'} key algo
value {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'CPO', 'env_id': 'SafetyWalker2dVelocity-v1'} key env_id
value {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'CPO', 'env_id': 'SafetyWalker2dVelocity-v1'} key algo
value {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'PCPO', 'env_id': 'SafetyWalker2dVelocity-v1'} key env_id
value {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'PCPO', 'env_id': 'SafetyWalker2dVelocity-v1'} key algo
value {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'CPO', 'env_id': 'SafetyHalfCheetahVelocity-v1'} key env_id
value {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'CPO', 'env_id': 'SafetyHalfCheetahVelocity-v1'} key algo
value {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'PCPO', 'env_id': 'SafetyHalfCheetahVelocity-v1'} key env_id
value {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'PCPO', 'env_id': 'SafetyHalfCheetahVelocity-v1'} key algo
value {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'CPO', 'env_id': 'SafetySwimmerVelocity-v1'} key env_id
value {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'CPO', 'env_id': 'SafetySwimmerVelocity-v1'} key algo
value {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'PCPO', 'env_id': 'SafetySwimmerVelocity-v1'} key env_id
value {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'PCPO', 'env_id': 'SafetySwimmerVelocity-v1'} key algo

Preparing to run the following experiments...

Benchmark_Safety_Velocity_envsafetyantvelocity-v1_algcpo
Benchmark_Safety_Velocity_envsafetyantvelocity-v1_algpcpo
Benchmark_Safety_Velocity_envsafetyhalfcheetahvelocity-v1_algcpo
Benchmark_Safety_Velocity_envsafetyhalfcheetahvelocity-v1_algpcpo
Benchmark_Safety_Velocity_envsafetyhoppervelocity-v1_algcpo
Benchmark_Safety_Velocity_envsafetyhoppervelocity-v1_algpcpo
Benchmark_Safety_Velocity_envsafetyhumanoidvelocity-v1_algcpo
Benchmark_Safety_Velocity_envsafetyhumanoidvelocity-v1_algpcpo
Benchmark_Safety_Velocity_envsafetyswimmervelocity-v1_algcpo
Benchmark_Safety_Velocity_envsafetyswimmervelocity-v1_algpcpo
Benchmark_Safety_Velocity_envsafetywalker2dvelocity-v1_algcpo
Benchmark_Safety_Velocity_envsafetywalker2dvelocity-v1_algpcpo

================================================================================
current_config {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'CPO', 'env_id': 'SafetyAntVelocity-v1'}
current_config {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'PCPO', 'env_id': 'SafetyAntVelocity-v1'}
current_config {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'CPO', 'env_id': 'SafetyHopperVelocity-v1'}
current_config {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'PCPO', 'env_id': 'SafetyHopperVelocity-v1'}
current_config {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'CPO', 'env_id': 'SafetyHumanoidVelocity-v1'}
current_config {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'PCPO', 'env_id': 'SafetyHumanoidVelocity-v1'}
current_config {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'CPO', 'env_id': 'SafetyWalker2dVelocity-v1'}
current_config {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'PCPO', 'env_id': 'SafetyWalker2dVelocity-v1'}
current_config {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'CPO', 'env_id': 'SafetyHalfCheetahVelocity-v1'}
current_config {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'PCPO', 'env_id': 'SafetyHalfCheetahVelocity-v1'}
current_config {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'CPO', 'env_id': 'SafetySwimmerVelocity-v1'}
current_config {'seed': 0, 'train_cfgs': {'total_steps': 100000, 'torch_threads': 1, 'vector_env_nums': 4}, 'algo_cfgs': {'steps_per_epoch': 20000}, 'logger_cfgs': {'use_wandb': False}, 'algo': 'PCPO', 'env_id': 'SafetySwimmerVelocity-v1'}
exp-x: 0 is training...
exp-x: 1 is training...
exp-x: 2 is training...
exp-x: 3 is training...
exp-x: 4 is training...
exp-x: 5 is training...
exp-x: 6 is training...
exp-x: 7 is training...
exp-x: 8 is training...
exp-x: 9 is training...
exp-x: 10 is training...
exp-x: 11 is training...

==================================================

./exp-x/Benchmark_Safety_Velocity/SafetyAntVelocity-v1---3c29d8453d7a7432fc1a73b27205c378529fdc3dbfb81751c9e41881c044d6ad
./exp-x/Benchmark_Safety_Velocity/SafetyHopperVelocity-v1---3e3ade4d521ed3a6ab1e92b32bbb748c6459ea09ca3c0ca1de98cb45106f9b44
./exp-x/Benchmark_Safety_Velocity/SafetyHumanoidVelocity-v1---e8f8e5aa7fb582bc40d6172f5b8459a6679c4c3a2c007c545e888e548cb040d2
./exp-x/Benchmark_Safety_Velocity/SafetyWalker2dVelocity-v1---49c17ac672dc7920aae918adf0d3655d3e4fbbe13ab47b377d91587c0da3f342
./exp-x/Benchmark_Safety_Velocity/SafetyHalfCheetahVelocity-v1---ea61340f6e75ae6b14c2bdecb5c08dd8f758cb2b0bb9db1e727ae7e6b05652e7
./exp-x/Benchmark_Safety_Velocity/SafetySwimmerVelocity-v1---6c3cbce769f28be4f12ab2955f61d8cbce0eddc079a33b0de0fdebecef46720a

==================================================
Plotting from...
==================================================

./exp-x/Benchmark_Safety_Velocity/SafetyAntVelocity-v1---57f9dc09d400ed8d0d75e2edaee55b783c4175579e9e9e0a7b53c981235becce
./exp-x/Benchmark_Safety_Velocity/SafetyHopperVelocity-v1---cb49c82da88debef98514e6c10d4c9e66c0aacffa8dd2a06ae5508c1ad7f31b9
./exp-x/Benchmark_Safety_Velocity/SafetyHumanoidVelocity-v1---fca9054131fe5f5396e047f870664c1775766f1892f3b3c480c406a419c1ff99
./exp-x/Benchmark_Safety_Velocity/SafetyWalker2dVelocity-v1---c31512cd6714b24b4efab7d7e10593e6df318550d31f4385bb0c7dddd0412e90
./exp-x/Benchmark_Safety_Velocity/SafetyHalfCheetahVelocity-v1---e94596568d033295de4c517ffa170e1de4f9bacd624d331e530e7e9aff5a23f5
./exp-x/Benchmark_Safety_Velocity/SafetySwimmerVelocity-v1---f3561764de78ccf2aed74ba8da430e52cbac76a62cf19ece09388fa4a33147df

==================================================
Eureka725 commented 5 months ago

` import warnings

import torch

from omnisafe.common.experiment_grid import ExperimentGrid from omnisafe.utils.exp_grid_tools import train

if name == 'main': eg = ExperimentGrid(exp_name='On-Policy-Benchmarks')

# set up the algorithms.
#base_policy = ['PolicyGradient', 'NaturalPG', 'TRPO', 'PPO']
#naive_lagrange_policy = ['PPOLag', 'TRPOLag', 'RCPO']
first_order_policy = ['P3O']
#second_order_policy = ['CPO']
#saute_policy = ['PPOSaute', 'TRPOSaute']
#simmer_policy = ['PPOSimmerPID', 'TRPOSimmerPID']
#pid_policy = ['CPPOPID', 'TRPOPID']
#early_mdp_policy = ['PPOEarlyTerminated', 'TRPOEarlyTerminated']

eg.add('algo',first_order_policy )
eg.add('logger_cfgs:use_wandb', [False])
eg.add('logger_cfgs:use_tensorboard', [True])
avaliable_gpus = list(range(torch.cuda.device_count()))
gpu_id = None
eg.add('env_id', ['SafetyCarGoal2-v0'])
eg.add('seed', [0])
eg.run(train, num_pool=1, gpu_id=gpu_id)`

This is the code for the run_experience_grid.py section.This thread error problem does not occur immediately after the code starts executing, but after a few or dozens of steps.