Safety actor-critic

[dtch1997]

Baseline PPO agent:

• Critic represents total reward
• Actor is trained to maximize critic

CBF PPO agent:

• Base critic represents nominal reward
• CBF critic represents safety reward
• Actor is trained to maximize base critic + lambda * CBF critic

Remarks

• If lambda is set large enough, then the combined reward function defines a SVF, so optimal actor is guaranteed to be safe.
• Additionally, because the CBF critic is trained on the 0/1 safety reward, it has an analytic form, so we can apply our tricks / metrics.

Experiments

• Baseline PPO
• PPO + penalty
• PPO + penalty + CBF tricks

[dtch1997]

Concrete tasks

• [ ] Define a safety reward; 1 if all previous states are safe, 0 otherwise.
  • Track this in the info
• [ ] Implement separate safety reward tracking in PPObuffer
• [ ] Implement multi-critic actor-critic architecture
• [ ] Modify training loop to also train the second critic