Open nikhil-pitta opened 9 months ago
You can try overriding the postprocess_fn
in PPOTOrchPolicy. In the MARLlib, such an example may be: https://github.com/Replicable-MARL/MARLlib/blob/368c6173577d0f9c0ad70fb5b4b6afa12c864c15/marllib/marl/algos/core/CC/coma.py#L116-L125
the signature for postprocess_fn
is fixed, which is
postprocess_fn(
policy: Policy, sample_batch: SampleBatch,
other_agent_batches: Optional[Dict[AgentID, SampleBatch]] = None,
episode: Optional[MultiAgentEpisode] = None
) -> SampleBatch
As displayed in the build_policy_class
in https://github.com/ray-project/ray/blob/55fc0710d8472a9abaf244ed6567eb3b13136531/rllib/policy/policy_template.py#L40-L43
You can manipulate the SampleBatch
instance provided in the signature to add custom trajectories.
@Aequatio-Space Is this function called before or after both the policy gradient and value function gradient update in ippo or in between?
@nikhil-pitta it is called before both the policy gradient and value function gradient. The pipeline is basically: extra_action_out_fn → postprocess_fn → loss_fn → compute_gradients → apply_gradients.
@Aequatio-Space Thanks so much for the info! We would like to add counterfactual experience to DDPG before adding to the replay buffer or when pulling from the replay buffer to evaluate. Specifically, we want to augment our current step/collected experiences and add to the replay buffer for the algorithm to use. What callback function could we do to use this, and would it be the same for every offline policy? If not a callback, what would you suggest we do?
@nikhil-pitta You mentioned "augment our current step/collected experiences and add to the replay buffer", and it sounds exactly what postprocess_fn
do, as our earlier discussion. This extra function applies to all MARLlib algorithms since the interface is identical.
@Aequatio-Space Thank you for the response, and yes that function is exactly what I need! I tried using it for PPO by overriding the postprocess_fn for the IPPOTorchPolicy (https://github.com/Replicable-MARL/MARLlib/blob/master/marllib/marl/algos/core/IL/ppo.py; line 31).
I wanted to do the same thing for IQL, and saw this file https://github.com/Replicable-MARL/MARLlib/blob/master/marllib/marl/algos/core/VD/iql_vdn_qmix.py, but I was unsure how to add a postprocess_fn or any other callback because it looks like a new policy is written up within this file that doesn't accept callback functions like the PPOTorchPolicy.
Do you know what I can do to add callback functionality for IQL?
@nikhil-pitta Note that JointQPolicy
inherits the Policy
class, which has a function postprocess_trajectory
.
https://github.com/ray-project/ray/blob/55fc0710d8472a9abaf244ed6567eb3b13136531/rllib/policy/policy.py#L361-L366.
Directly overriding this function may help.
@Aequatio-Space Thanks for the response again! So now I am trying use IQL, but an issue I'm facing is that I want each of my agents to have separate policies, but I run into the exception ValueError("joint Q learning does not support individual function"), which comes from setting share_policy="individual."
This is not my desired behavior because I would like each agent to have their own policies, which is how I expected IQL to be implemented. Is there any way to get around this?
I remember joint Q learning supports share_policy=all
, you can see the related logic at here.
https://github.com/Replicable-MARL/MARLlib/blob/368c6173577d0f9c0ad70fb5b4b6afa12c864c15/marllib/marl/algos/run_vd.py#L105-L118
Try to adapt the code under this setting.
I would like to try and do some counterfactual experience replay (specifically I want to use PPO and do CER on the critic). Is there something that exposes the replay buffer or something that allows me to add custom trajectories while training somehow?
Also, how do I access the trajectories taken by the model during training?