search

cage-challenge / cage-challenge-4

The TTCP CAGE Challenges are a series of public challenges instigated to foster the development of autonomous cyber defensive agents. This CAGE Challenge 4 (CC4) returns to a defence industry enterprise environment, and introduces a Multi-Agent Reinforcement Learning (MARL) scenario.
https://cage-challenge.github.io/cage-challenge-4/
Other
41 stars 10 forks source link

Moved RLlib-specific '__all__' key from general wrappers to RLlib-specific wrapper #13

Closed rykrr closed 9 months ago

rykrr commented 9 months ago

RLlib requires the terminated / truncated dictionaries to contain an 'all' key, however, PettingZoo treats this as another agent and throws a warning. While this does not appear to affect training in any way from our tests, it is RLlib-specific, so it makes more sense to limit it to RLlib wrappers.

TrainingSB3 now runs without any warnings. On the other hand, TrainingRay is currently throwing some warnings, but I am not familiar with the library, so help would be greatly appreciated in addressing these if necessary.

Closes #12.

MitchellKiely commented 9 months ago

Could you post what the warnings are below? I feel like Ray never runs without throwing at least one warning at you but would prefer to look into it before I approve the Merge request.

rykrr commented 9 months ago

Hi Mitch,

Here are the warnings I'm getting from RLlib. The first one is a result of not calling super(MultiAgentEnv).__init__. This complication arises from the fact that EnterpriseMAE inherits from both MultiAgentEnv and BlueEnterpriseWrapper. Most, if not all, the important functions and variables required for RLlib are already defined in BlueEnterpriseWrapper. I've tried my hand at fixing this really quickly, but calling init for MultiAgentEnv followed by init for BlueEnterpriseWrapper seems to cause a lot more problems.

Turns out the rest of the warnings are just a deprecation warning and a missing optional dependency.

2024-02-29 09:41:50,360 WARNING env.py:298 -- Your MultiAgentEnv <EnterpriseMAE instance> does not have some or all of the needed base-class attributes! Make sure you call `super().__init__()` from within your MutiAgentEnv's constructor. This will raise an error in the future.
2024-02-29 09:41:50,361 WARNING deprecation.py:50 -- DeprecationWarning: `rllib/algorithms/simple_q/` has been deprecated. Use `rllib_contrib/simple_q/` instead. This will raise an error in the future!
2024-02-29 09:41:51,472 WARNING util.py:62 -- Install gputil for GPU system monitoring.

Thanks!

MitchellKiely commented 9 months ago

That looks completely fine to me then. I think our current approach is sufficient, the code works and agents can be trained which is really all that matters. I suspect others will write up their own wrappers if they really want to avoid any deprecation issues, or implement a more specific approach. Thanks for investigating this @rykrr!