The NetSecGame (Network Security Game) is a framework for training and evaluation of AI agents in the network security tasks (both offensive and defensive). It builds a simulated local network using the CYST network simulator, adds many conditions on the environment and can train reinforcement learning (RL) algorithms on how to better attack and defend the network. Examples of implemented agents can be seen in the submodule NetSecGameAgents.
To run this code you need an environment and access to cyst code. However, the venv needs to be created for your own user
python -m venv ai-dojo-venv-<yourusername>
source ai-dojo-venv<yourusername>/bin/activate
python3 -m pip install -r requirements.txt
conda create --name aidojo python==3.10
conda activate aidojo
python3 -m pip install -r requirements.txt
The architecture of the environment can be seen here.
The NetSecGame environment has several components in the following files:
env/network_security_game.py
implements the game environmentenv/game_components.py
implements a library with objects used in the environment. See detailed explanation of the game components.utils/utils.py
is a collection of utils functions which the agents can useenv/scenarios
folder, such as env/scenarios/scenario_configuration.py
. Implements the network game's configuration of hosts, data, services, and connections. It is taken from CYST.
The scenarios define the topology of a network (number of hosts, connections, networks, services, data, users, firewall rules, etc.) while the task-configuration is to be used for definition of the exact task for the agent in one of the scenarios (with fix topology).ExploitService
action, it is expected that the agent has discovered this service before (by playing FindServices
in the target_host
before this action)Find Data
action finds all the available data in the host if successful.Find Data
action requires ownership of the target host.ExfiltrateData
requires controlling BOTH source and target hostsFind Services
can be used to discover hosts (if those have any active services)ScanNetwork
and FindServices
can be chosen arbitrarily (they don't have to be listed in known_newtworks
/known_hosts
)BlockIP
action needs its three parameters (Source host, Target host, and Blocked host) to be in the controlled list of the Agent. The defender does have the action to block an IP address in a target host.
[!NOTE]
The global defender, available in the previous environment versions, will not be supported in the future. To enable backward compatibility, the global defender functionality can be enabled by addinguse_global_defender: True
to the configuration YAML file in theenv
section. This option is disabled by default.
The actions are:
The environment should be created before starting the agents. The properties of the environment can be defined in a YAML file. The game server can be started by running:
python3 coordinator.py
When created, the environment:
When the game server is created, agents connect to it and interact with the environment. In every step of the interaction, agents submits an Action and receives Observation with next_state
, reward
, is_terminal
, end
, and info
values. Once the terminal state or timeout is reached, no more interaction is possible until the agent asks for a game reset. Each agent should extend the BaseAgent
class in agents.
The NetSecEnv is highly configurable in terms of the properties of the world, tasks, and agent interaction. Modification of the world is done in the YAML configuration file in two main areas:
env
section) controls the properties of the world (taxonomy of networks, maximum allowed steps per episode, probabilities of action success, etc.)The environment part defines the properties of the environment for the task (see the example below). In particular:
random_seed
- sets seed for any random processes in the environmentscenario
- sets the scenario (network topology) used in the task (currently, scenario1_tiny
, scenario1_small
, and scenario1
are available)max_steps
- sets the maximum number of steps an agent can make before an episode is terminatedstore_replay_buffer
- if True
, interaction of the agents is serialized and stored in a fileuse_dynamic_addresses
- if True
, the network and IP addresses defined in scenario
are randomly changed at the beginning of EVERY episode (the network topology is kept as defined in the scenario
. Relations between networks are kept, IPs inside networks are chosen at random based on the network IP and mask)use_firewall
- if True
firewall rules defined in scenario
are used when executing actions. When False
, the firewall is ignored, and all connections are allowed (Default)goal_reward
- sets reward which agent gets when it reaches the goal (default 100)detection_reward
- sets the reward that which agent gets when it is detected (default -50)step_reward
- sets reward which agent gets for every step taken (default -1)actions
- defines the probability of success for every ActionTypeenv:
random_seed: 42
scenario: 'scenario1'
max_steps: 15
use_dynamic_addresses: False
use_firewall: True
goal_reward: 100
detection_reward: -5
step_reward: -1
actions:
scan_network:
prob_success: 0.9
find_services:
prob_success: 0.9
exploit_services:
prob_success: 0.7
find_data:
prob_success: 0.8
exfiltrate_data:
prob_success: 0.8
The task configuration part (section coordinator[agents]
) defines the starting and goal position of the attacker and the type of defender that is used.
attackers
)Configuration of the attacking agents. Consists of two parts:
Goal definition (goal
) which describes the GameState
properties that must be fulfilled to award goal_reward
to the attacker:
known_networks:
(set)known_hosts
(set)controlled_hosts
(set)known_services
(dict)known_data
(dict)
Each of the parts can be empty (not part of the goal, exactly defined (e.g., known_networks: [192.168.1.0/24, 192.168.3.0/24]
) or include the keyword random
(controlled_hosts: [213.47.23.195, random]
, known_data: {213.47.23.195: [random]}
.
Additionally, if random
keyword is used in the goal definition,
randomize_goal_every_episode
. If set to True
, each keyword random
is replaced with a randomly selected, valid option at the beginning of EVERY episode. If set to False
, randomization is performed only once when the environment is
Definition of starting position (start_position
), which describes the GameState
in which the attacker starts. It consists of:
known_networks:
(set)known_hosts
(set)controlled_hosts
(set)known_services
(dict)known_data
(dict)The initial network configuration must assign at least one controlled host to the attacker in the network. Any item in controlled_hosts
is copied to known_hosts
, so there is no need to include these in both sets. known_networks
is also extended with a set of all networks accessible from the controlled_hosts
Example attacker configuration:
agents:
Attacker:
goal:
randomize_goal_every_episode: False
known_networks: []
known_hosts: []
controlled_hosts: []
known_services: {192.168.1.3: [Local system, lanman server, 10.0.19041, False], 192.168.1.4: [Other system, SMB server, 21.2.39421, False]}
known_data: {213.47.23.195: ["random"]}
known_blocks: {'all_routers': 'all_attackers'}
start_position:
known_networks: []
known_hosts: []
# The attacker must always at least control the CC if the goal is to exfiltrate there
# Example of fixing the starting point of the agent in a local host
controlled_hosts: [213.47.23.195, random]
# Services are defined as a target host where the service must be, and then a description in the form 'name, type, version, is_local'
known_services: {}
known_data: {}
known_blocks: {}
defenders
)Currently, the defender is a separate agent.
If you want a defender in the game, you must connect a defender agent. For playing without a defender, leave the section empty.
Example of defender configuration:
Defender:
goal:
description: "Block all attackers"
is_any_part_of_goal_random: False
known_networks: []
known_hosts: []
controlled_hosts: []
known_services: {}
known_data: {}
known_blocks: {'any_routers': 'all_attackers_controlled_hosts'}
start_position:
known_networks: [all_local]
known_hosts: [all_local]
controlled_hosts: [all_local]
known_services: {all_local}
known_data: {all_local}
blocked_ips: {}
known_blocks: {}
As in other agents, the description is only a text for the agent, so it can know what is supposed to do to win. In this example, the goal of the defender is determined by a state where the known blocks can be applied in any router's firewall and must include all the controlled hosts of all the attackers. These are magic
words that will push the coordinator to check these positions without revealing them to the defender.
The network topology and rules are defined using a CYST simulator configuration. Cyst defines a complex network configuration, and this environment does not use all Cyst features for now. CYST components currently used are:
In the current state, we support a single scenario: Data exfiltration to a remote C&C server.
For the data exfiltration we support 3 variants. The full scenario contains 5 clients (where the attacker can start) and 5 servers where the data that is supposed to be exfiltrated can be located. scenario1_small is a variant with a single client (the attacker always starts there) and all 5 servers. scenario1_tiny contains only a single server with data. The tiny scenario is trivial and intended only for debugging purposes.
Scenario 1 | Scenario 1 - small | Scenario 1 -tiny |
---|---|---|
The trajectory is a sequence of GameStates, Actions, and rewards in one run of a game. It contains the complete information of the actions played by the agent, the rewards observed and their effect on the state of the environment. Trajectory visualization and analysis tools are described in Trajectory analysis tools
Trajectories performed by the agents can be stored in a file using the following configuration:
env:
save_trajectories: True
[!CAUTION] Trajectory files can grow very fast. It is recommended to use this feature on evaluation/testing runs only. By default, this feature is not enabled.
Testing the environment
It is advised after every change you test if the env is running correctly by doing
tests/run_all_tests.sh
This will load and run the unit tests in the tests
folder.
The code can be adapted to new configurations of games and for new agents. See Agent repository for more details.
This code was developed at the Stratosphere Laboratory at the Czech Technical University in Prague.