Implement false positives/negatives in the Simulator

[mrkickling]

We want more realistic observations in the MAL Simulator to train the ML agent against a policy/agent more similar to the TYR demo.

If users can state a false positive / negative rate on some attack steps, that could be enough.

The defined false positive/negative rate should affect the observations. If a step is active in the attack graph and it has a false negative rate of 0.1, it should be a 90% chance that it shows up in the observation as an active step.

If a step is inactive in the attack graph and has a false positive rate of 0.1, it should be a 10% chance that it shows up in the observation as an active step.

Potentially: have the option to give probability distributions

Additionally: Should it work differently for cumulative and non-cumulative observations?

[mrkickling]

Any input on this issue @kasanari ?

[kasanari]

The way you describe it is basically how I implemented it in my old simulator, so I think it is sensible: https://github.com/KTH-SSAS/attack-simulator/blob/ed01cbd9031b94adb1608ec27e7bebbea3a5de73/src/defender_state.rs#L62

I think one could use the same setup with non-cumulative observations.

[kasanari]

One point to bear in mind is whether we consider "true negatives". I.e. does a node not being activated mean that we are certain that it has not happened, or does it mean that it is unknown? A negative, true or false, means that we get a report that an attack step has not happened, which does not really happen in an IDS. You only get positives, true and false, about things happening. If we are not considering negatives, then we would only need a TPR/FPR rate, and anything not observed would be considered as unknown.

[mrkickling]

Good points. But can you not see it like this?:

The IDS giving alert for an attack step can mean two things:

• A false positive
• A true positive

The IDS not giving an alert for an attack step can mean two things:

• A true negative
• A false negative

[mrkickling]

Your implementation looks a lot like how I thought of it, cool.