Spyderisk / system-modeller

Spyderisk web service and web client
Other
3 stars 4 forks source link

Risk assessment reporting #133

Open kenmeacham opened 6 months ago

kenmeacham commented 6 months ago

This issue outlines the requirements for risk assessment reporting in Spyderisk/SSM, for various projects.

@sjt1970 has created a background technote, which is available in "T14 SSM Output Format & Reporting" folder in Sharepoint.

Some related issues already exist in Github:

2: Fix the Risk Treatment Plan (Open)

36: Hide the risk treatment plan button (Closed)

36 states:

“We have agreed that we need to investigate reporting requirements across multiple projects (CyberKit4SME, Nemecys, SYNTHEMA) and potentially create a flexible reporting framework.”

Current status in SSM GUI is that the existing Risk Treatment Plan button has been hidden, so the option is not currently available, while we develop new ideas for displaying risk assessments (see above).

Generally, it seems that we require a display of system model Consequences in a table, similar to Table 1 in T14: image

Each row in the table represents a Consequence. The column headings depend on the standard being used (e.g. ISO 14971 or ISO 27001); we would presumably have either a way to configure this statically in an SSM deployment, or some form of selection in the UI. We will need mappings from SSM terms to corresponding terms in each standard used.

Grey boxes represent parts of the Risk Assessment, where initial risk levels are displayed.

Blue boxes represent Risk Controls, i.e. the situation after certain security controls have been applied; this results in Residual Risk values, etc.

For the two main sections (Risk Assessment (grey) vs Risk Controls (blue)), we will require data from two different system models (i.e. before/”baseline” and after models), so we will need a way of querying for this data from these models to produce the data required for the UI. It was discussed that this comparison might somehow (also) be done via the SSM Dashboard (currently the reports are only generated in the SSM Modeller page).

For the Control Strategies (CSGs) and related Control Sets most involved in reducing the risks, we agreed that this data is hard to determine, as it is likely that multiple CSGs/CS are involved in each Consequence, and so the data displayed might show many entries and often be very similar for multiple Consequences. For this reason we have decided for now to show a separate section in the report for ALL CGSs and CS involved in the residual risks (N.B. this may be subject to change).

In due course we will try to improve the data shown for the user, to provide more specific details on what controls have most influenced the results.

Timescales: Initial results are required for a deliverable for Nemecys in June. This should show good progress, however this is a first deliverable, so initially we agreed that the basic elements of the table should be there (grey and blue boxes), and a separate list of applied controls. More advanced features can be further developed for later on in the project(s).

CC @sjt1970, @scp93ch, @danshearer, @mike1813

scp93ch commented 6 months ago

Aside from outputting data, there are two technical issues involved in getting the data:

  1. Being able to compare two states of the same model (with different Control Set states).
  2. Assigning "credit" to CSGs for bringing the risk level of a Consequence down to where it currently is.

Regarding (1) we had a similar issue in the prototype ISO 27001 reporting from a few years ago. This was why we introduced the traffic lights for the Control Sets. The only influence of the amber light was so that in the Statement of Applicability we could show that the implementation of the Control Set was still "to do".

Under the hood, the amber light causes the same predicate ("isProposed") to be inserted into the asserted graph (indicating that the Control Set should be taken into account in the risk analysis) but also a second predicate ("isWorkInProgress") is added.

We have two choices to hold the before and after states: (a) use separate system models; (b) use a mechanism similar to the traffic lights to indicate within a single system model what the "before" and "after" states are.

If we use two separate system models then we would need to be able to link them explicitly or have the user select them for reporting. Having them separate means you could also potentially incorporate changes in the system model asserted structure into the before and after states.

The traffic lights as they are cannot hold the necessary data as, in comparing a before and after state, we need to be able to store that a Control Set is activated AND that a Control Set is deactivated (as well as active in both, inactive in both). That's 4 states and we only have 3 lights/options. The missing state that we cannot indicate is that going from the before to the after state, a Control Set is deactivated.

mike1813 commented 6 months ago

Is this the time to grab the bull by the horns, where the bull is the system model graph structure.

At present, a system model comprises three graphs, with the following prefixes

The risk calculator must delete results from a previous risk calculation when it starts up, and this has to be done by finding and removing the properties added by the risk calculator (last three sub-bullets under the 'inferred' graph.

We could move the risk calculator outputs to a different graph, so the risk calculator just has to drop that graph when repeating the calculation. If we did that, it may make sense to use four different graphs to store current vs future risks, and for risks with or without 'isWorkInProgress' controls. Clients could then select which graph to request when retrieving information via the API, and we could add methods to run comparisons between graphs.

Among those clients are two algorithms and a web u/i embedded in system-modeller itself.

As far as I know, these don't store anything permanently in the triple store, but the CSG recommender does make temporary changes to control selection status properties. Both would need to be modified so they refer to the relevant graphs if they were made separate graphs from the inferred graph produced by the validator.

scp93ch commented 6 months ago

It's worth considering as adding additional graphs could certainly simplify some things. Just refactoring what we have so that the risk calculation result is stored in a separate graph sounds quite useful in that it makes it easier to drop the data and recalculate.

One point though is that with the current vs future risks it is common that there are some TWA level changes from one calculation to the other. The point being that storing a current and a future risk calculation result separately may not help much as the calculations are done on different models.

Storing a before and after situation in separate graphs may help with compiling a report, but we would need to add a mechanism for indicating that a Control Set is deactivated going from before to after (as mentioned). This would also limit the reporting to a comparison of models where the only difference is the active Control Sets. That goes a long way, but we know that sometimes to fix a problem the assets and/or asset links (the "model structure") need to be changed so we'd have to accept that we weren't going to do that.

If we accept we are not going to make comparisons across models wit different model structure then having the before and after risk calculations stored simultaneously in the triplestore would be neat. The other alternative is to run one risk calculation, query and store the data needed in the report in Java objects in memory, run the other risk calculation and then compile the report based on the data in the triplestore and in the Java objects. That's more clunky.

scp93ch commented 5 months ago

Feedback from @sjt1970 following NEMECYS meeting 20224-02-14.

Proposed output format:

image

Challenges:

Proposed:

image

Is the situation on the last slide acceptable? That is, a holistic perspective of whole set of controls rather than individual controls addressing specific risks.

Response from the user partners was that they really need to see which controls applied to which risk, so the above is not really OK but it is possible to have many control sets addressing a risk. This could be at the limit many control sets mapping to one risk and only small differences between the control sets applied to different risks.

The other point was about risk identification vs risk control:

This was broadly accepted but the key point was that the risk identification section is about showing the assumptions present, i.e. it is important to show the controls that are assumed to be applied before risk identification is performed.

scp93ch commented 5 months ago

On the first point, we have the statement that partners "really need to see which controls applied to which risk" but it would be good to know why. Any ideas @sjt1970?

The second point ("risk identification" and "risk control") is a helpful steer. It's not clear to me what "add the medical device" means in terms of the system model. For instance, are we imagining these steps:

  1. Build a system model with hosts, networks, data, etc and make it secure (low/very low residual risk)
  2. Add a medical device asset to the system model and connect it up
  3. Validate and run the risk calculation -> baseline system model
  4. Add controls to bring risk down to an acceptable level

Then the report (grey part) would include just the higher risk threats introduced by adding the device (those in 3) and the controls (blue part) would be just the controls added in (4)?

Could (2) involve adding more than one asset? Where is the boundary between the previously "controlled" part and the new part?

What I've written above doesn't actually fit with the final sentence in the comment above: "the key point was that the risk identification section is about showing the assumptions present, i.e. it is important to show the controls that are assumed to be applied before risk identification is performed"

I don't understand that point. How can the risk identification section show the controls that are assumed to be applied before risk identification is performed? The risk identification section shows no controls.

scp93ch commented 5 months ago

Broader point on reporting

Although this issue up to now has been mostly discussing generic risk assessment reporting, one of the targets for risk assessment reporting is data protection impact assessments (DPIA), required in SYNTHEMA. Data-flow diagrams are useful in DPIAs and should be included in any report. The concept of "data" is a domain-model specific thing. Our current (unwritten) strategy is for the system-modeller and system-modeller UI to be agnostic to the domain model, so adding in a reporting option that created a data-flow diagram would be contrary to this strategy.

Options:

sjt1970 commented 5 months ago

I talked to some of the NEMECYS end users. I suggested that we could have e.g. a separate model with the assumed basic controls applied but their perspective was that we should start simple, so the grey risk identification should (at least for the first iteration) show the completely uncontrolled situation. We can implement this and then they can test it to see if there are too many risks identified or if there are risks that are nothing to do with the MD under test. So for the initial situation and for June delivery, I think it is OK to have this.

danshearer commented 1 month ago

Is this exploratory risk reporting repo by @scp93ch addressing this issue?

scp93ch commented 1 month ago

Is this exploratory risk reporting repo by @scp93ch addressing this issue?

Yes. That's a standalone prototype to help work out how to do it. Later it needs to be integrated into the Java service.