Audit trail

[madelondohmen]

This is an epic ticket

About this epic

Detailed description

Currently within OpenKAT, the "Who did it?" piece is still missing. For example:

• Who added a new user
• Who changed the clearance level

That's why we want a structured log, which people can then filter into. We bring out the JSON, then people can use it for all purposes.

But we also want to process some information in OpenKAT itself, to be able to use this in the future to display in the frontend. Therefore, we store some data from the structured log. (See "Implementation" section for how we can do this.)

Feature benefit/User story

As a user, I want to see who made changes in the organization so that I can detect abuse.

Implementation

Possible solution

There are two paths:

1. Spit out structured JSON when an operation is performed
2. Store in our own data models and possibly make it visible in our frontend

Storing the data

• Where do we store the data? Clearances and OOIs can be given a "user" column in XTDB, in which the "owner" of these objects is put. All objects then belong to a particular user at a particular time. Scheduler jobs and the KATalogus (plugins) must also be linked to a user. So this has to be done in a new database table in the KATalogus. (Actually, you would want the KATalogus to also live in XTDB instead of Postgress. Then you could easily give the owner along with it as well). It is not crazy to write this audit-loggin data to different places. After all, some of the data is functional data (user functionality) and some is not.

  • What data do we store? You may not store much data in the function of logs, but you may store it in the function of functionality.

• How do we store data? UserIDs could be encrypted/hashed.

Excessive logs/alerts.

If you start looking for abuse in data and you link it to names, then you have to deal with the works council, with GDPR, etc.

This is where Kelvin comes in (we're not going to run Elastricsearch, just Kelvin). That looks to see if any outrageous things are happening to a user. In that context, a tool such as Kelvin can automatically detect that something excessive is happening. No person is directly linked to this. But depending on the behavior and risk, you can then pass along a hashed/encrypted userID. Collecting audit logs is a duty, viewing them is a duty, but only if there is a reason. You may also only keep the data for a certain amount of time.

Steps

We need to:

• Invent schema that defines logging (what do you show)
• Write Python library that shows structured logging (JSON)
• Embed and use the library

Scope is now the structured JSON side. But we now need to try to get a picture of what we all want to make insightful.

First logical step:

• Make up schema
• List event codes
• Write Python library that spits out JSON

Alternatives considered

Storing it centrally, but we can't, because the KATalogus is not in XTDB.

Related tickets

• [ ] #3043
• [ ] #3044

[underdarknl]

We need to investigate which events we want to add the our Audit Log.

[underdarknl]

We need to create a Concise way of reporting about events based on the use cases of our users.

[underdarknl]

We need to create a Library of method that we can include in the various locations / services that trigger Auditable Events.