Encrypt PII instead of hashing

[sergei-maertens]

Currently we hash sensitive data like BSN's when a submission is deemed finished processing (which can be when it's failed the maximum times of failure during registration).

This breaks additional attempts where manual retries are performed, but now we can't include this data anymore for registration backends.

A better approach would be encrypting this data so that we can decrypt it for additional attempts while storing it in a secure manner. As a bandaid, we can skip the hashing for failed registrations.

[sergei-maertens]

Extra info for case observed:

• The submission was paused and then resumed
• This apparently lead to the hashed values still being stored instead of the raw values to be able to register the data

openforms.submissions.views.ResumeFormMixin needs to store the non-hashed value back into the field instead of keeping the hashed value from suspension.

[sergei-maertens]

A fix in the submission resume is applied everywhere, putting this back in the backlog to change hashing to encryption.

[sergei-maertens]

Current thoughts is to toggle between two modes: encrypted data (for things that are non-finalized) and hashing (for finalized) submissions.

• Completed submissions -> hash - then you only need to be able to validate given a BSN/KVK that this indeed belonged to them and that's possible because the algorithm, iteration count and salt are recorded with the hash. Given the same output and parameters, the same hash output is produced.
• Started, in progress, suspended and abandoned (timed out sessions) submissions should encrypt the data to make it a possible data leak less catastrophic. This also keeps the data in a format that can still be sufficiently efficient queried.

Constraints for encryption

• Needs to be transparent to the ORM - whenever a record is loaded from the database, the field needs to perform the decryption so that the raw values can be used in memory (relevant for template engine context, sending data to external systems etc.)
• Encryption should happen automatically without active thought by the developer (extends previous bullet)
• Encryption keys must support rotation - a datastructure (in the settings, via .env...) keeps track of key ID (unique) and private key to use. This requires a management command to be able to rotate keys when new keys are deployed (decrypt with old key, re-encrypt with new key)
• Must be possible to query -> when a BSN is given as input, the encrypted variant for every available encryption key can be calculated, which can in turn be used for a AuthAttribute.objects.filter(value__in=[encrypted_permutations]) query. If multiple BSNs need to be queried at the same time, this requires a python level check too which checks against the decrypted database value (see first bullet).
• Encryption keys should be managed at the infrastructure level, NOT in the database. Ideally they only live in memory of the machine (passed via envvar) and aren't even stored on disk anywhere. Keeping them in the database itself beats the purpose of encryption.
• Hashing should still work and the hashes themselves should not be encrypted, so whether to encrypt/decrypt also depends on the attribut_hashed boolean field value.

Resources:

• https://cryptography.io/en/latest/fernet/#cryptography.fernet.Fernet
• https://cryptography.io/en/latest/fernet/#cryptography.fernet.MultiFernet (maybe suitable depending on if we want to record key IDs with the encrypted value or just rely on the "try to decrypt" behaviour -> check impact on performance?)

(taken from https://github.com/open-formulieren/open-forms/issues/2775#issuecomment-1478075565)

This is currently not planned, removing milestone.