API for retrieving the forms that the user has yet to complete

[alextreme]

Thema / Theme

API

Omschrijving / Description

Dimpact / @robwagelaar has requested a functionality to show in OIP which not-yet-completed forms a user can continue with, as an expansion of the MijnAanvragen/MijnZaken module.

In cooperation with Joeri I wrote up the following API spec, this to allow external systems to make use of the OpenFormulieren or eSuite API to determine which forms a (with DigiD logged-in) user can continue with:

https://redocly.github.io/redoc/?url=https://gist.githubusercontent.com/alextreme/61dbd2605ea7f714744d6a36f786dd8e/raw/209538fb28617c0ae3d9ab2a9046d9876260e0e1/FormulierenAPI&nocors

https://taiga.maykinmedia.nl/project/open-inwoner/us/975?milestone=445

Atos has developed a proof-of-concept API and an initial integration is being worked on in OIP. I'd like to request the same API to become available from Open Formulieren.

Added value / Toegevoegde waarde

Het tonen van de openstaande formulieren waar de inwoner met Open Formulieren is gestart in de Mijn-omgeving

Aanvullende opmerkingen / Additional context

Eerste OF PoC mag ontwikkeld worden door @Bartvaderkin , na Joeri's vakantie kunnen we de integratie verder bespreken.

[sergei-maertens]

See also #2301 and #2785

[sergei-maertens]

Adding prio medium since we need to properly design the encryption/hashing of PII so they can still be queried.

[sergei-maertens]

Current thoughts is to toggle between two modes: encrypted data (for things that are non-finalized) and hashing (for finalized) submissions.

• Completed submissions -> hash - then you only need to be able to validate given a BSN/KVK that this indeed belonged to them and that's possible because the algorithm, iteration count and salt are recorded with the hash. Given the same output and parameters, the same hash output is produced.
• Started, in progress, suspended and abandoned (timed out sessions) submissions should encrypt the data to make it a possible data leak less catastrophic. This also keeps the data in a format that can still be sufficiently efficient queried.

Constraints for encryption

• Needs to be transparent to the ORM - whenever a record is loaded from the database, the field needs to perform the decryption so that the raw values can be used in memory (relevant for template engine context, sending data to external systems etc.)
• Encryption should happen automatically without active thought by the developer (extends previous bullet)
• Encryption keys must support rotation - a datastructure (in the settings, via .env...) keeps track of key ID (unique) and private key to use. This requires a management command to be able to rotate keys when new keys are deployed (decrypt with old key, re-encrypt with new key)
• Must be possible to query -> when a BSN is given as input, the encrypted variant for every available encryption key can be calculated, which can in turn be used for a AuthAttribute.objects.filter(value__in=[encrypted_permutations]) query. If multiple BSNs need to be queried at the same time, this requires a python level check too which checks against the decrypted database value (see first bullet).
• Encryption keys should be managed at the infrastructure level, NOT in the database. Ideally they only live in memory of the machine (passed via envvar) and aren't even stored on disk anywhere. Keeping them in the database itself beats the purpose of encryption.
• Hashing should still work and the hashes themselves should not be encrypted, so whether to encrypt/decrypt also depends on the attribut_hashed boolean field value.

Resources:

• https://cryptography.io/en/latest/fernet/#cryptography.fernet.Fernet
• https://cryptography.io/en/latest/fernet/#cryptography.fernet.MultiFernet (maybe suitable depending on if we want to record key IDs with the encrypted value or just rely on the "try to decrypt" behaviour -> check impact on performance?)

[sergei-maertens]

This is blocked by #2301 - but there is no planning or prio for this at the moment.