It is possible that address and personal information ends up in the comment field of the raw data we receive. It seems prudent to stop publishing this field on Github.
The opmerkingen fields are used in the mapping. So, we do actually need them for now. I see two options to handle this:
Encrypt and decrypt these columns based on a Github Secret key
Load the raw data into the environment, and don't write this column to data/raw, the mapping can then use the object from the environment instead of data raw.
Option 2 is easier to implement, but will permanently break the possibility to recreate past mappings. As the fields needed to recreate these occurrence files, will not be stored.
The opmerkingen fields are used in the mapping. So, we do actually need them for now. I see two options to handle this:
data/raw
, the mapping can then use the object from the environment instead of data raw.Option 2 is easier to implement, but will permanently break the possibility to recreate past mappings. As the fields needed to recreate these occurrence files, will not be stored.