riparias / rato-occurrences

DwC mapping of RATO vwz occurrences
MIT License
0 stars 1 forks source link

78 drop opmerkingen field from data rawcsv #87

Closed PietrH closed 10 months ago

PietrH commented 11 months ago

It is possible that address and personal information ends up in the comment field of the raw data we receive. It seems prudent to stop publishing this field on Github.

The opmerkingen fields are used in the mapping. So, we do actually need them for now. I see two options to handle this:

  1. Encrypt and decrypt these columns based on a Github Secret key
  2. Load the raw data into the environment, and don't write this column to data/raw, the mapping can then use the object from the environment instead of data raw.

Option 2 is easier to implement, but will permanently break the possibility to recreate past mappings. As the fields needed to recreate these occurrence files, will not be stored.

PietrH commented 11 months ago

Blocked by #94

PietrH commented 11 months ago

Unblocked

PietrH commented 10 months ago

Encryption is (partially?) implemented in this branch

PietrH commented 10 months ago

130 Actually encrypts the whole raw_data file instead. This is a competing solution I want to explore first.