Bug: we do not have a consistent way to describe security classifications and PII

[MatMoore]

Currently the way we are storing security classifications is inconsistent, so the data-platform-catalogue library is slightly broken.

This will prevent us from being able to distinguish sensitive datasets, and doing something useful with the classification filter.

It is also likely to cause confusion when we start looking at automatically detecting PII.

Problems

• We've written a DatahubClient class can ingest metadata programatically, but this doesn't allow you to set OFFICIAL-SENSITIVE
• The above causes a bug in the get_table_details method, where the classification is reset to the default OFFICIAL, even if it's OFFICIAL-SENSITIVE in datahub
• The SecurityClassification enum doesn't align with what we want to use it for
• OFFICIAL-SENSITIVE contains a hyphen so can't be used as a python identifier

Missing code in get_table_details

We are missing something like this when return TableMetadata objects:

            data_sensitivity_level = custom_properties.get(
                "sensitivityLevel", SecurityClassification.OFFICIAL.name
            )

            try:
                data_sensitivity_level = SecurityClassification[data_sensitivity_level]
            except KeyError:
                logger.error(
                    f"Ignoring unknown classification: {data_sensitivity_level}"
                )
                data_sensitivity_level = SecurityClassification.OFFICIAL

Inconsistencies

• We have a custom property named sensitivityLevel, which we display to the user as "Security Classification"
• In the python library, we have a SecurityClassification enum, that includes OFFICIAL, SECRET, TOP_SECRET - this is pointless as we will only ever catalogue OFFICIAL information
• We have metadata tagged with OFFICIAL-SENSITIVE
• The government security classification policy describes both markings (-SENSITIVE), handling instructions (-FOR-PUBLIC-RELEASE), and descriptors (-PERSONAL-DATA) whereas we just have the one property at the moment

Proposal

1. Make sure the "Security classification" filter still meets an identified user need (currently it does nothing useful, as it just has one checkbox for OFFICIAL)
2. Confirm the list of values we actually want to use here (e.g. OFFICIAL vs OFFICIAL-SENSITIVE?)
3. Update the python library to be able to represent all these values, and consistently return the value from get_table_details
4. Either rename the sensitivityLevel property to securityClassification or introduce new tags (TBD)
5. Make sure we can still distinguish metadata that has been automatically detected as having PII vs marked as having PII or being OFFICIAL-SENSITIVE

[jemnery]

Can we decorate a security classification enum with descriptions to get over the hyphen issue? Not Python example 😄

public enum SecurityClassification {
  [Description("OFFICIAL-SENSITIVE")]
  OfficialSensitive,

  [Description("OFFICIAL")]
  Official
}

[MatMoore]

We can use whatever we want for the enum value, we would just need to change the code to use the value rather than the name https://docs.python.org/3/howto/enum.html#using-a-descriptive-string

[MatMoore]

This is not currently an issue as we've removed this field for now. Can be revisited later if we have a need for recording markings like official-sensitive etc when manually registering data to the catalogue