Added masking_strategy_override at field level

Linker44 commented 3 weeks ago

Closes #LA-60

Description Of Changes

This pr adds masking_strategy_overrides at the dataset field-level.

Many masking strict erasure requests fail because some fields dont have the correct format. This is caused by applying the same masking to each field when some of them should remain with a certain format.

ex: an endpoint will only succeed at updating an email field if it has an email format.

          - name: email_address
            data_categories: [user.contact.email]
            fidesops_meta:
              data_type: string
              masking_strategy_override:
                strategy: random_string_rewrite
                configuration:
                  length: 10
                  format_preservation:
                    suffix: "@test.com"

Code Changes

[x] Added field-level masking strategy override.
[x] Added Testing for update_value_map method in QueryConfig.
[x] Added Testing for dsr database erasure request.
[x] Added Validation for masking strategies that require secrets.
[x] Added Testing for masking strategies that require secrets validation.
[x] Renamed class MaskingOverride to something more appropiate (MaskingTruncation).

Steps to Confirm

Confirm masking override validation on fides admin:

[x] Try to create a dataset that contains a field-level masking_strategy_override that requires secrets. It should fail to create it
[x] Try to update a dataset so that it contains a field-level masking_strategy_override that requires secrets. It should fail to create it
[x] Try to create a dataset that contains a field-level masking_strategy_override that doesn’t require secrets. It should succeed

Confirm masking override validation on startup (nox -s dev):

[x] Update one of the existing datasets so that it contains a field-level masking_strategy_override that requires secrets. It should raise a validation error on the terminal and fail to create it.
[x] Update one of the existing datasets so that it contains a field-level masking_strategy_override that doesn't requires secrets. It should succeed.

Confirm masking override is working:

[x] Alter a dsr saas dataset in a user collection field so that it contains a field_level_masking override that doesnt require secrets -> create a system with the integration. (make sure to have the enviroment variable FIDES__DEV_MODE set to true to see the request body in the terminal) -> make an erasure request -> Confirm that the request has the correct masking on the specified field.

Pre-Merge Checklist

[x] All CI Pipelines Succeeded
Documentation:
- [x] documentation complete, PR opened in fidesdocs
- [ ] documentation issue created in fidesdocs
- [ ] if there are any new client scopes created as part of the pull request, remember to update public-facing documentation that references our scope registry
[x] Issue Requirements are Met
[ ] Relevant Follow-Up Issues Created
[x] Update CHANGELOG.md
[ ] For API changes, the Postman collection has been updated
If there are any database migrations:
- [ ] Ensure that your downrev is up to date with the latest revision on main
- [ ] Ensure that your downgrade() migration is correct and works
- [ ] If a downgrade migration is not possible for this change, please call this out in the PR description!

vercel[bot] commented 3 weeks ago

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

cypress[bot] commented 3 weeks ago

fides Run #11040

Run Properties: Passed #11040 • 7d643dc386 ℹ️: Merge b31509540c43bb1e5a1727fa9796c865e264460e into 523c1ab716c666feeca4636b844c...

Project	`fides`
Branch Review	`refs/pull/5446/merge`
Run status	`Passed #11040`
Run duration	`00m 40s`
Commit	`7d643dc386 ℹ️: Merge b31509540c43bb1e5a1727fa9796c865e264460e into 523c1ab716c666feeca4636b844c...`
Committer	`Facundo Lopez Janza`
View all properties for this run ↗︎

Test results
Failures	`0`
Flaky	`0`
Pending	`0`
Skipped	`0`
Passing	`4`
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.
View all changes introduced in this branch ↗︎

codecov[bot] commented 3 weeks ago

Codecov Report

Attention: Patch coverage is 78.33333% with 13 lines in your changes missing coverage. Please review.

Project coverage is 79.16%. Comparing base (52ac330) to head (1d0382c). Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
src/fides/api/api/v1/endpoints/router_factory.py	30.00%	7 Missing :warning:
src/fides/api/service/connectors/query_config.py	78.57%	1 Missing and 2 partials :warning:
src/fides/api/util/data_category.py	81.81%	1 Missing and 1 partial :warning:
src/fides/core/api.py	0.00%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #5446 +/- ## ========================================== - Coverage 85.20% 79.16% -6.04% ========================================== Files 386 387 +1 Lines 24247 24303 +56 Branches 2644 2654 +10 ========================================== - Hits 20659 19239 -1420 - Misses 3033 4550 +1517 + Partials 555 514 -41 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

cypress[bot] commented 1 week ago

fides Run #11047

Run Properties: Passed #11047 • 861201b96c: Added masking_strategy_override at field level (#5446)

Project	`fides`
Branch Review	`main`
Run status	`Passed #11047`
Run duration	`00m 35s`
Commit	`861201b96c: Added masking_strategy_override at field level (#5446)`
Committer	`Facundo Lopez Janza`
View all properties for this run ↗︎

Test results
Failures	`0`
Flaky	`0`
Pending	`0`
Skipped	`0`
Passing	`4`
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.
View all changes introduced in this branch ↗︎

ethyca / fides