Add a pre-commit check for secrets

ThomasLaPiana commented 2 years ago

Is your feature request related to a specific problem?

We want to avoid accidentally committing secrets to the repo

Describe the solution you'd like

Add a pre-commit task that scans for secrets. We should additionally leverage the noxfile to verify that developers have this enabled.

Describe alternatives you've considered, if any

Not sure what other options we have besides pre-commits here, or what might be better

sanders41 commented 1 year ago

I did some looking into this a while back and this using pre-commit was the best option I found. It could potentially take some customizing the hook to match our exact secrets, but this was true of everything I found.

daveqnet commented 1 year ago

Here are the tools that I turned up when looking into this a few months ago:

trufflehog (OSS)
gitleaks (OSS)
gitguardian / ggshield (commercial, but free for OSS projects)
yelp/detect-secrets (OSS)

I like GitGuardian, if only because their documentation is good e.g. https://blog.gitguardian.com/setting-up-a-pre-commit-git-hook-with-gitguardian-shield-to-scan-for-secrets/

Also, as a commercial tool, GitGuardian might be better at dealing with false positives and false positive noise is probably going to be our biggest challenge in introducing this (it's the biggest challenge with any appsec tool). Devs just learn to ignore noise, which means they can also unfortunately learn to ignore true positives.

Some other thoughts:

I think secrets detection hooks should be executed pre-push, not pre-commit.
With the exception of trufflehog all of the above suggestions are listed on https://pre-commit.com/hooks.html
@PSalant726 has some strong opinions on native git hooks vs python scripts (he'll be able to explain better than I can).

sanders41 commented 1 year ago

I saw gitguardian but didn’t realize they had a free oss version. It is worth considering also with that.

Is there a reason you would do pre-push? At that point the secret is committed so you would have to back out all the your you have done to remove it. pre-commit would stop it from ever getting committed.

The pre-commit framework uses native git hooks, it just makes them easy to setup and maintain.

daveqnet commented 1 year ago

Ah, good insight, Paul, I didn't consider that about pre-commit vs pre-push. My thinking was that pre-push would be less intrusive for devs while still stopping the publication of secrets to the remote repo, but if re-writing local git history would be more of a problem for devs, then that's worth thinking about for sure.

From my perspective, we just need to guard against secrets being pushed to remote. If that guardrail is present pre-commit rather than pre-push, that still works for me.

PSalant726 commented 1 year ago

Here are the tools that I turned up when looking into this a few months ago:

trufflehog (OSS)

gitleaks (OSS)

gitguardian / ggshield (commercial, but free for OSS projects)

yelp/detect-secrets (OSS)

Gitleaks is the one that I happen to be the most familiar with. It works well, but I don't feel strongly about the tool that we integrate.

I like GitGuardian, if only because their documentation is good e.g. blog.gitguardian.com/setting-up-a-pre-commit-git-hook-with-gitguardian-shield-to-scan-for-secrets

If it's only free for OSS, then keep in mind that we also have private repos where we'll be developing connectors and other tools that integrate 3rd party APIs. Ideally we don't need to integrate multiple tools.

I think secrets detection hooks should be executed pre-push, not pre-commit.

I agree with Paul - a pre-commit hook is where we should enforce things. It's true that it would be less "noisy" on pre-push, but if you only learn about your mistake after it's already made, then you have the cognitive load of dealing with cleanup. The pre-commit hook is intended to be used to prevent those mistakes from being made at all, and the check should pass for 99% of commits anyway.

@PSalant726 has some strong opinions on native git hooks vs python scripts (he'll be able to explain better than I can).

The pre-commit framework uses native git hooks, it just makes them easy to setup and maintain.

I wouldn't call my opinion on this strong, but I prefer to avoid adding dependencies whenever possible. I've never had an issue setting up native git hooks, so the dependency feels unnecessary here.

daveqnet-alt commented 1 year ago

I did some very basic local performance testing of the three OSS tools on a personal fork of the fides repo. Results below, but I think trufflehog can be discarded as an option immediately as it took a few minutes to complete. gitleaks and detect-secrets were both very fast, a few seconds each.

I installed pre-commit using homebrew and deleted the existing content of .pre-commit-config.yaml before running any of these tests.

1. trufflehog

~/projects/github/ethyca/fides-fork main ❯ cat .pre-commit-config.yaml 
minimum_pre_commit_version: "2"

repos:
- repo: local
  hooks:
    - id: trufflehog
      name: TruffleHog
      description: Detect secrets in your data.
      entry: bash -c 'trufflehog git file://. --only-verified --fail'
      language: system
      stages: ["commit", "push"]%                                                                                                                                                 
~/projects/github/ethyca/fides-fork main ❯ touch testfile
~/projects/github/ethyca/fides-fork main ?1 ❯ git add .
~/projects/github/ethyca/fides-fork main +1 ❯ git commit -m "commiting a blank file" 
TruffleHog...............................................................Passed
[main 489b6922] commiting a blank file
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 testfile

The test took 2m 32s to complete and redlined my CPU. A sample top output:

Processes: 583 total, 5 running, 578 sleeping, 3358 threads                                                                                                               21:30:52
Load Avg: 5.78, 2.49, 2.04  CPU usage: 88.27% user, 2.27% sys, 9.44% idle  SharedLibs: 434M resident, 82M data, 24M linkedit.
MemRegions: 205989 total, 3271M resident, 284M private, 2624M shared. PhysMem: 15G used (1963M wired), 133M unused.
VM: 226T vsize, 3831M framework vsize, 0(0) swapins, 0(0) swapouts. Networks: packets: 143882288/146G in, 65122718/31G out. Disks: 12461290/237G read, 14202822/288G written.

PID    COMMAND      %CPU  TIME     #TH   #WQ  #PORT MEM    PURG   CMPRS PGRP  PPID  STATE    BOOSTS          %CPU_ME %CPU_OTHRS UID  FAULTS    COW   MSGSENT     MSGRECV   
372    trufflehog   689.4 04:41.66 16/7  2    56    98M-   0B     0B    355   371   running  *0[1]           0.00000 0.00000    501  14455     705   123         20

2. gitleaks

~/projects/github/ethyca/fides-fork main !1 ❯ cat .pre-commit-config.yaml
minimum_pre_commit_version: "2"

repos:
  - repo: https://github.com/zricethezav/gitleaks
    rev: v8.15.0
    hooks:
      - id: gitleaks
~/projects/github/ethyca/fides-fork main !1 ❯ touch testfile
~/projects/github/ethyca/fides-fork main !1 ?1 ❯ git add .
~/projects/github/ethyca/fides-fork main +2 ❯ git commit -m "testing gitleaks pre-commit hook"
Detect hardcoded secrets.................................................Passed
[main 58060c7e] testing gitleaks pre-commit hook
 2 files changed, 3 insertions(+), 35 deletions(-)
 create mode 100644 testfile

The test took about 1 second. It requires a local installation of go to work.

3. detect-secrets

~/projects/github/ethyca/fides-fork main !1 ❯ cat .pre-commit-config.yaml 
minimum_pre_commit_version: "2"

repos:
-   repo: https://github.com/Yelp/detect-secrets
    rev: v1.4.0
    hooks:
    -   id: detect-secrets
        args: ['--baseline', '.secrets.baseline']
        exclude: package.lock.json
~/projects/github/ethyca/fides-fork main !1 ❯ detect-secrets scan > .secrets.baseline                    
~/projects/github/ethyca/fides-fork main !1 ?1 ❯ touch test file                                                                                                               5s
~/projects/github/ethyca/fides-fork main !1 ?3 ❯ git add .
~/projects/github/ethyca/fides-fork main +4 ❯ git commit -m "testing yelp / detect-secrets hook"
Detect secrets...........................................................Passed
[main b6ecdd0f] testing yelp / detect-secrets hook
 4 files changed, 2470 insertions(+), 35 deletions(-)
 create mode 100644 .secrets.baseline
 create mode 100644 file
 create mode 100644 test

The test took about 1 second, but notice that I had to create a baseline first. I think this is only needed once per repo, and took about 5 seconds.

For the baseline I installed detect-secrets locally using homebrew. For the hook I think it just needs a local copy of python.

What's next?

I'll take a look at GitGuardian performance next. After that, perhaps start testing with some example secrets. I'll also need to look how each tool allows us to ignore false positives.

sanders41 commented 1 year ago

You are correct about the baseline for detect-secrets. It is created once and shared in the repo.

The ones run with pre-commit will in theory get faster after the initial run because it uses cache, however at 1 second they are already really fast.

ThomasLaPiana commented 1 year ago

@PSalant726 using pre-commit lowers the barrier to entry/setup and helps us easily track/configure our githooks. This is a reasonable dependency to me, and in my opinion gives us much more value than it takes away in added complexity

daveqnet-alt commented 1 year ago

Continuing on, I ran roughly the same test with GitGuardian ggshield. Its initial setup is a bit more complicated but performance is about on par with gitleaks and detect-secrets.

4. GitGuardian ggshield

~/projects/github/ethyca/fides-fork main !1 ❯ cat .pre-commit-config.yaml 
minimum_pre_commit_version: "2"

repos:
- repo: local
  hooks:
    - id: ggshield
      name: GitGuardian Shield
      entry: ggshield secret scan pre-commit
      language: python
      language_version: python3
      stages: [commit]
~/projects/github/ethyca/fides-fork main !1 ❯ touch testfile
~/projects/github/ethyca/fides-fork main !1 ?1 ❯ git add .
~/projects/github/ethyca/fides-fork main +2 ❯ git commit -m "testing ggshield pre-commit hook" 
[INFO] Initializing environment for local.
[INFO] Installing environment for local.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
GitGuardian Shield.......................................................Passed
[main 2352de78] testing ggshield pre-commit hook
 2 files changed, 8 insertions(+), 36 deletions(-)
 create mode 100644 testfile

The test took about 1 second to complete even though the log message says [INFO] This may take a few minutes...

I installed a local copy of ggshield using homebrew to get this working, it's a hefty enough install (~ 90 seconds to complete). I also needed to create an account on gitguardian.com and authenticate / generate an api key.

daveqnet commented 1 year ago

Okay, I'm eliminating ggshield as a candidate. There's a quota of 1000 API calls per month, and with a scan performed by every dev for every local commit I'm sure that would be exceeded quickly.

I might look into it as a backup for GitHub secrets detection though, with real-time monitoring of our public repos.

Screen Shot 2022-10-21 at 11 42 31

daveqnet-alt commented 1 year ago

I took a look at gitleaks vs detect-secrets when scanning the entire commit history of the fides repo today [edit: actually only gitleaks will scan commit history, detect-secrets scans the current state]. False positives are going to be a challenge alright, a few hundred were identified by each tool.

Aside from the introduction of a secrets detection pre-commit hook to the team, we may have to look at our approach to creating dummy and test secrets so that they can be easily ignored by regexes or wordlists. Ignoring by path is dangerous - we can't assume that everything added under tests/ is automatically safe.

~/projects/github/daveqnet-alt/fides-fork main ❯ git log -1 | head -n 3
commit 31ed9567caf3fa0e25267a7a0f216aca01a58770
Author: Paul Sanders <paul@ethyca.com>
Date:   Fri Oct 21 20:45:47 2022 -0400
~/projects/github/daveqnet-alt/fides-fork main ❯ gitleaks detect --report-path ~/projects/logs/gitleaks/gitleaks-report.json

    ○
    │╲
    │ ○
    ○ ░
    ░    gitleaks

5:12PM INF 2316 commits scanned.
5:12PM INF scan completed in 39.9s
5:12PM WRN leaks found: 506
~/projects/github/daveqnet-alt/fides-fork main ❯ detect-secrets scan > ~/projects/logs/detect-secrets/.secrets.baseline                                                        
~/projects/github/daveqnet-alt/fides-fork main ❯ detect-secrets audit --report ~/projects/logs/detect-secrets/.secrets.baseline > ~/projects/logs/detect-secrets/detect-secrets-audit.json
~/projects/github/daveqnet-alt/fides-fork main ❯ jq '.results | length' ~/projects/logs/detect-secrets/detect-secrets-audit.json 
278

I won't post the entire scan results themselves here (you can easily generate these yourself if you're interested), but here are examples of findings that each identified in the security section of fides.toml

gitleaks

{
"Description": "Generic API Key",
"StartLine": 33,
"EndLine": 33,
"StartColumn": 17,
"EndColumn": 56,
"Match": "key = \"OLMkv91j8DHiDAULnK5Lxx3kSCov30b3\"",
"Secret": "OLMkv91j8DHiDAULnK5Lxx3kSCov30b3",
"File": ".fides/fides.toml",
"SymlinkFile": "",
"Commit": "5a485387d8af247ec6479e4115088cbbb8394d77",
"Entropy": 4.4528193,
"Author": "Thomas",
"Email": "thomas.lapiana+github@pm.me",
"Date": "2022-10-06T16:18:42Z",
"Message": "Merge unified fides into main (#1245)\n\n* Bump pydash from 5.0.2 to 5.1.0 (#920)\n\nBumps [pydash](https://github.com/dgilland/pydash) from 5.0.2 to 5.1.0.\n- [Release notes](https://github.com/dgilland/pydash/releases)\n- [Changelog](https://github.com/dgilland/pydash/blob/develop/CHANGELOG.rst)\n- [Commits](https://github.com/dgilland/pydash/compare/v5.0.2...v5.1.0)\n\n---\nupdated-dependencies:\n- dependency-name: pydash\n  dependency-type: direct:production\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Update boto3 requirement from ~=1.18.14 to ~=1.24.34 (#917)\n\nUpdates the requirements on [boto3](https://github.com/boto/boto3) to permit the latest version.\n- [Release notes](https://github.com/boto/boto3/releases)\n- [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)\n- [Commits](https://github.com/boto/boto3/compare/1.18.14...1.24.34)\n\n---\nupdated-dependencies:\n- dependency-name: boto3\n  dependency-type: direct:production\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump dask from 2022.6.1 to 2022.7.0 (#915)\n\nBumps [dask](https://github.com/dask/dask) from 2022.6.1 to 2022.7.0.\n- [Release notes](https://github.com/dask/dask/releases)\n- [Changelog](https://github.com/dask/dask/blob/main/docs/release-procedure.md)\n- [Commits](https://github.com/dask/dask/compare/2022.6.1...2022.7.0)\n\n---\nupdated-dependencies:\n- dependency-name: dask\n  dependency-type: direct:production\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Fix integration set up scripts for postgres and mariadb - casing has changed for config variables. (#921)\n\n* zendesk and salesforce connection docs (#908)\n\n* Adobe Campaign access and erasure (#905)\n\n* Updated tutorial to match latest fidesdemo (#772)\n\n* Correct build arg variable name (#925)\n\n* Correct build arg variable name\n\n* Update CHANGELOG\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Default `FIDESOPS__ADMIN_UI__ENABLED` to `True` (#936)\n\n* serve AdminUI by default\n\n* updates changelog\n\n* Update python docker base image from slim-buster to slim-bullseye (#928)\n\n* Update python docker base image from slim-buster to slim-bullseye\n\n* Update CHANGELOG\n\n* Remove ipython from dev-requirements.txt\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\nCo-authored-by: Sean Preston \u003csean@ethyca.com\u003e\n\n* Update boto3 requirement from ~=1.24.34 to ~=1.24.36 (#939)\n\nUpdates the requirements on [boto3](https://github.com/boto/boto3) to permit the latest version.\n- [Release notes](https://github.com/boto/boto3/releases)\n- [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)\n- [Commits](https://github.com/boto/boto3/compare/1.24.34...1.24.36)\n\n---\nupdated-dependencies:\n- dependency-name: boto3\n  dependency-type: direct:production\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump faker from 13.15.0 to 13.15.1 (#941)\n\nBumps [faker](https://github.com/joke2k/faker) from 13.15.0 to 13.15.1.\n- [Release notes](https://github.com/joke2k/faker/releases)\n- [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md)\n- [Commits](https://github.com/joke2k/faker/compare/v13.15.0...v13.15.1)\n\n---\nupdated-dependencies:\n- dependency-name: faker\n  dependency-type: direct:development\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump types-ujson from 5.2.0 to 5.4.0 (#947)\n\nBumps [types-ujson](https://github.com/python/typeshed) from 5.2.0 to 5.4.0.\n- [Release notes](https://github.com/python/typeshed/releases)\n- [Commits](https://github.com/python/typeshed/commits)\n\n---\nupdated-dependencies:\n- dependency-name: types-ujson\n  dependency-type: direct:development\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Add db vs saas to connection type api (#937)\n\n* add db vs saas to connection type api\n\n* adds changelog line\n\n* mypy\n\n* fix test\n\n* format\n\n* more tests\n\n* formatting\n\n* adds system type query param\n\n* format\n\n* adjust test saas search\n\n* correct query param\n\n* Use Nox as the build tool instead of Make (#919)\n\n* Create noxfile.py\n\n* update the dockerfile with more stages\n\n* add GitPython as a dev requirement (used by nox builds)\n\n* add the noxfiles, all docker builds work\n\n* fix the \"make compose-build\" command\n\n* remove the worker docker stage due to it being redundant with prod\n\n* create a unified compose file for integrations\n\n* update the password prompt to be accurate\n\n* copy/pasta run_infrastructure into the noxfiles dir so it can be called directly via python\n\n* add create_user and seed_test_data to the nox utils\n\n* docs commands work\n\n* get the generic dev command working\n\n* add db commands to utils\n\n* clean up mypy configuration in pyproject.toml and remove config from setup.cfg\n\n* simplify some of the CI targets and start updating the pytest targets\n\n* cleanup run_infrastructure\n\n* update ci_suite and other CI nox targets\n\n* add nox as a dev-requirement\n\n* get dev commands working, sans quickstart\n\n* tweak to the compose_down constant\n\n* get the new pylint target passing\n\n* remove analytics_id\n\n* updated the changelog\n\n* Apply suggestions from code review\n\nCo-authored-by: Paul Sanders \u003cpsanders1@gmail.com\u003e\n\n* fix the xenon command\n\n* add a few tweaks to the worker dev command\n\n* fix an import issue\n\n* update dockerignore\n\n* move mssql to the bottom of the datastore list\n\nCo-authored-by: Paul Sanders \u003cpsanders1@gmail.com\u003e\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* 863 - Retry a DSR (FE) (#938)\n\n* Update docs docker base image from slim-buster to slim-bullseye (#949)\n\n* Changed Debian base image\n\n* Update the python version to be the same as used in the fidesops app image\n\nCo-authored-by: Paul Sanders \u003cpsanders1@gmail.com\u003e\n\n* Updated changelog\n\nCo-authored-by: Dave Quinlan \u003cdave@ethyca.com\u003e\nCo-authored-by: Paul Sanders \u003cpsanders1@gmail.com\u003e\n\n* Experimenting with fixes for the failing MSSQL CI tests (#918)\n\n* Change docker password environment variable to MSSQL_SA_PASSWORD\n\n* Revert password environment variable name\n\n* Set MSSQL user to root in docker-compose\n\n* Revert setting root user in docker-compose\n\n* Change environment variable name from REQUIRE_MSSQL to SKIP_MSSQL_INSTALLATION\n\n* Add healthcheck to mssql compose file\n\n* Modify healthcheck command\n\n* Revert healthcheck\n\n* Try for more loging information\n\n* Another try for logging\n\n* Another try for logging\n\n* Try running only mssql tests to avoid timeout\n\n* Revert mssql only flag\n\n* Extend time out to try to get logs\n\n* Revert extra logging and extended timeout\n\n* Set network mode to host\n\n* Make mssql run on its own\n\n* Remove network from docker-compose and only run mssql in integration\n\n* Increase integration test logging\n\n* Revert mssql only\n\n* Use cache for docker\n\n* Fix workflow error\n\n* Fix workflow error\n\n* Fix target\n\n* Fix make traget\n\n* Fix make traget\n\n* Revert cache\n\n* Verify that mssql is running from pytest fixture\n\n* Add restart to mssql\n\n* Revert rester in docker-compose\n\n* Revert wait for mssql in pytest fixture\n\n* Wait for mssql to be ready before adding test data\n\n* Add count of retries\n\n* Update CHANGELOG\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Delete custom GitHub issue templates (#955)\n\nWe've defined generic issue templates for all Ethyca repos here: https://github.com/ethyca/.github/tree/main/.github/ISSUE_TEMPLATE\n\nRemoving the templates from this repo allows it to automatically pull in the organization templates for consistency.\n\n* Bump sqlalchemy-redshift from 0.8.8 to 0.8.10 (#940)\n\nBumps [sqlalchemy-redshift](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift) from 0.8.8 to 0.8.10.\n- [Release notes](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/releases)\n- [Changelog](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/blob/main/CHANGES.rst)\n- [Commits](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/compare/0.8.8...0.8.10)\n\n---\nupdated-dependencies:\n- dependency-name: sqlalchemy-redshift\n  dependency-type: direct:production\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Move tests into an \"ops\" subdir (#935)\n\n* Create __init__.py\n\n* move all of the test files down one dir into `ops`\n\n* update test paths\n\n* update paths where needed\n\n* update the changelog\n\n* Dispatch a repository event on new published releases (#945)\n\n* Add release dispatch event action\n\n* Update `CHANGELOG.md`\n\n* Reduce # of clients connected to the application db [#810] (#944)\n\n* Reduce number of open connections:\n\n- Limit task concurrency to two per worker.\n- Create one Engine per celery process which opens up a connection pool.  Create one Session per celery process and use that session across privacy requests.\n- Close the session after the privacy request has finished executing.  This just resets the session and returns connections back to the pool. It can be reused.\n- Remove unnecessary places where session is closed manually because the session is being used as a context manager and is already closed through that.\n- Pass the same Session that the privacy request is using through to TaskResources to be re-used to create ExecutionLogs instead of opening up a new Session.\n- Don't close the session when passing it into the Execution Log, wait until the entire privacy request is complete/exited.\n\n* Define \"self\" for run_privacy_task - it's the task itself.\n\nFor mypy's benefits, define that the session is a context manager.\n\n* Make a session non-optional for graph_task.run_access_request, graph_task.run_erasure, and for instantiating taskResources\n\n* Use missing db fixture.\n\n* Add missing db resource.\n\n* Update test to reflect new behavior that disabling a datasource while a request is in progress can cause related collections to be skipped once the current session is expired and the connection config has the most recent state.\n\nBecause the same Session that is being used to run the PrivacyRequest is now being used for ExecutionLogs, the process of saving an ExecutionLog runs a session.commit() which expires the Session and causes the ConnectionConfig to have the most recent state the next time it is accessed.\n\n* Update CHANGELOG.\n\n* enable worker by default in our dockerfile (#958)\n\n* add extra steps to make clean (#767)\n\n* Push `dev` image on pushes to `main` (#956)\n\n* Update publish_to_dockerhub.yml\n\n* add a dev step and use nox\n\n* update the changelog\n\n* Move Client Code into an `ops` subdir (#964)\n\n* Move Client Code into an `ops` subdir\n\n* move all of the files\n\n* update the dockerfile\n\n* update package.json\n\n* update codepaths for workflow tests\n\n* Update the changelog and docs references\n\n* Update .github/dependabot.yaml\n\n* Bump gitpython from 3.1 to 3.1.27 (#971)\n\nBumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.1 to 3.1.27.\n- [Release notes](https://github.com/gitpython-developers/GitPython/releases)\n- [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES)\n- [Commits](https://github.com/gitpython-developers/GitPython/compare/3.1.0...3.1.27)\n\n---\nupdated-dependencies:\n- dependency-name: gitpython\n  dependency-type: direct:development\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Fix the `nox -s dev` command not spinning up the webserver (#959)\n\n* Update dev_nox.py\n\n* spin up the app before running a container shell\n\n* [#927, #929, #930] sendgrid, adobe, outreach connector docs (#951)\n\n* Bump sqlalchemy-utils from 0.37.8 to 0.38.3 (#968)\n\nBumps [sqlalchemy-utils](https://github.com/kvesteri/sqlalchemy-utils) from 0.37.8 to 0.38.3.\n- [Release notes](https://github.com/kvesteri/sqlalchemy-utils/releases)\n- [Changelog](https://github.com/kvesteri/sqlalchemy-utils/blob/master/CHANGES.rst)\n- [Commits](https://github.com/kvesteri/sqlalchemy-utils/compare/0.37.8...0.38.3)\n\n---\nupdated-dependencies:\n- dependency-name: sqlalchemy-utils\n  dependency-type: direct:production\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump pyodbc from 4.0.32 to 4.0.34 (#980)\n\nBumps [pyodbc](https://github.com/mkleehammer/pyodbc) from 4.0.32 to 4.0.34.\n- [Release notes](https://github.com/mkleehammer/pyodbc/releases)\n- [Commits](https://github.com/mkleehammer/pyodbc/compare/4.0.32...4.0.34)\n\n---\nupdated-dependencies:\n- dependency-name: pyodbc\n  dependency-type: direct:production\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump dask from 2022.7.0 to 2022.7.1 (#967)\n\nBumps [dask](https://github.com/dask/dask) from 2022.7.0 to 2022.7.1.\n- [Release notes](https://github.com/dask/dask/releases)\n- [Changelog](https://github.com/dask/dask/blob/main/docs/release-procedure.md)\n- [Commits](https://github.com/dask/dask/compare/2022.7.0...2022.7.1)\n\n---\nupdated-dependencies:\n- dependency-name: dask\n  dependency-type: direct:production\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump alembic from 1.8.0 to 1.8.1 (#989)\n\nBumps [alembic](https://github.com/sqlalchemy/alembic) from 1.8.0 to 1.8.1.\n- [Release notes](https://github.com/sqlalchemy/alembic/releases)\n- [Changelog](https://github.com/sqlalchemy/alembic/blob/main/CHANGES)\n- [Commits](https://github.com/sqlalchemy/alembic/commits)\n\n---\nupdated-dependencies:\n- dependency-name: alembic\n  dependency-type: direct:production\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* fix erroneous values in the Outreach config (#988)\n\n* Add documentation for new nox commands (#981)\n\n* Add documentation for new nox commands\n\n* changelog\n\n* missed make to nox edits\n\n* review edits\n\n* update `nox -s dev` to not open a shell, add it as a posarg option\n\n* Apply suggestions from code review\n\n* remove the analytics id\n\n* update typo\n\n* remove extra numbering\n\nCo-authored-by: Thomas \u003cthomas.lapiana+github@pm.me\u003e\n\n* Bump types-toml from 0.10.7 to 0.10.8 (#998)\n\nBumps [types-toml](https://github.com/python/typeshed) from 0.10.7 to 0.10.8.\n- [Release notes](https://github.com/python/typeshed/releases)\n- [Commits](https://github.com/python/typeshed/commits)\n\n---\nupdated-dependencies:\n- dependency-name: types-toml\n  dependency-type: direct:development\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump fideslog from 1.2.1 to 1.2.2 (#996)\n\nBumps [fideslog](https://github.com/ethyca/fideslog) from 1.2.1 to 1.2.2.\n- [Release notes](https://github.com/ethyca/fideslog/releases)\n- [Commits](https://github.com/ethyca/fideslog/compare/v1.2.1...v1.2.2)\n\n---\nupdated-dependencies:\n- dependency-name: fideslog\n  dependency-type: direct:production\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Added Choose your connection feature (#987)\n\n* Added Choose your connection feature\n\n760-Add a Connection - Select a connector to configure (front end)\n866-Add a Connection - Front End layout structure\n\n* Updated CHANGELOG.md file\n\n* Fix lint issues\n\n* Fix build issue\n\n* Hide overflow\n\nCo-authored-by: Andrew Jackson \u003candrew.c.j1995@gmail.com\u003e\n\n* saas request overrides (#986)\n\n* initial cut of saas request overrides. include mailchimp as an example and test case. minor refactor of some of the saas request execution to enable smoother override\n\n* fix rebase issue by moving saas override tests into ops subdir\n\n* import path updates to resolve conflicts caused by rebase\n\n* add session parameter into graph task calls to fix saas override integration tests caused by rebase\n\n* update changelog\n\n* tweaks to saas connector overrides and associated tests\n\n* expose override factory register as module variable for clenaer decorator calls\n\nCo-authored-by: Adam Sachs \u003cadam@Adams-MacBook-Pro.local\u003e\n\n* Update OAuth strategy to be able to perform local testing (#962)\n\n* Update boto3 requirement from ~=1.24.36 to ~=1.24.42 (#1001)\n\nUpdates the requirements on [boto3](https://github.com/boto/boto3) to permit the latest version.\n- [Release notes](https://github.com/boto/boto3/releases)\n- [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)\n- [Commits](https://github.com/boto/boto3/compare/1.24.36...1.24.42)\n\n---\nupdated-dependencies:\n- dependency-name: boto3\n  dependency-type: direct:production\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump fastapi[all] from 0.78.0 to 0.79.0 (#1002)\n\nBumps [fastapi[all]](https://github.com/tiangolo/fastapi) from 0.78.0 to 0.79.0.\n- [Release notes](https://github.com/tiangolo/fastapi/releases)\n- [Commits](https://github.com/tiangolo/fastapi/compare/0.78.0...0.79.0)\n\n---\nupdated-dependencies:\n- dependency-name: fastapi[all]\n  dependency-type: direct:production\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* update config reference to use lowercase values (#952)\n\n* [#909] update config reference to use lowercase values\n\n* changelog\n\n* missed uppercase variables\n\n* update usages of False to false in connector docs\n\n* Added Auth0 Connector (#991)\n\n* add pagination back to connection types endpoints (#1019)\n\n* add pagination back to connection types endpoints\n\n* check for pagination in system_type search\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Subject Request Events and Logs Section (#1018)\n\n* Fix small issue with eslint config\n\n* Get initial drawer working\n\n* Add jest config to eslintignore\n\n* Fix small css issue\n\n* Refactor components and get initial functionality\n\n* Fix small logic error\n\n* Fix couple of bugs and format code\n\n* Conditinally display error tag\n\n* Fix issues with merge\n\n* Format and lint\n\n* Sort imports\n\n* Update to new solution\n\n* Format code\n\n* Update changelog\n\nCo-authored-by: Sean Preston \u003csean@ethyca.com\u003e\n\n* First draft of OAuth documentation (#963)\n\n* Send Errored Requests / Reprocessed Requests Info to FidesLog [#754] (#993)\n\n* Add a method to format a representation of the graph for caching in Redis and a separate method to build a summary of the differences in the graph when a privacy request is rerun to fideslog.\n\n- Adds FieldAddress.from_string method.\n\n* Add methods to cache a representation of the access graph when it is built and a separate method to retrieve it from the cache.\n\n- Also add a method to build a \"rerun_access_graph\" AnalyticsEvent for fideslog where applicable.\n\n* When running the access portion of the privacy request, log stats about a rerun and then cache the current access graph.\n\n- Give a different prefix when we're caching the access graph to not get mixed up with access request results.\n\n* Show skipped new edges that are directly upstream of completed nodes instead. These edges are intentionally dropped from the new graph on rerun, so want to surface this count.\n\n* Log if a privacy request fails during the \"erasure\" step of privacy request execution.  Even though the access step is not rerun here, compare the previously cached access graph with the access graph that would have been run to determine what data has changed.\n\n* Add missing session variables.\n\n* Send an event to Fideslog when privacy request execution fails.\n\n* Add missed session - bad merge.\n\n* Update changelog.\n\n* Update docstrings.\n\n* Remove copy/paste comment.\n\n* Respond to CR comments.\n\n* Currently AnalyticsEvent.local_host cannot be None.\n\n* Update the compose file and workflows to expect an already-built image (#966)\n\n* Update the compose file and workflows to be in line with fidesctl\n\n* update the compose file to look for a specific image\n\n* rename files and update the safe PR checks\n\n* add check_migrations to the ci checks and nox\n\n* fix the failing PR checks\n\n* fix CI failures\n\n* update the unsafe checks workflow\n\n* update run_infrastructure to use the compose service name\n\n* remove the makefile and the old run_infra script\n\n* bump pylint version, pin isort, fix issues\n\n* update pytest setup path\n\n* update the changelog\n\n* make OPS_TEST_DIR a constant\n\n* fix nox missing vars\n\n* specify that the nox imports are relative imports\n\n* remove relative import paths\n\n* run isort\n\nCo-authored-by: Sean Preston \u003csean@ethyca.com\u003e\n\n* Create `AuditLog` on privacy request approval (#1038)\n\n* Create approval audit logs\n\n* Add tests\n\n* disable a pylint lint\n\n* Update changelog\n\n* Update privacy_request fixture\n\n* Updating Salesforce to use OAuth2 authentication code flow (#1039)\n\n* Removing saas_config.toml (#1043)\n\n* Bump types-pyyaml from 6.0.9 to 6.0.11 (#1047)\n\nBumps [types-pyyaml](https://github.com/python/typeshed) from 6.0.9 to 6.0.11.\n- [Release notes](https://github.com/python/typeshed/releases)\n- [Commits](https://github.com/python/typeshed/commits)\n\n---\nupdated-dependencies:\n- dependency-name: types-pyyaml\n  dependency-type: direct:development\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump docker/build-push-action from 2 to 3 (#1044)\n\nBumps [docker/build-push-action](https://github.com/docker/build-push-action) from 2 to 3.\n- [Release notes](https://github.com/docker/build-push-action/releases)\n- [Commits](https://github.com/docker/build-push-action/compare/v2...v3)\n\n---\nupdated-dependencies:\n- dependency-name: docker/build-push-action\n  dependency-type: direct:production\n  update-type: version-update:semver-major\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump dask from 2022.7.1 to 2022.8.0 (#1046)\n\nBumps [dask](https://github.com/dask/dask) from 2022.7.1 to 2022.8.0.\n- [Release notes](https://github.com/dask/dask/releases)\n- [Changelog](https://github.com/dask/dask/blob/main/docs/release-procedure.md)\n- [Commits](https://github.com/dask/dask/compare/2022.7.1...2022.8.0)\n\n---\nupdated-dependencies:\n- dependency-name: dask\n  dependency-type: direct:production\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Update boto3 requirement from ~=1.24.42 to ~=1.24.46 (#1045)\n\nUpdates the requirements on [boto3](https://github.com/boto/boto3) to permit the latest version.\n- [Release notes](https://github.com/boto/boto3/releases)\n- [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)\n- [Commits](https://github.com/boto/boto3/compare/1.24.42...1.24.46)\n\n---\nupdated-dependencies:\n- dependency-name: boto3\n  dependency-type: direct:production\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Refactor static checks to run outside of Docker (#1053)\n\n* Refactor static checks to run outside of Docker\n\n* run static checks outside of docker, reusing cached virtual envs\n\n* update the changelog\n\n* Create CHANGELOG.md\n\n* Update CHANGELOG.md\n\n* update the static checks in the workflows\n\n* remove the static check dependency on \"build\"\n\n* Create finished audit log for privacy requests (#1040)\n\n* Add finished AuditLog\n\n* Bump fideslib\n\n* Update test\n\n* Fix format lints\n\n* Lowercase encoding\n\n* Update changelog\n\n* Fix issue with migration downgrade\n\n* Sort lists in test data tests\n\n* format file\n\n* Update quickstart to use docker-compose and docker network for all commands (#1056)\n\n* Bump snowflake-sqlalchemy from 1.3.2 to 1.3.4 (#1051)\n\nBumps [snowflake-sqlalchemy](https://github.com/snowflakedb/snowflake-sqlalchemy) from 1.3.2 to 1.3.4.\n- [Release notes](https://github.com/snowflakedb/snowflake-sqlalchemy/releases)\n- [Commits](https://github.com/snowflakedb/snowflake-sqlalchemy/commits)\n\n---\nupdated-dependencies:\n- dependency-name: snowflake-sqlalchemy\n  dependency-type: direct:production\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* updates changelog (#1063)\n\n* Reorganize application code (#1058)\n\n* Reorganize application code\n\n* first round of moving everything and updating the import paths\n\n* checkpoint, more path updates\n\n* more path updates\n\n* fix imports\n\n* fix isort\n\n* fix mypy, isort and setup.py issues\n\n* fix unsafe checks build step running on any label\n\n* update the changelog\n\n* delete temp files\n\n* Update docs/fidesops/docs/development/contributing_details.md\n\nCo-authored-by: Paul Sanders \u003cpsanders1@gmail.com\u003e\n\n* update more file references\n\nCo-authored-by: Paul Sanders \u003cpsanders1@gmail.com\u003e\n\n* Patch versioneer to allow editable installs (#1070)\n\n* Patch versioneer to allow editable installs\n\n* Update CHANGELOG\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Preserving headers in SaaSRequestParams during pagination (#1069)\n\n* Add setuptools to dev-requirements to fix versioneer error (#1072)\n\n* Add setuptools to dev-requirements to fix versioneer error and revert patch\n\n* Update CHANGELOG\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Bump fideslang to 1.1.0 (#890)\n\n* Bump fideslang to 1.1.0\n\n* find/replace user.provided.identifiable -\u003e user\n\n* remove derived mentions\n\n* don't remove volumes on teardown\n\n* update address fields\n\n* replace user.derived and user.provided -\u003e user\n\n* fix two more tests\n\n* fix pylint errors\n\n* fix integration tests\n\n* fix failing mongo tasks\n\n* update the changelog\n\n* fix the failing mongo task test\n\n* another mongo task fix\n\n* more mongo task fixes\n\n* Revert test back to two addresses being masked.\n\n* Update mongo array access test to reflect that underlying dataset has changed, and policy has changed, so more fields are returned.\n\n* add the noxfiles\n\n* update the dockerfile and get the nox docker commands working\n\n* Revert \"update the dockerfile and get the nox docker commands working\"\n\nThis reverts commit 4b98c62163d419996977b7bb2dd17d181aac2f07.\n\n* remove noxfiles\n\n* updates from comments\n\n* Update test\n\n* Add migration\n\n* Update categories in test config files\n\n* Fix data categories\n\n* Fix more data categories\n\n* Change user.provided.nonidentifiable to user\n\n* Update migraiton with review suggestions\n\n* Run black\n\n* Add more logging to migration\n\n* Increment counter\n\n* fix migration conflict\n\nCo-authored-by: Dawn Pattison \u003cpattisdr@users.noreply.github.com\u003e\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* 797 datadog (#1060)\n\n* 515 SaaS connector Logi ID (#1074)\n\n* Combine Execution and Audit Logs in Request Status Endpoint [#1024] (#1068)\n\n* Update the request status endpoint, so when the verbose query param is used and execution logs are embedded, also return audit logs.\n\nExecution Logs are created at the collection level while audit logs are for the overall privacy request level, so most fields returned for audit logs are None.\nLogs are also grouped at the dataset level here, so give the audit logs a fake dataset name for display purposes, for example, \"Request approved\".\n\n* Update CHANGELOG and update docs to reflect that audit logs are included in a verbose request status response.\n\n* Notify fidesdemo on new releases (#1075)\n\n* Notify fidesdemo on new releases\n\n* Update `CHANGELOG.md`\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Bump pytest from 6.2.2 to 7.1.2 (#1081)\n\nBumps [pytest](https://github.com/pytest-dev/pytest) from 6.2.2 to 7.1.2.\n- [Release notes](https://github.com/pytest-dev/pytest/releases)\n- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)\n- [Commits](https://github.com/pytest-dev/pytest/compare/6.2.2...7.1.2)\n\n---\nupdated-dependencies:\n- dependency-name: pytest\n  dependency-type: direct:development\n  update-type: version-update:semver-major\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump faker from 13.15.1 to 14.0.0 (#1080)\n\nBumps [faker](https://github.com/joke2k/faker) from 13.15.1 to 14.0.0.\n- [Release notes](https://github.com/joke2k/faker/releases)\n- [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md)\n- [Commits](https://github.com/joke2k/faker/compare/v13.15.1...v14.0.0)\n\n---\nupdated-dependencies:\n- dependency-name: faker\n  dependency-type: direct:development\n  update-type: version-update:semver-major\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump actions/setup-python from 3 to 4 (#1078)\n\nBumps [actions/setup-python](https://github.com/actions/setup-python) from 3 to 4.\n- [Release notes](https://github.com/actions/setup-python/releases)\n- [Commits](https://github.com/actions/setup-python/compare/v3...v4)\n\n---\nupdated-dependencies:\n- dependency-name: actions/setup-python\n  dependency-type: direct:production\n  update-type: version-update:semver-major\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump fideslog from 1.2.2 to 1.2.3 (#1079)\n\nBumps [fideslog](https://github.com/ethyca/fideslog) from 1.2.2 to 1.2.3.\n- [Release notes](https://github.com/ethyca/fideslog/releases)\n- [Commits](https://github.com/ethyca/fideslog/compare/v1.2.2...v1.2.3)\n\n---\nupdated-dependencies:\n- dependency-name: fideslog\n  dependency-type: direct:production\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump types-urllib3 from 1.26.15 to 1.26.22 (#1084)\n\nBumps [types-urllib3](https://github.com/python/typeshed) from 1.26.15 to 1.26.22.\n- [Release notes](https://github.com/python/typeshed/releases)\n- [Commits](https://github.com/python/typeshed/commits)\n\n---\nupdated-dependencies:\n- dependency-name: types-urllib3\n  dependency-type: direct:development\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump fideslang from 1.1.0 to 1.2.0 (#1085)\n\nBumps [fideslang](https://github.com/ethyca/fideslang) from 1.1.0 to 1.2.0.\n- [Release notes](https://github.com/ethyca/fideslang/releases)\n- [Changelog](https://github.com/ethyca/fideslang/blob/main/CHANGELOG.md)\n- [Commits](https://github.com/ethyca/fideslang/compare/1.1.0...1.2.0)\n\n---\nupdated-dependencies:\n- dependency-name: fideslang\n  dependency-type: direct:production\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump next-auth from 4.9.0 to 4.10.3 in /clients/ops/privacy-center (#1014)\n\nBumps [next-auth](https://github.com/nextauthjs/next-auth) from 4.9.0 to 4.10.3.\n- [Release notes](https://github.com/nextauthjs/next-auth/releases)\n- [Changelog](https://github.com/nextauthjs/next-auth/blob/main/CHANGELOG.md)\n- [Commits](https://github.com/nextauthjs/next-auth/compare/next-auth@v4.9.0...next-auth@v4.10.3)\n\n---\nupdated-dependencies:\n- dependency-name: next-auth\n  dependency-type: direct:production\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\nCo-authored-by: Sean Preston \u003csean@ethyca.com\u003e\n\n* Bump fideslib (#1092)\n\n* Bump fideslib to fix docs auth issue\n\n* Update CHANGELOG\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* 983 Adds infra for email config and dispatch (#1059)\n\n* Email config mvp crud / db layer\n\n* adds name to email config model\n\n* remove unintended changes\n\n* gets POC working\n\n* removes org name config var, updates crud endpoints to better handle supporting only 1 config, formatting\n\n* updates postman collection, adds to changelog, bumps downrev on migration\n\n* sort\n\n* formatting\n\n* use correct response model\n\n* Delete base.py\n\n* update migration annotation\n\n* Create a Saas Connector from a Template [#814] (#1076)\n\n* Starting point for SaaS connector templates\n\n* Fix imports from restructuring.\n\n* Get happy path working for instantiate connector from template endpoint.\n\n* Remove updating connector instances for now - out of scope.\n\n* Test nonexistent templates, secrets validation, instance key / fides key already exists.\n\n* Create DatasetConfigs and ConnectionConfigs instead of create_or_update in the template endpoint.  Don't save ConnectionConfig until secrets are validated.\n\n* Add the other saas connectors to the registry and update their configs and datasets with instance_fides_key.\n\n- Fix datadog yaml so it can be included in the saas connector registry. There was an error in how the saas config was formatted.\n\n* Update the fides_keys in the existing saas configs and dataset yamls to have brackets around the \"instance_fides_key\" to indicate these will be replaced.\n\nUpdate the fides_key definition to allow \"\u003cinstance_fides_key\u003e\" with brackets specifically to pass validation.\n\n* Fix a side effect on a separate endpoint that returns the types of secrets that should be supplied for a given connector.  Use the saas config type instead of the fides key for the model title. Add test verifying that fides key /instance key validation works as expected.\n\n* - Update CHANGELOG\n- Add new endpoint to postman collection\n- Add drafts doc.\n- Update old response body in docs for connection types.\n\n* Replace the \u003cinstance_fides_key\u003e with a properly formatted fides_key in the saas fixtures.\n\n* If DatasetConfig creation fails, delete the recently created ConnectionConfig.\n\n* Address some of the saas integration tests where I've changed the fides_key.\n\n* Fix typos.\n\n* Fix typo.\n\n* Fix unrelated bug where hubspot dataset has new datacategories with user-* data categories after the fideslang update, so they would show up if the user picked a \"user\" data category.\n\n* Respond to CR.\n\nCo-authored-by: Dawn Pattison \u003cpattisdr@users.noreply.github.com\u003e\n\n* Update the \"instantiate_connection_from_template\" to return a portion of both the connection config and the dataset. (#1105)\n\n* Adds `AuditLog` and `ExecutionLog`s to `seed_test_data` command (#1097)\n\n* Reduce Idle Connections from Health Checks [#1102] (#1107)\n\n* Don't create a new engine as part of running the health checks and share a single engine across the application, including for the health checks.  Currently we're using the default pool_size and max_overflow.\n\n* Update changelog.\n\n* Fix that health checks are still supposed to run, even if the database is disabled.\n\n* Need to yield instead -  'generator' object has no attribute 'query'\n\n* Escape redis user and password (#1104)\n\n* Escape redis user and password\n\n* Update CHANGELOG\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* pass in analytics id env to worker (#1113)\n\n* pass in analytics id env to worker\n\n* changelog\n\n* Adds tests for email endpoints and dispatch service (#1112)\n\n* Endpoint to Verify User Identity [#1062] (#1111)\n\n* Add an endpoint to verify a user's identity before queuing the privacy request provided it doesn't need separate manual approval by a system admin.\n\n- Add a new PrivacyRequest.identity_verified_at timestamp\n- Add a new PrivacyRequestStatus - \"identity_unverified\".\n- Add methods to cache the verification code in Redis for comparison with a default ttl of 10 minutes\n\n* - Fix linting/copy-paste errors.\n- Update changelog.\n- Add endpoint to postman\n\n* Add new keys to response bodies.\n\n* Instead of using a new VerificationCode schema, use the SubjectIdentityVerificationBodyParams that already exists.\n\n* Revert \"Instead of using a new VerificationCode schema, use the SubjectIdentityVerificationBodyParams that already exists.\"\n\nThis reverts commit 40fcf6d119135d08a6d3ecfc40c5d73846bf2205.\n\n* Bump hashicorp/vault-action from 2.4.1 to 2.4.2 (#1119)\n\nBumps [hashicorp/vault-action](https://github.com/hashicorp/vault-action) from 2.4.1 to 2.4.2.\n- [Release notes](https://github.com/hashicorp/vault-action/releases)\n- [Changelog](https://github.com/hashicorp/vault-action/blob/main/CHANGELOG.md)\n- [Commits](https://github.com/hashicorp/vault-action/compare/v2.4.1...v2.4.2)\n\n---\nupdated-dependencies:\n- dependency-name: hashicorp/vault-action\n  dependency-type: direct:production\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump fastapi[all] from 0.79.0 to 0.79.1 (#1120)\n\nBumps [fastapi[all]](https://github.com/tiangolo/fastapi) from 0.79.0 to 0.79.1.\n- [Release notes](https://github.com/tiangolo/fastapi/releases)\n- [Commits](https://github.com/tiangolo/fastapi/compare/0.79.0...0.79.1)\n\n---\nupdated-dependencies:\n- dependency-name: fastapi[all]\n  dependency-type: direct:production\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump types-urllib3 from 1.26.22 to 1.26.23 (#1117)\n\nBumps [types-urllib3](https://github.com/python/typeshed) from 1.26.22 to 1.26.23.\n- [Release notes](https://github.com/python/typeshed/releases)\n- [Commits](https://github.com/python/typeshed/commits)\n\n---\nupdated-dependencies:\n- dependency-name: types-urllib3\n  dependency-type: direct:development\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump nox from 2022.1.7 to 2022.8.7 (#1118)\n\nBumps [nox](https://github.com/wntrblm/nox) from 2022.1.7 to 2022.8.7.\n- [Release notes](https://github.com/wntrblm/nox/releases)\n- [Changelog](https://github.com/wntrblm/nox/blob/main/CHANGELOG.md)\n- [Commits](https://github.com/wntrblm/nox/compare/2022.1.7...2022.8.7)\n\n---\nupdated-dependencies:\n- dependency-name: nox\n  dependency-type: direct:development\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump faker from 14.0.0 to 14.1.0 (#1122)\n\nBumps [faker](https://github.com/joke2k/faker) from 14.0.0 to 14.1.0.\n- [Release notes](https://github.com/joke2k/faker/releases)\n- [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md)\n- [Commits](https://github.com/joke2k/faker/compare/v14.0.0...v14.1.0)\n\n---\nupdated-dependencies:\n- dependency-name: faker\n  dependency-type: direct:development\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump sqlalchemy-redshift from 0.8.10 to 0.8.11 (#1121)\n\nBumps [sqlalchemy-redshift](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift) from 0.8.10 to 0.8.11.\n- [Release notes](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/releases)\n- [Changelog](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/blob/main/CHANGES.rst)\n- [Commits](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/compare/0.8.10...0.8.11)\n\n---\nupdated-dependencies:\n- dependency-name: sqlalchemy-redshift\n  dependency-type: direct:production\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* POC for dynamic routing (#1033)\n\n* Add initial POC for dynamic routing\n\n* fix a couple lints\n\n* Fix mypy lint\n\n* making pylint happy\n\n* Remove log\n\n* Fix another pylint issue\n\n* Add docstring\n\n* Update index if check\n\n* Handle nested nextjs routes\n\n* Update changelog\n\n* Add Sass connector configuration (#1099)\n\n* Sass Connecter feature development\n\n761 - Add a Connection - SaaS connector's configuration parameters\n984 - Saas Connector configuration - left navigation to toggle between connection params and dataset config\n985 - SaaS Connector Configuration - Testing a Connection\n\n* Skipping unit test temporarily\n\n* Update flags.json file\n\nBy default, turning off the createNewConnection flag which is still under development.\n\n* Updated CHANGELOG.md file\n\n* Resolved ESLint issues\n\n* Resolved npm build issue\n\n* Resolved npm build issue\n\n* Updated Saas connector configuration\n\n* Updated Saas connector configuration\n\n* Removed unnecessary import statement\n\n* add new privilege for creating SaaS connectors to user management interface\n\n* add connections read as a privilege\n\n* Applied code review feedback\n\n* Updated the CSS visibility of the CircleHelpIcon component\n\n* Added toast success when a user creates a Saas config\n\n* Resolved ESLint issue\n\nCo-authored-by: Sean Preston \u003csean@ethyca.com\u003e\n\n* 1128-Add Retry button back into the subject request detail view (#1131)\n\n* 1128-Add Retry button back into the subject request detail view\n\n* Updated CHANGELOG.md file\n\n* provide a way to give invited users the resume permission\n\nCo-authored-by: Sean Preston \u003csean@ethyca.com\u003e\n\n* Fix localhost setting for non-endpoint analytics calls [#1029] (#1130)\n\n* Set local_host to None for non-endpoint analytics calls.  These are logging various tasks coming out of celery.\n\n* Update Changelog.\n\n* Add email_templates module (#1123)\n\n* Add email_templates module\n\n* run isort\n\n* Add unit tests\n\n* Update ttl calculation\n\n* Add ttl minutes test\n\n* fix lint issues\n\n* fix pylint issue\n\n* fix pylint issue\n\n* fix isort\n\n* Update template constant\n\n* Update changelog\n\n* fix lints\n\n* Add jinja to requirements.txt\n\n* update templates directory\n\n* update unit test\n\n* Update imports\n\n* fix issue template path\n\n* Add templates to manifest\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Formatting Issues in Webhooks Documentation (#1114)\n\n* Fix typo in `derived_identity`\n\n* Update CHANGELOG.md\n\n* Missed PR Link\n\n* replaced `\u003c\u003e` with `{}`\n\nIn the live version of the docs, the `\u003c\u003e`s were being stripped from our code example titles. I've replaced them with `{}` to align with some of the other pages I've seen.\n\n* Send Identity Verification Email [#1010] (#1115)\n\n* If identity verification required, send email to the user with the verification code.\n\n* Adjust the identity_verification_required autouse fixture, and add an autouse override for just the tests where we want to turn on identity verification.\n\n* Add starting docs and updating the changelog.\n\nStart with identity_verification_required set to False for now until all the related pieces are in.\n\n* Update some of the docstrings.\n\n* Add unverified status color in the FE.\n\n* Add new privacy request status to types and constants.\n\n* Restore trailing comma.\n\n* Update identity_verification_required to subject_identity_verification_required for clarity.\n\n* Adjust email_body_params to accommodate new template.\n\nCo-authored-by: Sean Preston \u003csean@ethyca.com\u003e\n\n* Support case-insensitive connection type search [Unticketed] (#1133)\n\n* Make connection type search case-insensitive.\n\n* Update changelog.\n\n* Add option to login as root user from config (#1116)\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Move logging to %-style formatting [#837] (#1132)\n\n* Add a new Pii class and use it to wrap arguments not already wrapped with NonPii in those logs that are currently using %-style formatting.\n\n* Switch logging formatting to %-style instead of f-string.\n\n* Continue to address lingering f string instances, and wrap some arguments in Pii, such as raw exceptions.\n\n* Remove NotPii class and update tests.\n\n* Adjust errors made in %-style conversion.\n\n* Remove accidental Pii on print statements, update some PII wrappings.\n\n* Adjust string formatting of newly added log.\n\n* Update Changelog.\n\n* Fix missed closing curly brackets.\n\n* Remove missed curly brackets.\n\n* Add future annotations for backwards compatibility (#1136)\n\n* Add future annotations for backwards compatibility\n\n* Update CHANGELOG\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Release 1.7.1 (#1141)\n\n* update changelog\n\n* update ERD\n\n* make Hubspot test more robust\n\n* Fix docs build in CI (#1138)\n\n* Fix docs build in CI\n\n* Update CHANGELOG\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\nCo-authored-by: Sean Preston \u003csean@ethyca.com\u003e\n\n* fix command syntax (#1143)\n\n* hook up API request to logout action [Unticketed] (#1139)\n\n* hook up API request to logout action\n\n* run format\n\n* update import orer\n\n* add lint:fix\n\n* Update hubspot users and owners configurations (#1091)\n\n* Update CHANGELOG.md\n\n* SaaS Connector Template Creation Fix: Integer fides_key (#1166)\n\n* Use quotes when replacing \u003c\"instance_fides_key\"\u003e in the saas config and dataset config files to force a string.\n\n* Update CHANGELOG.\n\n* Start a new \"email\" ConnectionConfig type [#1134] (#1142)\n\n* Start a new \"email\" ConnectionConfig type.\n\n* Hide \"email\" type from the get_connection_types endpoint for now, as the email connector isn't fleshed out yet.\n\n* Update CHANGELOG.\n\n* Simplify by sending one email to start?\n\n* Update request body in postman collection.\n\n* Fix CHANGELOG formatting.\n\n* Revert \"SaaS Connector Template Creation Fix: Integer fides_key (#1166)\" (#1171)\n\nThis reverts commit 28b6317fc7bf86019eb6559824226dc261511a95.\n\n* Fix analytics opt out environment variable name (#1170)\n\n* 1004 OAuth2 client credentials flow (#1159)\n\n* update Subject Request detail page (#1164)\n\n* [#495] clarify additions to subject request event log\n\n* [#863] note retry capabilities in subject requests\n\n* changelog\n\n* fidesops.toml\n\n* changelog\n\nCo-authored-by: Cole Garbo \u003ccolegarbo@Ethycas-MacBook-Pro.local\u003e\n\n* Bump fastapi[all] from 0.79.1 to 0.81.0 (#1178)\n\nBumps [fastapi[all]](https://github.com/tiangolo/fastapi) from 0.79.1 to 0.81.0.\n- [Release notes](https://github.com/tiangolo/fastapi/releases)\n- [Commits](https://github.com/tiangolo/fastapi/compare/0.79.1...0.81.0)\n\n---\nupdated-dependencies:\n- dependency-name: fastapi[all]\n  dependency-type: direct:production\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Build docker image for privacy center (#1165)\n\n* Build docker image for privacy center\n\n* Update docs and add docker publish to workflow\n\n* Update CHANGELOG\n\n* Move privacy center docker publish to its own workflow\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Update `StorageConfig` to handle multiple auth methods (#1140)\n\n* Initial automatic impl\n\n* Lints\n\n* disable pylint error\n\n* Fix unit tests\n\n* fix imports\n\n* Update fixture\n\n* Fix config tests\n\n* Covert type to enum\n\n* Add new unit tests\n\n* Update changelong\n\n* Update docs\n\n* Update phone number\n\n* patch task scheduler\n\n* remove pylint ignore and add exception\n\n* Update docs\n\n* 1009 id verification required endpoint (#1221)\n\n* Adds identity verification config endpoint\n\n* adds test, reuse existing config var, adds postman collection\n\n* update changelog\n\n* add type ignore\n\n* move ignore line\n\n* Update src/fidesops/ops/graph/config.py\n\nCo-authored-by: Paul Sanders \u003cpsanders1@gmail.com\u003e\n\n* remove duplicate lines in method\n\n* another dupe line\n\n* remove scope, update order of classmethod decorator\n\n* remove dependencies on endpoint\n\n* unused imports\n\nCo-authored-by: Paul Sanders \u003cpsanders1@gmail.com\u003e\n\n* fix reference in data categories upgrade migration (#1223)\n\n* Publish on every release (#1226)\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Make log send async (#1174)\n\n* Make log send async\n\n* Update CHANGELOG\n\n* Add async to additional fideslog calls\n\n* WIP\n\n* Fix issue with async function is celery\n\n* Make __send work with name mangling\n\n* Remove extra await\n\n* Await coroutines in tests\n\n* Remove analytics id\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* 1106 saas config shopify access endpoints (#1220)\n\n* Adds email communications guide to docs, exposes other hidden guides in nav bar (#1233)\n\n* adds email communications guide to docs, exposes other hidden guides in nav bar\n\n* update changelog\n\n* email docs copyedits\n\n* remove dupe items from nav\n\nCo-authored-by: Cole \u003ccole@ethyca.com\u003e\n\n* Fix version number (#1232)\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Bump pylint from 2.14.5 to 2.15.0 (#1182)\n\n* Bump pylint from 2.14.5 to 2.15.0\n\nBumps [pylint](https://github.com/PyCQA/pylint) from 2.14.5 to 2.15.0.\n- [Release notes](https://github.com/PyCQA/pylint/releases)\n- [Commits](https://github.com/PyCQA/pylint/compare/v2.14.5...v2.15.0)\n\n---\nupdated-dependencies:\n- dependency-name: pylint\n  dependency-type: direct:development\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\n* Ignore missing-timeout warning\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\nCo-authored-by: Paul Sanders \u003cpaul@ethyca.com\u003e\n\n* Bump fideslib (#1236)\n\n* Bump fideslib\n\n* Update CHANGELOG\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Email Connector: Build Masking Instructions  (#1168)\n\n* Pass in input_data to erasure requests, and not just access requests, so it can be used for the email connector, which won't have any rows returned from an access request.\n\n- Add an EmailConnector.build_masking_instructions method with a draft of data needed to instruct the user how to query/mask/what fields to mask on their end.\n\n* Have the EmailConnector.mask_data  cache the raw details of what needs to be masked in Redis. We'll use this to send one email at the end for each \"email\"-based dataset at end, instead of sending one email for each collection.\n\nReuse some of the caching code created for manual connectors / failed privacy requests where similar to the EmailConnectors, we have some separate action that is required on a given collection.  Rename to make more generic.\n\n* Remove restriction that a ManualAction needs a get or update value.  The manual action could just be locating data for another collection downstream.\n\nCache email template details, even if there are no actions needed on that specific collection,\n\n* Update the expected number of collections in the email dataset.\n\n* build_masking_instructions is not required to return a ManualAction.\n\n* Reconcile this test with the work to make log send asynchronous.\n\nCo-authored-by: Sean Preston \u003csean@ethyca.com\u003e\n\n* Adds email scopes to postman collection (#1241)\n\n* Adds email scopes to postman collection\n\n* update changelog\n\n* Bump black from 22.6.0 to 22.8.0 (#1238)\n\nBumps [black](https://github.com/psf/black) from 22.6.0 to 22.8.0.\n- [Release notes](https://github.com/psf/black/releases)\n- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)\n- [Commits](https://github.com/psf/black/compare/22.6.0...22.8.0)\n\n---\nupdated-dependencies:\n- dependency-name: black\n  dependency-type: direct:development\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Rename fidesops-privacy-center image to fides-privacy-center (#1237)\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Bump eslint from 8.9.0 to 8.23.0 in /clients/ops/privacy-center (#1180)\n\nBumps [eslint](https://github.com/eslint/eslint) from 8.9.0 to 8.23.0.\n- [Release notes](https://github.com/eslint/eslint/releases)\n- [Changelog](https://github.com/eslint/eslint/blob/main/CHANGELOG.md)\n- [Commits](https://github.com/eslint/eslint/compare/v8.9.0...v8.23.0)\n\n---\nupdated-dependencies:\n- dependency-name: eslint\n  dependency-type: direct:development\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* #1008 User  identity verification flow (#1231)\n\n* Update lint commands and eslint ignore\n\n* Run lints and refactor privacy cards\n\n* Refactor modal\n\n* Finish verification flow\n\n* get config from server\n\n* Update changelong\n\n* Fix test failures\n\n* Format file\n\n* Mock out route\n\n* Format file\n\n* Add code resending\n\n* Update test to use hostUrl\n\n* Add headers util function and PrivacyRequestStatus status enum\n\n* Bump @chakra-ui/utils from 1.10.4 to 2.0.9 in /clients/ops/admin-ui (#1145)\n\nBumps [@chakra-ui/utils](https://github.com/chakra-ui/chakra-ui/tree/HEAD/packages/utils) from 1.10.4 to 2.0.9.\n- [Release notes](https://github.com/chakra-ui/chakra-ui/releases)\n- [Changelog](https://github.com/chakra-ui/chakra-ui/blob/main/packages/utils/CHANGELOG.md)\n- [Commits](https://github.com/chakra-ui/chakra-ui/commits/@chakra-ui/utils@2.0.9/packages/utils)\n\n---\nupdated-dependencies:\n- dependency-name: \"@chakra-ui/utils\"\n  dependency-type: direct:production\n  update-type: version-update:semver-major\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Allows worker to start successfully in both dev and dev_with_worker (#1250)\n\n* Allows worker to start successfully in both dev and dev_with_worker\n\n* update changelog\n\n* formatting\n\n* Add `execution_timeframe` to `Policy` model and schema (#1244)\n\n* Add execution_timeframe column to model and schema\n\n* Fix test failures\n\n* Fix Policy patch route\n\n* Update Policy patch test\n\n* Run black and isort lints\n\n* Update changelog\n\n* fix typo\n\nCo-authored-by: Sean Preston \u003csean@ethyca.com\u003e\n\n* Fix for pytest-asyncio bug (#1260)\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Clean up docker build (#1252)\n\n* Add files to .dockerignore\n\n* Add config path to docker-compose\n\n* Mount volume for build check\n\n* Fix pylint error\n\n* Update CHANGELOG\n\n* Fix failing external test\n\n* Remove worker env var from docker-compose\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Email Connector: Send Email with Erasure Instructions [#1158] (#1246)\n\n* Send an email for each email-based dataset at the end of privacy request execution.\n\n- Add a migration to create a new audit log type.  Create an audit log for the email send.\n-  Throw an exception for email-based connectors and catch to override the default execution log.\n- Add a draft of an email template\n- Connect sending a \"test email\" with dummy data.  A fidesops admin could configure to check their email config was working.\n\n* Add more \"checkpoints\" to privacy request execution - these are locations from which we can resume privacy request execution without having to run from the beginning.\n\n- Add more options to CurrentStep Enum\n- Cache the checkpoint if an email send fails, so we can retry from the same step.\n\n* Don't send an email if the connection config is read only or there are no updates to be applied to any of the collections on the dataset.\n\n* Don't assume there's a collection when building \"resume\" details. A failed privacy request can be resumed outside of the traversal.\n\n* Add a first draft of docs for setting up an email connector.\n\n* Moves the email connector send method to the email connector file.\n\n* Update mock location.\n\n* Bump downrev.\n\n* update email connector guides\n\n* correct link, broken sentence\n\n* Create a new EmailRequestFulfillmentBodyParams type to be used once the cached email details are extracted by dataset.\n\n* Fix missed test.\n\nCo-authored-by: Cole \u003ccole@ethyca.com\u003e\n\n* Fix download link (#1264)\n\n* Fix download link\n\n* Update changelog\n\n* Add `due_date` to `PrivacyRequest` model (#1259)\n\n* Add `due_date` to `PrivacyRequest` model\n\n* Add `due_date` test and run lints\n\n* Update CHANGELOG.md\n\n* Fix test failures\n\n* Fix type error\n\n* Fix mypy issue\n\n* Switch to `strptime`\n\n* Fix migration down revision\n\n* Move date format string into constant\n\n* Refactor strategy instantiation for more extensitiliby (#1254)\n\n* Instantiate strategies via abstract Strategy base class\n\nA generalized Strategy abstract base class provides generalized getter methods\nthat instantiate strategy subclasses (implementations).\nThese methods rely on the builtin __subclasses__() method to identify Strategy subclasses,\nwhich allows for more dynamic and extensible strategy implementation, removing the need\nfor a hardcoded enumeration of supported Strategy implementations.\nAbstract strategy types inherit from this new abstract base class,\nand strategy subclasses (implementations) must provide `name` and `configuration_model` attributes\nthat are leveraged by new instantiation mechanism in the abstract base class.\n\n* Update get_description() to be a class rather than static method\n\nThis allows the method to leverage the new `name` class variable rather than\nrelying on a static constant variable.\n\n* Remove strategy factories and update references\n\nStrategy factories are no longer needed with refactored Strategy getters.\nUpdate the uses (references) of strategy factories throughout the codebase\nto now rely on the new Strategy getters.\nStrategy subclasses (implementations) now need to be imported explicitly\nin __init__.py's because they used to be imported in factory modules.\nAlso remove the old MaskingStrategy registration/factory mechanisms.\n\n* Remove strategy name constants\n\nNow that the abstract Strategy base class enforces implementation subclasses\nto have a `name` class attribute, this attribute should be relied upon rather than\nthe arbitrary name constants declared previously.\nThe get_strategy_name() abstract method is also superfluous, as the `name`\nclass attribute can be used as a standardized way to retrieve the strategy name.\n\n* Remove get_configuration_model() abstract method\n\nThe generalized strategy getter now relies upon the `configuration_model`\nclass variable that's on each Strategy. Therefore we no longer need the\nget_configuration_model() getter on each Strategy subclass.\n\n* Update MaskingStrategy docs with new Strategy functionality\n\n* Update changelog\n\n* Improve recursion in _find_all_strategy_subclasses\n\n* Fix recursion bug when finding all strategies\n\nUpdate associated tests to make sure the recursion is properly tested\n\n* Tweak conditional for falsy check\n\n* Make get_strategies endpoint test more robust\n\n* Fix typo in documentation\n\nCo-authored-by: Adam Sachs \u003cadam@Adams-MacBook-Pro.local\u003e\n\n* Make admin ui work when volumes are mounted (#1266)\n\n* Make admin ui work when volumes are mounted\n\n* Update changelog\n\n* Fix path issue with hard refresh\n\n* Update from code review\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Update the Erasure Request Email Fulfillment template [#1265] (#1270)\n\n* Update email template for email request fulfilment.\n\n* Formatting changes.\n\n* Bump types-redis from 4.3.4 to 4.3.20 (#1255)\n\nBumps [types-redis](https://github.com/python/typeshed) from 4.3.4 to 4.3.20.\n- [Release notes](https://github.com/python/typeshed/releases)\n- [Commits](https://github.com/python/typeshed/commits)\n\n---\nupdated-dependencies:\n- dependency-name: types-redis\n  dependency-type: direct:development\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Update boto3 requirement from ~=1.24.46 to ~=1.24.68 (#1272)\n\nUpdates the requirements on [boto3](https://github.com/boto/boto3) to permit the latest version.\n- [Release notes](https://github.com/boto/boto3/releases)\n- [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)\n- [Commits](https://github.com/boto/boto3/compare/1.24.46...1.24.68)\n\n---\nupdated-dependencies:\n- dependency-name: boto3\n  dependency-type: direct:production\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump requests-mock from 1.9.3 to 1.10.0 (#1276)\n\nBumps [requests-mock](https://github.com/jamielennox/requests-mock) from 1.9.3 to 1.10.0.\n- [Release notes](https://github.com/jamielennox/requests-mock/releases)\n- [Commits](https://github.com/jamielennox/requests-mock/compare/1.9.3...1.10.0)\n\n---\nupdated-dependencies:\n- dependency-name: requests-mock\n  dependency-type: direct:development\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump fastapi[all] from 0.81.0 to 0.82.0 (#1277)\n\nBumps [fastapi[all]](https://github.com/tiangolo/fastapi) from 0.81.0 to 0.82.0.\n- [Release notes](https://github.com/tiangolo/fastapi/releases)\n- [Commits](https://github.com/tiangolo/fastapi/compare/0.81.0...0.82.0)\n\n---\nupdated-dependencies:\n- dependency-name: fastapi[all]\n  dependency-type: direct:production\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Fix typo (#1280)\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Pass in Vault environment variables if they are present locally (#1275)\n\n* Pass in Vault environment variables if they are present locally\n\n* Changelog and minor cleanup\n\n* Add Foundation for \"Manual Webhooks\" [#1224] (#1267)\n\n* Add a new AccessManualWebhook model, a new \"manual_webhook\" ConnectionType, and a new \"requires_input\" PrivacyRequestStatus.\n\n- Add endpoints to get/create/update/delete the single AccessManualWebhook associated with a ConnectionConfig of type \"manual_webhook\".  The new AccessManualWebhook primarily stores the fields that we are going to need to upload. The values uploaded will be passed directly to the user.\n\n* Add requires_input privacy request status to the frontend, and exposes a \"manual_webhook\" as a manual connector type.\n\n* Add the endpoints to create the manual connection config and its corresponding webhook to the postman collection. Clean up some copy/paste issues with the email connector.\n\n* Run black.\n\n* Add an empty ManualWebhookSchema to prevent a KeyError if you attempt to fetch secret types for this schema - we are not currently collecting secrets for this connector at this time but will in the futuer.\n\n* Apply suggestions from code review\n\nCo-authored-by: Paul Sanders \u003cpaul@ethyca.com\u003e\n\n* Catch possible integrity error.\n\nCo-authored-by: Paul Sanders \u003cpaul@ethyca.com\u003e\n\n* Added fixture to create a new data everytime test runs. (#1269)\n\n* Improved param_value aggregation to support one-to-many relationships (#1253)\n\n* Improved param_value aggregation to support one-to-many relationships\n\n* pylint and mypy fixes\n\n* Fixing mypy and unit tests\n\n* Adding async await to Shopify test\n\n* Fixing tests\n\n* Adding tests for generate_product_list\n\n* Updating changelog\n\n* Fixing tests\n\n* Simplifying flatten logic\n\nCo-authored-by: Adam Sachs \u003cadam@ethyca.com\u003e\n\n* Adding comment to clarify test\n\n* Renaming flatten to unpack\n\n* Move method example to docstring\n\nCo-authored-by: Adam Sachs \u003cadam@ethyca.com\u003e\nCo-authored-by: Adam Sachs \u003cadam@Adams-MacBook-Pro.local\u003e\n\n* Bump python from 3.9.13 to 3.9.14 (#1287)\n\n* Bump python from 3.9.13 to 3.9.14\n\n* Update changelog\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Add `days_left` field (#1281)\n\n* Add `days_left` field\n\n* Fix lints and update tests\n\n* Move property to model\n\n* Run black\n\n* Update CHANGELOG.md\n\n* Add new test and fix issue\n\n* Fix pylint issue\n\n* Fix brittle tests issue\n\n* Address PR feedback\n\n* Remove Masking of Redis Connection Error [#1271] (#1288)\n\n* Remove masking of redis connection error.\n\n* Update Changelog.\n\n* 1183 saas connectors shopify erasure (#1289)\n\n* Initial commit for erasure endpoints\n\n* Updated endpoints and erasure data approach\n\n* Formatting before making changes\n\n* Fixing duplicate customer_addresses\n\n* Added assertions for update verification\n\n* Fixed isort issue\n\nCo-authored-by: Hamza W \u003chamza@Hamzas-MacBook-Pro.local\u003e\nCo-authored-by: Adrian Galvan \u003cadrian@ethyca.com\u003e\n\n* 1156 email upon privacy request completion (#1282)\n\n* `days_left` field in admin UI (#1283)\n\n* Add `days_left` field\n\n* Fix lints and update tests\n\n* Move property to model\n\n* Run black\n\n* Update CHANGELOG.md\n\n* Add new test and fix issue\n\n* Fix pylint issue\n\n* Fix brittle tests issue\n\n* Add days left tag to table and details page\n\n* Update CHANGELOG.md\n\n* Remove old code left from auto-merge\n\n* Address PR feedback\n\n* Sort privacy requests by `due_date` (#1284)\n\n* Add initial sorting code\n\n* Add test stub\n\n* Finish sorting unit test\n\n* Update CHANGELOG.md\n\n* Fix mypy and pylint issues\n\n* Fix another mypy issue\n\n* Move nulls_last import\n\n* Potentially fix nulls_last mypy issue\n\n* Implement PR feedback\n\n* Address mypy and pylint issues\n\n* Update unit test to test `due_date` sort better\n\n* Human Readable Names for Connection Types [#1096] (#1297)\n\n* Expose a human readable name in the connection type endpoints.\n\nAdd a human_readable_name to the saas connector registry and a separate human readable mapping for ConnectionTypes.\n\n* Update changelog and connection type endpoint docs.\n\n* mypy\n\n* Update ConnectorTemplate test.\n\n* Mypy\n\n* Update tests/ops/models/test_connectionconfig.py\n\nCo-authored-by: Paul Sanders \u003cpaul@ethyca.com\u003e\n\nCo-authored-by: Paul Sanders \u003cpaul@ethyca.com\u003e\n\n* Change references to FIDESOPS__CONFIG_PATH to FIDES__CONFIG_PATH (#1302)\n\n* Change references to FIDESOPS__CONFIG_PATH to FIDES__CONFIG_PATH\n\n* Update changelog\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Enable Manual Webhooks in Request Execution [#1228] (#1285)\n\n* Add a method to cache data supplied for a manual webhook on a particular privacy request.\n\n* Add an endpoint to retrieve all enabled access manual webhooks.\n\n* Add an endpoint for uploading manual data corresponding to fields in a manual webhook for a given privacy request with \"requires_input\" status.\n\n* Add an endpoint to view data manually uploaded for an access manual webhook.\n\n- Add new scopes for the endpoints to upload/view manual data for webhooks.\n- Enforce that at least one field is added when defining a manual webhook, and add a fallback if no fields were defined.\n\n* Add an endpoint to resume a privacy request from \"requires_input\" status once all input has been added.  None of the fields are required, but the a key for each manual webhook still needs to exist in the cache to proceed.\n\nAs part of request execution check if data has been uploaded (data can be empty) for all manual webhooks. If True, we can proceed with request execution, otherwise, we put the PrivacyRequest in \"requires_input\" status and exits.\n\nAlso adds the manual data uploaded directly to the packet we upload to the user at the very end.\n\n* Update postman collection.\n\n* Fix request_id query param in existing postman request.\n\n* Include additional details about how to resume a \"requires_input\" privacy request when getting its status.\n\n* Add docs and update changelog.\n\n* Upload new ERD diagram.\n\n* Don't put a privacy request in requires_input state if this policy only has erasure rules.\n\n* Respond to CR!\n\n* Update manual_webhooks.md\n\n* [#1153] Adds warning to subject request denial UI (#1298)\n\n* add warning to subject request denial UI\n\n* run prettier\n\n* [#1088] Adds new Celery queue for async email dispatch (#1173)\n\nCo-authored-by: Dawn Pattison \u003cpattisdr@users.noreply.github.com\u003e\n\n* Add consent table (#1301)\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Bump pylint from 2.15.0 to 2.15.2 (#1278)\n\nBumps [pylint](https://github.com/PyCQA/pylint) from 2.15.0 to 2.15.2.\n- [Release notes](https://github.com/PyCQA/pylint/releases)\n- [Commits](https://github.com/PyCQA/pylint/compare/v2.15.0...v2.15.2)\n\n---\nupdated-dependencies:\n- dependency-name: pylint\n  dependency-type: direct:development\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Allow the user to logout with a malformed or expired token [#1257] (#1305)\n\n* Allow the user to logout with a malformed or expired token.\n\n* Fix formatting.\n\n* Fix test comment.\n\n* Update changelog.\n\n* Bump fideslib version to raise a 403 if the supplied token is malformed instead of a 500.\n\n* Allow the root user to logout.\n\n* Bump faker from 14.1.0 to 14.2.0 (#1315)\n\nBumps [faker](https://github.com/joke2k/faker) from 14.1.0 to 14.2.0.\n- [Release notes](https://github.com/joke2k/faker/releases)\n- [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md)\n- [Commits](https://github.com/joke2k/faker/compare/v14.1.0...v14.2.0)\n\n---\nupdated-dependencies:\n- dependency-name: faker\n  dependency-type: direct:development\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Update fastapi-pagination[sqlalchemy] requirement (#1316)\n\nUpdates the requirements on [fastapi-pagination[sqlalchemy]](https://github.com/uriyyo/fastapi-pagination) to permit the latest version.\n- [Release notes](https://github.com/uriyyo/fastapi-pagination/releases)\n- [Commits](https://github.com/uriyyo/fastapi-pagination/compare/0.9.3...0.10.0)\n\n---\nupdated-dependencies:\n- dependency-name: fastapi-pagination[sqlalchemy]\n  dependency-type: direct:production\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump types-urllib3 from 1.26.23 to 1.26.24 (#1317)\n\nBumps [types-urllib3](https://github.com/python/typeshed) from 1.26.23 to 1.26.24.\n- [Release notes](https://github.com/python/typeshed/releases)\n- [Commits](https://github.com/python/typeshed/commits)\n\n---\nupdated-dependencies:\n- dependency-name: types-urllib3\n  dependency-type: direct:development\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump pandas from 1.4.3 to 1.4.4 (#1318)\n\nBumps [pandas](https://github.com/pandas-dev/pandas) from 1.4.3 to 1.4.4.\n- [Release notes](https://github.com/pandas-dev/pandas/releases)\n- [Changelog](https://github.com/pandas-dev/pandas/blob/main/RELEASE.md)\n- [Commits](https://github.com/pandas-dev/pandas/compare/v1.4.3...v1.4.4)\n\n---\nupdated-dependencies:\n- dependency-name: pandas\n  dependency-type: direct:production\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Manual Webhook Followup: UI scopes / Delete [#1228] (#1308)\n\n* Add new scopes related to uploading privacy request data or viewing privacy request data to the UI.\n\n* Set up manual webhooks to cascade delete if their connection config is deleted.\n\n* Mypy issues.\n\n* Utility to update saas configs and datasets on system startup (#1307)\n\n* Add utility to update SaaS connector instances based on template updates\n\nUtility is invoked on server bootstrap.\nUpdates made to the connector template registry over time are meant to be\nautomatically applied to existing SaaS connector instance configurations\nthat are already present in the DB.\n\n* Refactor SaaS config instantation to use shared helpers\n\n* Update changelog\n\n* Improve SaaS config update logic to safely update configs\n\nInstead of deleting existing configs, update in place.\nSimplify logic to not validate secrets unnecessarily.\nRevert intsantation endpoint to existing implementation as it no longer\nshares functionality with update logic.\n\n* Improve db session management to use and close shared session\n\n* Update fixture to not rely on API endpoints\n\nRevert instantantiation endpoint test back to its original functionality\n\n* Remove new get_config classmethod as it is no longer needed\n\n* Adjust dataset creation function name for clarity\n\n* Fix up outdated mock function reference\n\n* Add test coverage for removal or addition of a connector param\n\nInclude some variable renaming to try to clarify test cases\n\nCo-authored-by: Adam Sachs \u003cadam@Adams-MacBook-Pro.local\u003e\n\n* 1154 send email upon privacy request receipt (#1303)\n\n* Add generic request sorting button (#1320)\n\n* Add generic request sort button\n\n* Update changelog\n\n* Implement PR feedback\n\n* Rename PrivacyRequestIdentity schema to Identity (#1324)\n\n* Rename PrivacyRequestIdentity schema to Identity\n\n* Update changelog\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* 1155 email upon privacy request review (#1306)\n\n* Manual Webhook Followup: UI Sync [#1326] (#1323)\n\n* Make a manual webhook \"testable\" - which just asserts that a valid manual webhook configuration exists.\n\n* Allow dsr_package_label to be an empty string but then convert to match pii field in that case.\n\n* Changelog\n\n* Respond to CR\n\n* Fix mypy error.\n\n* Explicitly install the `toml` package (#1338)\n\n* Explicitly install `toml` as a requirement\n\nPreviously this package was included as an indirect dependency only, but\nimported in source code as if it were a direct dependency. This led to\ninstallation issues for consumers of the `fidesops` `pip` package.\n\n* Reduce `toml` import volume\n\n* Update `CHANGELOG.md`\n\n* Add consent request table (#1340)\n\n* Add consent crequest table\n\n* Update changelog\n\n* Fix isort and black\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Updated Configuration settings: Connector parameters and Dataset configuration (#1247)\n\n* Added Saas dataset configuration\n\n* Edits to Saas dataset configuration\n\n* Edits to Saas dataset configuration\n\n* Edits to Saas dataset configuration\n\n* Refactor Connector Parameters\n\n* Refactor Connector Parameters\n\n* Refactor Connector Parameters\n\n* Refactor Connector Parameters\n\n* Reset Connection state when AddConnection component is unmounted\n\n* Added Connection Identifier validation\n\n* Updated ConnectionIdentifier validation to be removed from the JSX markup inline\n\n* Added the verify=false to the query string for the API route to update the connection config secrets. This is to not connect directly to a given DB connector.\n\n* Refactored Connector Parameters\n\n* Reset Connection object state when the ConfigureConnector component is mounted\n\n* Updated Connector Parameters\n\n* Updated the toast notification placement to be positioned via top\n\n* Updated CHANGELOG.md file\n\n922 - Add a Connector - DB connector configs\n923 - Add a Connector - upload a DB Dataset YAML\n1090 - Add a Connector - SaaS Dataset Management (YAML method)\n\n* Updated CHANGELOG.md file\n\n* Resolved Prettier format issue\n\n* Resolved npm run build issues\n\n* add permissions to list\n\n* Refactored Dataset Configuration\n\n* Refactored Dataset Configuration\n\n* Refactored Dataset Configuration\n\n* Refactored Dataset Configuration\n\n* 1015 - Frontend - Configure a Manual entry connector\n\n* 1015 - Frontend - Configure a Manual entry connector\n\n* 1015 - Frontend - Configure a Manual entry connector\n\n* 1015 - Frontend - Configure a Manual entry connector\n\n* Updated the UI to display the human readable value for the selected Connector\n\n* Code review feedback changes\n\n* Remove commented out code which is unnecessary\n\n* Redirect the user back to the Datastore Connections landing page\n\n* Updated DSR Customization user instruction verbiage\n\n* Removed hard coded string literals in ConfigureConnector component\n\n* If a connection has been initially created, then auto redirect the user accordingly\n\n* Resolved issue with ConnectionsEmptyState component not be displayed when there are no connections\n\n* Reverted fidesops.toml file\n\n* add scopes to allowed list of permissions\n\nCo-authored-by: Sean Preston \u003csean@ethyca.com\u003e\n\n* pin version of pyjwt (#1343)\n\n* Upgrades `pyjwt` to `2.5.0` (#1344)\n\n* pin version of pyjwt\n\n* assert against encoded identity not hardcoded\n\n* Fixing Stripe Tests (#1352)\n\n* Updating assertions and using a unique erasure identity email per test run\n\n* Fixing sort order\n\n* Fix Corner Case: Retrieving Access Results from Cache on Retry [#1348] (#1349)\n\n* Fix bug where erasure count confirmations may be returned instead of selected access request results when attempting to get access requests from the cache.\n\nBecause erasure requests are run after access requests, there normally isn't erasure results in the cache when we attempt to get access results.  However, if we were retrying a privacy request from a checkpoint that had already erased data, we could possibly return erasure results instead of access results, depending on if the access or erasure key was accessed last.\n\n* Update CHANGELOG.\n\n* Bump pytest from 7.1.2 to 7.1.3 (#1342)\n\nBumps [pytest](https://github.com/pytest-dev/pytest) from 7.1.2 to 7.1.3.\n- [Release notes](https://github.com/pytest-dev/pytest/releases)\n- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)\n- [Commits](https://github.com/pytest-dev/pytest/compare/7.1.2...7.1.3)\n\n---\nupdated-dependencies:\n- dependency-name: pytest\n  dependency-type: direct:development\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump types-ujson from 5.4.0 to 5.5.0 (#1355)\n\nBumps [types-ujson](https://github.com/python/typeshed) from 5.4.0 to 5.5.0.\n- [Release notes](https://github.com/python/typeshed/releases)\n- [Commits](https://github.com/python/typeshed/commits)\n\n---\nupdated-dependencies:\n- dependency-name: types-ujson\n  dependency-type: direct:development\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* v1.8.0 Release Checklist (#1345)\n\n* updates changelog\n\n* updates ERD\n\n* add an execution_timeframe to quickstart policies\n\n* turn feature flag on\n\n* format flags\n\n* Manual Webhooks: Surface if Data Received even if Empty [#1350] (#1339)\n\n* For UI purposes, when retrieving any manually input data, distinguish whether the user has confirmed they looked for data, in that they've manually uploaded something, even if everything was empty. In this case: checked=True.\n\n* Prevent entering a pii or dsr_label as \"      \".\n\n* Fix unit test.\n\n* Update Changelog.\n\nCo-authored-by: Sean Preston \u003csean@ethyca.com\u003e\n\n* 1310- use existing celery queue for all email dispatch (#1341)\n\n* 1310- use existing celery task for all email dispatch\n\n* changelog\n\n* force sync for some use cases, update tests accordingly\n\n* format\n\n* fix integration tests\n\n* Fix Redis Cache Expiring Prematurely in Tests  (#1358)\n\n* Reset the cache expiration in testing so it doesn't potentially affect other long-running tests that depend on the cache.\n\n* Update Changelog.\n\n* 334 api key authentication strategy (#1331)\n\n* Add api_key authentication strategy\n\n* Update SaaS configs to use new api_key auth strategy\n\n* Update changelog\n\n* Simplify api_key auth config by requiring lists for fields\n\nUpdate field names to be plural to reflect that they are multi-value lists/arrays.\n\n* Remove query_param auth strategy\n\nThe query_param auth strategy can now easily be implemented as an\n`api_key` auth strategy, which provides the same functionality and more.\n\n* Update api_key auth model to reference Header model class\n\n* Update changelog to include query_param auth strategy removal note\n\nCo-authored-by: Adam Sachs \u003cadam@Adams-MacBook-Pro.local\u003e\n\n* Initial Refactor for Consent Intake UI (#1363)\n\n* Refactor Card markup out and create ConsentCard\n\n* Refactor Modals and create Consent page items\n\n* Make `VerificationForm` generic\n\n* Remove consent code\n\n* Remove consent.svg\n\n* Remove consent config options\n\n* Update CHANGELOG.md\n\n* Switch policy_key back to download\n\n* Updating offset pagination strategy to parse numeric strings as integers (#1364)\n\n* Rollbar (Access) (#1361)\n\n* Updating changelog (#1366)\n\n* Removed connector from repository (#1372)\n\n* Removed connector from repository\n\n* Remove unused test fixture\n\n* Delete unused integration test\n\n* Update pytest markers\n\n* Remove connector dataset file\n\n* Update CHANGELOG.md\n\n* Fix number of connectors in SaaS tests\n\nCo-authored-by: Neville Samuell \u003cneville@ethyca.com\u003e\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Add a Timescale Connector [#1311] (#1327)\n\n* Add a Timescale POC using the existing postgres connector.\n\n* Add some basic tests for timescale including verifying we have connected and populated the database, and a simple access and erasure request.\n\n* Use the setup script to populate.\n\n* Update host.\n\n* Update the port for timescale testing. Should be 5432 because we are inside the network, not outside.\n\n* Update changelog.\n\n* Remove breakpoint.\n\n* Create a timescale hypertable and demonstrate running an access query.\n\n* Demonstrate masking of records in the timescale onsite_personnel hypertable.\n\n* Update data/dataset/timebase_example_test_dataset.yml\n\nCo-authored-by: Paul Sanders \u003cpaul@ethyca.com\u003e\n\n* Close a timescale session and add supporting code comments for why we're subclassing the postgresql connector.\n\n* Use uuid's for privacy request id's in the test_sql_task file\n\nCo-authored-by: Paul Sanders \u003cpaul@ethyca.com\u003e\n\n* Set Schema for Postgres Connector [#1362] (#1375)\n\n* Allow a schema to be specified for Postgres beyond the default schema.\n\n- Add db_schema as an optional postgres secret.\n- Add an empty set_schema method on the Base SQL Connector that can optionally be added on a subclass to set a schema for an entire session.\n- Define PostgreSQLConnector.set_schema to set the search path\n- Add a required secrets_schema property to be defined on every SQLConnector\n- Move \"create_client\" to the base SQLConnector and remove most locations where it was overridden because the primary element that was changing was the secrets schema.\n- Remove Redshift overrides for retrieve_data and mask_data since their only purposes are to set the schema, which the base sql connector can now do.\n\n* Update CHANGELOG.\n\n* Update the secrets format in testing now that db_schema can be optionally set.\n\n* Update separate missed test concerning new db_schema secrets attribute\n\n* Update CHANGELOG.md\n\n* Random import removed from this file.\n\n* move everything into src/fides/\n\n* update code paths for ops\n\n* update fidesctl imports\n\n* run isort\n\n* Unified fides (#1121)\n\n* Delete Datastore [Frontend] (#683)\n\n* Link Delete Connection menu item with confirmation modal, patterned off of user delete modal.\n\n* Update changelog.\n\n* Vertically center modals.\n\n* Add a spinner while collection is being deleted. Only close modal after successful deletion.\n\n* Prevent closing modal while we're still making the delete request.\n\n* 546 - [Admin UI] Change \"Policy Name\" to \"Request Type\" on SR list page (#696)\n\n* 546 - [Admin UI] Change \"Policy Name\" to \"Request Type\" on SR list page\n\n* Rollback file change\n\n* Updated CHANGELOG.md file\n\nCo-authored-by: Christopher Calhoun \u003cchris@ethyca.com\u003e\n\n* Enable/Disable Datastores [Frontend] (#693)\n\n* Add the ability to enable/disable a connectionconfig.\n\n* Fix other location to prevent from closing modal while in progress.\n\n* Update changelog.\n\n* 659 Add Postgres and Redis to health endpoint (#690)\n\n* adds db and redis to health endpoint, untested\n\n* updates typing, suppresses sqlalchemy and alembic logs for healthchecks\n\n* adds to makefile, updates docs and tests\n\n* format\n\n* adds check for cache enabled, update tests\n\n* format\n\n* standardize health endpoint structure\n\n* Execute Privacy Requests with Celery (#621)\n\n Updates the way privacy requests are dispatched into processing from a background process into a Celery task\n\n* 512 db redis health (#686)\n\n* catches exceptions from db/redis and provides better errors on startup\n\n* move health endpoint changes to separate branch\n\n* update changelog\n\n* catch appropriate redis err, stop app upon db and cache connection failure\n\n* adds check for cache enabled\n\n* Datastore Connection Filtering (#691)\n\n* Refactor routes into enums and create connections page\n\n* Test switching back to double quotes\n\n* Convert back to double quotes\n\n* Add placeholder connection filters\n\n* Set up api scaffolding\n\n* Get basic grid going\n\n* Initial grid card styling\n\n* Fix simple eslint issues\n\n* Add development config back in\n\n* Finish draft of card\n\n* Add working test button and landing page\n\n* Add pagination and small fixes\n\n* Fix testing issues\n\n* Add auth tests for datastore connection page\n\n* run formatter\n\n* Update changelog\n\n* update the create_test_data command to add connectionconfigs\n\n* Disable create buttons \u0026 fix text overflow\n\n* Update filter dropdown values\n\n* Fix test timestamp bug\n\n* Remove development variable\n\n* Add working filter dropdowns\n\n* Add outside click hook \u0026 polish things\n\n* Fix imports\n\n* Update changelog\n\n* Update button hover color\n\n* remove commented out code\n\n* fix typo\n\n* Remove Saas Option\n\n* Fix welcome screen bug\n\n* Remove edit button\n\n* Fix lint and formatting issues\n\n* removes commented-out code\n\nCo-authored-by: Sean Preston \u003csean@ethyca.com\u003e\nCo-authored-by: eastandwestwind \u003ceastandwestwind@gmail.com\u003e\n\n* update domain -\u003e host config references in stripe and sentry test infrastructure (#698)\n\nCo-authored-by: Adam Sachs \u003cadam@Adams-MacBook-Pro.local\u003e\nCo-authored-by: Adrian Galvan \u003cadriang430@gmail.com\u003e\n\n* Clean up postman collection (#704)\n\n* Revoke a Pending Privacy Request [#525] (#592)\n\n* Add the ability to cancel a pending task.  The celery task is not actually cancelled yet.\n\n- Track cancel reason, datetime cancelled, and add a new cancelled status.\n\n* Add drp revoke request to postman collection.\n\n* Add drp revoke docs.\n\n* Update down_rev after rebase.\n\n* Fix incorrect check.\n\n* Restore new canceled state.\n\n* Check that the privacy request is not canceled right before starting execution.  This is really our last chance to check before we start executing the graph in dask.    The use case here might be it was canceled shortly after it was approved.\n\n* Attempt to revoke a queued celery task if we cancel it before it starts executing.\n\n* Prettier.\n\n* Changelog updated.\n\n* Add a few unit tests around how triggering the run_privacy_request_task with a cancelled task id doesn't do anything and how you can't approve a canceled privacy request.\n\n* Fix SQLAlchemy logging to console - logging in migration propagates to the rest of the application.\n\n* Refresh session instead of creating a new one.\n\n* Add 200 character limit.\n\n* Add some assertions that db.refresh is doing what we think it's doing.\n\n* Update CHANGELOG.md, bump to `fideslib==2.1.0`  (#705)\n\n* bump version of fideslib\n\n* updates changelog pre-release\n\n* Update datastore connection filters to support clearing (#701)\n\n* Update connection filters to clear\n\n* Use new common header function for analytics\n\n* Update prepareHeaders function to make type checker happy\n\n* Update changelog\n\n* Management UI updates (#702)\n\n* reorder ui docs\n\n* admi ui\n\n* subject request detail finalization, datastore stub\n\n* config edit\n\n* comment out docs not implemented\n\n* remove tracking\n\n* changelog\n\n* remove stray comment, add Canceled status\n\n* Remove stray hyphen (#709)\n\n* Reduce docker image size (#707)\n\n* Reduce docker image size\n\n* Update CHANGELOG\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Bump next-auth from 4.3.2 to 4.5.0 in /clients/privacy-center (#694)\n\nBumps [next-auth](https://github.com/nextauthjs/next-auth) from 4.3.2 to 4.5.0.\n- [Release notes](https://github.com/nextauthjs/next-auth/releases)\n- [Changelog](https://github.com/nextauthjs/next-auth/blob/main/CHANGELOG.md)\n- [Commits](https://github.com/nextauthjs/next-auth/compare/next-auth@v4.3.2...next-auth@v4.5.0)\n\n---\nupdated-dependencies:\n- dependency-name: next-auth\n  dependency-type: direct:production\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* update tutorial directions to install fidesdemo from the root directory (#715)\n\n* update tutorial directions to install fidesdemo from the root directory\n\n* changelog\n\n* [SaaS Connector] Salesforce (access) (#676)\n\nCo-authored-by: Adam Sachs \u003cadam@Adams-MacBook-Pro.local\u003e\nCo-authored-by: Hamza W \u003chamza@Hamzas-MacBook-Pro.local\u003e\n\n* Parallelize CI safe checks (#717)\n\n* Parallelize CI safe checks\n\n* Update CHANGELOG\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Make reading of environment variables case insensitive (#712)\n\n* Make reading of environment variables case insensitive\n\n* Update CHANGELOG\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Add dependabot (#718)\n\n* Add dependabot\n\n* Update CHANGELOG\n\n* Update CHANGELOG\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Bump requests-mock from 1.8.0 to 1.9.3 (#732)\n\nBumps [requests-mock](https://github.com/jamielennox/requests-mock) from 1.8.0 to 1.9.3.\n- [Release notes](https://github.com/jamielennox/requests-mock/releases)\n- [Commits](https://github.com/jamielennox/requests-mock/compare/1.8.0...1.9.3)\n\n---\nupdated-dependencies:\n- dependency-name: requests-mock\n  dependency-type: direct:development\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump packaging from 20.9 to 21.3 (#733)\n\nBumps [packaging](https://github.com/pypa/packaging) from 20.9 to 21.3.\n- [Release notes](https://github.com/pypa/packaging/releases)\n- [Changelog](https://github.com/pypa/packaging/blob/main/CHANGELOG.rst)\n- [Commits](https://github.com/pypa/packaging/compare/20.9...21.3)\n\n---\nupdated-dependencies:\n- dependency-name: packaging\n  dependency-type: direct:development\n  update-type: version-update:semver-major\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump types-redis from 4.2.6 to 4.3.2 (#729)\n\nBumps [types-redis](https://github.com/python/typeshed) from 4.2.6 to 4.3.2.\n- [Release notes](https://github.com/python/typeshed/releases)\n- [Commits](https://github.com/python/typeshed/commits)\n\n---\nupdated-dependencies:\n- dependency-name: types-redis\n  dependency-type: direct:development\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump faker from 8.12.1 to 13.14.0 (#730)\n\nBumps [faker](https://github.com/joke2k/faker) from 8.12.1 to 13.14.0.\n- [Release notes](https://github.com/joke2k/faker/releases)\n- [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md)\n- [Commits](https://github.com/joke2k/faker/compare/v8.12.1...v13.14.0)\n\n---\nupdated-dependencies:\n- dependency-name: faker\n  dependency-type: direct:development\n  update-type: version-update:semver-major\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump actions/setup-python from 2 to 4 (#724)\n\nBumps [actions/setup-python](https://github.com/actions/setup-python) from 2 to 4.\n- [Release notes](https://github.com/actions/setup-python/releases)\n- [Commits](https://github.com/actions/setup-python/compare/v2...v4)\n\n---\nupdated-dependencies:\n- dependency-name: actions/setup-python\n  dependency-type: direct:production\n  update-type: version-update:semver-major\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump actions/checkout from 2 to 3 (#723)\n\nBumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3.\n- [Release notes](https://github.com/actions/checkout/releases)\n- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)\n- [Commits](https://github.com/actions/checkout/compare/v2...v3)\n\n---\nupdated-dependencies:\n- dependency-name: actions/checkout\n  dependency-type: direct:production\n  update-type: version-update:semver-major\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump github/codeql-action from 1 to 2 (#725)\n\nBumps [github/codeql-action](https://github.com/github/codeql-action) from 1 to 2.\n- [Release notes](https://github.com/github/codeql-action/releases)\n- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)\n- [Commits](https://github.com/github/codeql-action/compare/v1...v2)\n\n---\nupdated-dependencies:\n- dependency-name: github/codeql-action\n  dependency-type: direct:production\n  update-type: version-update:semver-major\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump docker/login-action from 1 to 2 (#722)\n\nBumps [docker/login-action](https://github.com/docker/login-action) from 1 to 2.\n- [Release notes](https://github.com/docker/login-action/releases)\n- [Commits](https://github.com/docker/login-action/compare/v1...v2)\n\n---\nupdated-dependencies:\n- dependency-name: docker/login-action\n  dependency-type: direct:production\n  update-type: version-update:semver-major\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump actions/setup-node from 2 to 3 (#726)\n\nBumps [actions/setup-node](https://github.com/actions/setup-node) from 2 to 3.\n- [Release notes](https://github.com/actions/setup-node/releases)\n- [Commits](https://github.com/actions/setup-node/compare/v2...v3)\n\n---\nupdated-dependencies:\n- dependency-name: actions/setup-node\n  dependency-type: direct:production\n  update-type: version-update:semver-major\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* [#716] update datastore documentation (#742)\n\n* Adds `fidesops worker` command to start the Celery worker [#663] (#673)\n\n* add fidesops worker cmd\n\n* rename Dockerfiles, adds worker specific dockerfile with no frontend\n\n* formatting\n\n* update container names, add separate dockerfile for worker\n\n* pass worker args through to worker_main\n\n* rename containers fidesops -\u003e webserver, celery -\u003e worker\n\n* fix imports\n\n* add return type\n\n* update dockerfile\n\n* describe changelog\n\n* specify same argv for all celery instantiation\n\n* add logging to cli\n\n* add back .egg\n\n* specify dockerfile to build from\n\n* 708 - fix console warning in disable connections (#750)\n\n* removes unneeded code in disable connection\n\n* Bump fideslib from version 2.1.0 to 2.1.1 (#721)\n\n* Bump fideslib from version 2.1.0 to 2.1.1\n\n* Update CHANGELOG\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Password Hashing update (#749)\n\n* Fix no such container error with docker-compose (#758)\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\nCo-authored-by: Sean Preston \u003csean@ethyca.com\u003e\n\n* 1.6.1 release (#763)\n\n* Update CHANGELOG for release\n\n* Update CHANGELOG\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\nCo-authored-by: Sean Preston \u003csean@ethyca.com\u003e\n\n* Bump psycopg2-binary from 2.9.1 to 2.9.3 (#783)\n\nBumps [psycopg2-binary](https://github.com/psycopg/psycopg2) from 2.9.1 to 2.9.3.\n- [Release notes](https://github.com/psycopg/psycopg2/releases)\n- [Changelog](https://github.com/psycopg/psycopg2/blob/master/NEWS)\n- [Commits](https://github.com/psycopg/psycopg2/commits)\n\n---\nupdated-dependencies:\n- dependency-name: psycopg2-binary\n  dependency-type: direct:production\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump dask from 2021.10.0 to 2022.6.1 (#781)\n\nBumps [dask](https://github.com/dask/dask) from 2021.10.0 to 2022.6.1.\n- [Release notes](https://github.com/dask/dask/releases)\n- [Changelog](https://github.com/dask/dask/blob/main/docs/release-procedure.md)\n- [Commits](https://github.com/dask/dask/compare/2021.10.0...2022.6.1)\n\n---\nupdated-dependencies:\n- dependency-name: dask\n  dependency-type: direct:production\n  update-type: version-update:semver-major\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump alembic from 1.6.5 to 1.8.0 (#780)\n\nBumps [alembic](https://github.com/sqlalchemy/alembic) from 1.6.5 to 1.8.0.\n- [Release notes](https://github.com/sqlalchemy/alembic/releases)\n- [Changelog](https://github.com/sqlalchemy/alembic/blob/main/CHANGES)\n- [Commits](https://github.com/sqlalchemy/alembic/commits)\n\n---\nupdated-dependencies:\n- dependency-name: alembic\n  dependency-type: direct:production\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump sqlalchemy-bigquery from 1.3.0 to 1.4.4 (#779)\n\nBumps [sqlalchemy-bigquery](https://github.com/googleapis/python-bigquery-sqlalchemy) from 1.3.0 to 1.4.4.\n- [Release notes](https://github.com/googleapis/python-bigquery-sqlalchemy/releases)\n- [Changelog](https://github.com/googleapis/python-bigquery-sqlalchemy/blob/main/CHANGELOG.md)\n- [Commits](https://github.com/googleapis/python-bigquery-sqlalchemy/compare/v1.3.0...v1.4.4)\n\n---\nupdated-dependencies:\n- dependency-name: sqlalchemy-bigquery\n  dependency-type: direct:production\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump pytest-cov from 2.11.1 to 3.0.0 (#787)\n\nBumps [pytest-cov](https://github.com/pytest-dev/pytest-cov) from 2.11.1 to 3.0.0.\n- [Release notes](https://github.com/pytest-dev/pytest-cov/releases)\n- [Changelog](https://github.com/pytest-dev/pytest-cov/blob/master/CHANGELOG.rst)\n- [Commits](https://github.com/pytest-dev/pytest-cov/compare/v2.11.1...v3.0.0)\n\n---\nupdated-dependencies:\n- dependency-name: pytest-cov\n  dependency-type: direct:development\n  update-type: version-update:semver-major\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump apscheduler from 3.8.0 to 3.9.1 (#789)\n\nBumps [apscheduler](https://github.com/agronholm/apscheduler) from 3.8.0 to 3.9.1.\n- [Release notes](https://github.com/agronholm/apscheduler/releases)\n- [Changelog](https://github.com/agronholm/apscheduler/blob/3.9.1/docs/versionhistory.rst)\n- [Commits](https://github.com/agronholm/apscheduler/compare/3.8.0...3.9.1)\n\n---\nupdated-dependencies:\n- dependency-name: apscheduler\n  dependency-type: direct:production\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump types-pyyaml from 6.0.8 to 6.0.9 (#791)\n\nBumps [types-pyyaml](https://github.com/python/typeshed) from 6.0.8 to 6.0.9.\n- [Release notes](https://github.com/python/typeshed/releases)\n- [Commits](https://github.com/python/typeshed/commits)\n\n---\nupdated-dependencies:\n- dependency-name: types-pyyaml\n  dependency-type: direct:development\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* [#681] add documentation on fideslog use (#751)\n\n* [#681] add documentation on fideslog use\n\n* changelog\n\n* update fidesops use wording\n\n* Fideslib models (#700)\n\n* WIP\n\n* WIP\n\n* WIP\n\n* WIP\n\n* Use available exceptions from fideslib\n\n* Fix failing tests\n\n* Fix policy tests\n\n* Remove debugging code\n\n* Fix failing tests\n\n* Fix failing tests\n\n* Fix failiing tests\n\n* Run black and isort\n\n* Make pylint in docker happy\n\n* Clean up migrations\n\n* Move downgrade point of table renames\n\n* Remove Dockerfile temp workaround and fix pylint errors\n\n* Fix failing tests\n\n* Remove jwt.py\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* 660 Add support for multiple statuses to be selected for filtering subject requests  (#802)\n\nCo-authored-by: Dawn Pattison \u003cpattisdr@users.noreply.github.com\u003e\n\n* Resolve issue with MyPy seeing files in fidesops as missing imports (#719)\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* 249 saas connector zendesk ticket erasure (#775)\n\nCo-authored-by: Hamza W \u003chamza@Hamzas-MacBook-Pro.local\u003e\nCo-authored-by: Adrian Galvan \u003cadriang430@gmail.com\u003e\n\n* Fixing `check-migration` command (#806)\n\n* Fix issue requiring separate install of snowflake-connector-python (#807)\n\n* Fix issue requiring separate install of snowflake-connector-python\n\n* Update CHANGELOG\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* 513 - [Admin UI] Update Subject Request status filter to be a multise… (#764)\n\nCo-authored-by: Christopher Calhoun \u003cchris@ethyca.com\u003e\n\n* 706 Adds SaaS connection type to SaaS yaml config (#748)\n\n* Adds Saas type to saas yaml config\n\n* To allow data migration that queries on a connectiontype enum to work, update previous schema migrations that used \"alter type\" to update connection type to rename the enum type, create a new enum with the new types, and then delete the old enum type.\n\nCo-authored-by: Dawn Pattison \u003cpattisdr@users.noreply.github.com\u003e\n\n* Make `worker` node optional (#770)\n\n* default fidesops to running the worker and webserver on same container\n\n* default to using a worker, add docker config for worker\n\n* update changelog\n\n* USE_DEDICATED_WORKER -\u003e WORKER_ENABLED\n\n* add basic descriptions for celery vars to docs\n\n* remove unused import\n\n* add Make command for a server + worker\n\n* GET Available Connectors [#706] (#768)\n\n* Adds Saas type to saas yaml config\n\n* alter postman collection\n\n* updates changelog\n\n* lint fixes\n\n* Add endpoint to surface all available connectors including database options and saas options.\n\n* Exclude custom and manual types from list of available connectors.\n\n- Add docs and postman collection.\n\n* Update changelog.\n\n* Remove committed ANALYTICS_ID.\n\n* Import ClientDetail from fideslib instead of fidesops.\n\n* Fix import order.\n\nCo-authored-by: eastandwestwind \u003ceastandwestwind@gmail.com\u003e\n\n* Endpoint: Return Secrets for a Connector Type [#753] (#795)\n\n* Adds Saas type to saas yaml config\n\n* alter postman collection\n\n* updates changelog\n\n* lint fixes\n\n* Add endpoint to surface all available connectors including database options and saas options.\n\n* Exclude custom and manual types from list of available connectors.\n\n- Add docs and postman collection.\n\n* Update changelog.\n\n* Add an endpoint to fetch the types of secrets that should be supplied for a given connection type.\n\n- Relocate \"load_config\" which we use to load saas config yamls, now that we have another use case beyond unit tests.\n\n* Dynamically override the SaaSSchema docstring for a given saas connector type, so the description isn't abstract.\n\n- Update changelog\n- Add docs\n- Add endpoint to postman collection\n\n* Add missing import.\n\n* Add a request method to docs.\n\n* Update docstring.\n\n* Remove committed ANALYTICS_ID.\n\n* Import ClientDetail from fideslib instead of fidesops.\n\n* Fix import order.\n\n* Restore removed items in changelog.\n\nCo-authored-by: eastandwestwind \u003ceastandwestwind@gmail.com\u003e\n\n* Add fixture to clear tables between test (#680)\n\n* Add fixture to clear tables between test runs\n\n* Update CHANGELOG\n\n* Add missing ordering to customer_details logs query in test.\n\n* update import path\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\nCo-authored-by: Dawn Pattison \u003cpattisdr@users.noreply.github.com\u003e\nCo-authored-by: Sean Preston \u003csean@ethyca.com\u003e\n\n* Replace user authentication routes with fideslib routes (#811)\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* test fixing publish_docs ci action (#818)\n\n* test fixing publish_docs ci action\n\n* update branch target to\n\n* try another commit to see if publish_docs is triggered\n\n* revert change to branch target\n\n* allow publish docs on test branch (#819)\n\n* allow publish docs on test branch\n\n* remove main\n\n* trying new empty commit\n\n* reverting to main\n\n* Temporarly disable paths\n\n* Revert temporary test\n\n* Add makefile\n\n* pull latest\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Update Celery config defaults (#808)\n\n* update celery config defaults, set to redis settings if none provided\n\n* rename REDIS_CONNECTION_URL to CONNECTION_URL\n\n* add var to config reference\n\n* make DB index optional, use f'string\n\n* adds support for redis USER\n\n* add default for db_index in format string\n\n* set correct default for USER\n\n* Bump next-auth from 4.5.0 to 4.9.0 in /clients/privacy-center (#823)\n\nBumps [next-auth](https://github.com/nextauthjs/next-auth) from 4.5.0 to 4.9.0.\n- [Release notes](https://github.com/nextauthjs/next-auth/releases)\n- [Changelog](https://github.com/nextauthjs/next-auth/blob/main/CHANGELOG.md)\n- [Commits](https://github.com/nextauthjs/next-auth/compare/next-auth@v4.5.0...next-auth@v4.9.0)\n\n---\nupdated-dependencies:\n- dependency-name: next-auth\n  dependency-type: direct:production\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump fideslib to handle base64 encoded password (#820)\n\n* Handle hashed password\n\n* Update CHANGELOG\n\n* Fix failing test and use UserPasswordReset schema from fideslib\n\n* Restore fidesops.toml\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* update local deployment for serving the ui [#644] (#827)\n\n* update local deployment for serving the ui\n\n* changelog\n\n* Skip Masking of Uvicorn Logs [#766] (#831)\n\n* Update get_fides_log_record_factory to skip masking of all uvicorn logs.\n\n* Update changelog.\n\n* 832 - Create new user gives HTTP 422 Unprocessable Entity exception (#833)\n\n* 832 - [User Management] Create new user gives HTTP 422 Unprocessable Entity exception\n\n* Updated CHANGELOG.md file\n\n* [Admin UI] Change Login Page Wording (#774)\n\n* sign in to instead of sign into\n\nChanging the wording of the admin ui page to improve grammatical accuracy.\n\n* fides admin-ui wording changelog\n\n* added link to pr\n\n* fix typo on subject identities\n\n* updated changelog\n\n* typo\n\n* Fix bug in client with no scopes (#830)\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\nCo-authored-by: eastandwestwind \u003ceastandwestwind@gmail.com\u003e\n\n* 599 clipboard icon (#838)\n\n* replace clipboard icon\n\n* temp fix for auth\n\n* revert last change\n\n* changelog\n\n* adds concurrency to unsafe check jobs (#835)\n\n* Reduce docker image size (#846)\n\n* Reduce docker image size\n\n* Update CHANGELOG\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* [#743] Store provided identity data in application database (#834)\n\n* adds identity fields to PrivacyRequest model\n\n* store identity data inside database\n\n* update changelog\n\n* add identities in test data command\n\n* store identities provided via the DRP creation endpoint\n\n* black + isort\n\n* store provided identity data in request creation from onetrust\n\n* remove deprecated migration\n\n* adds new provided identity table\n\n* use new provided identity table\n\n* add docstring, remove comment\n\n* update DRP privacy request creation to use ProvidedIdentity model\n\n* update identity creation in test data command\n\n* use persisted identity in OneTrust\n\n* update test to use persisted identity\n\n* isort update\n\n* use enums\n\n* optionally receive a salt in hash_value cmd\n\n* use a constant salt for provided identity hashing\n\n* remove import\n\n* use typehints\n\n* update typedef\n\n* use enum in dict\n\n* Bump faker from 13.14.0 to 13.15.0 (#848)\n\nBumps [faker](https://github.com/joke2k/faker) from 13.14.0 to 13.15.0.\n- [Release notes](https://github.com/joke2k/faker/releases)\n- [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md)\n- [Commits](https://github.com/joke2k/faker/compare/v13.14.0...v13.15.0)\n\n---\nupdated-dependencies:\n- dependency-name: faker\n  dependency-type: direct:development\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump unidecode from 1.2.0 to 1.3.4 (#849)\n\nBumps [unidecode](https://github.com/kmike/text-unidecode) from 1.2.0 to 1.3.4.\n- [Release notes](https://github.com/kmike/text-unidecode/releases)\n- [Commits](https://github.com/kmike/text-unidecode/commits)\n\n---\nupdated-dependencies:\n- dependency-name: unidecode\n  dependency-type: direct:production\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Update fastapi-pagination[sqlalchemy] requirement (#852)\n\nUpdates the requirements on [fastapi-pagination[sqlalchemy]](https://github.com/uriyyo/fastapi-pagination) to permit the latest version.\n- [Release notes](https://github.com/uriyyo/fastapi-pagination/releases)\n- [Commits](https://github.com/uriyyo/fastapi-pagination/compare/0.8.3...0.9.3)\n\n---\nupdated-dependencies:\n- dependency-name: fastapi-pagination[sqlalchemy]\n  dependency-type: direct:production\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump black from 22.3.0 to 22.6.0 (#855)\n\nBumps [black](https://github.com/psf/black) from 22.3.0 to 22.6.0.\n- [Release notes](https://github.com/psf/black/releases)\n- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)\n- [Commits](https://github.com/psf/black/compare/22.3.0...22.6.0)\n\n---\nupdated-dependencies:\n- dependency-name: black\n  dependency-type: direct:development\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Updating SaaSType enum (#857)\n\n* Serve UI from root (#720)\n\n* Refactor user management UI code (#839)\n\n* Refactor pages to use the same Layout\n\n* Add UserForm.tsx\n\n* Add Yup validation schema\n\n* Integrate refactored form to New route\n\n* Styling\n\n* WIP refactor EditUserForm\n\n* Fix EditUserForm typescript errors\n\n* Refactor handleSubmit\n\n* Small fixes to EditUserForm\n\n* Make call to get scopes earlier so form will be populated on first render\n\n* Update changelog\n\n* Search for `PrivacyRequest` based on hashed identity [#765] (#847)\n\n* adds identity fields to PrivacyRequest model\n\n* store identity data inside database\n\n* update changelog\n\n* add identities in test data command\n\n* store identities provided via the DRP creation endpoint\n\n* black + isort\n\n* store provided identity data in request creation from onetrust\n\n* remove deprecated migration\n\n* adds new provided identity table\n\n* use new provided identity table\n\n* add docstring, remove comment\n\n* update DRP privacy request creation to use ProvidedIdentity model\n\n* update identity creation in test data command\n\n* use persisted identity in OneTrust\n\n* update test to use persisted identity\n\n* isort update\n\n* use enums\n\n* optionally receive a salt in hash_value cmd\n\n* use a constant salt for provided identity hashing\n\n* remove import\n\n* use typehints\n\n* update typedef\n\n* use enum in dict\n\n* test for exact match search\n\n* added exact match search to request status api\n\n* import order\n\n* update CHANGELOG\n\n* documentation reorganization and page standardization (#858)\n\n* copy changes to fix docs outlining [#427] and standardize formatting [DOX-216]\n\n* Fix `create_test_data` (#862)\n\n* Fix create_test_data\n\n* Update CHANGELOG\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\nCo-authored-by: Sean Preston \u003csean@ethyca.com\u003e\n\n* Bump pre-commit from 2.9.3 to 2.20.0 (#853)\n\nBumps [pre-commit](https://github.com/pre-commit/pre-commit) from 2.9.3 to 2.20.0.\n- [Release notes](https://github.com/pre-commit/pre-commit/releases)\n- [Changelog](https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md)\n- [Commits](https://github.com/pre-commit/pre-commit/compare/v2.9.3...v2.20.0)\n\n---\nupdated-dependencies:\n- dependency-name: pre-commit\n  dependency-type: direct:development\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Return persisted identities in `get_request_status` view (#860)\n\n* 671 - [Datastore Management] Include Icon with datastore connection (FE) (#845)\n\n* 671 - [Datastore Management] Include Icon with datastore connection (FE)\n\n* 671 - [Datastore Management] Include Icon with datastore connection (FE)\n\n* Updated CHANGELOG.md file\n\n* 671 - [Datastore Management] Include Icon with datastore connection (FE)\n\nResolved import statement issues\n\n* 671 - [Datastore Management] Include Icon with datastore connection (FE)\n\nResolved import statement issues\n\n* 671 - [Datastore Management] Include Icon with datastore connection (FE)\n\nResolved import statement issues\n\n* 671 - [Datastore Management] Include Icon with datastore connection (FE)\n\nResolved import statement issues\n\n* 671 - [Datastore Management] Include Icon with datastore connection (FE)\n\n1. Updated connector svg images\n2. Added default style to Spinner component\n\n* 671 - [Datastore Management] Include Icon with datastore connection (FE)\n\nDeleted unnecessary file\n\n* 671 - [Datastore Management] Include Icon with datastore connection (FE)\n\nUpdated MySQL and Outreach .svg files\n\n* Renamed connectors folder to connector-logos\n\n* Resolved misspelling error in import statement\n\n* Code review feedback\n\n* Code review feedback\n\n* Populate dataset (#844)\n\n* Correct test name for mypy in safe_pr_checks.yml (#875)\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Adds `celery.toml` for loading custom Celery config [#821] (#865)\n\n* adds option to configure EVENT_QUEUE_PREFIX for celery\n\n* provide the option to specify a default queue name too\n\n* update celery config to load in from its own config toml file\n\n* updates changelog\n\n* update value for event_queue_prefix\n\n* test celery config overrides\n\n* include config_path arg\n\n* add type def\n\n* add config path to execution settings\n\n* correct values\n\n* add celery configuration to docs (#872)\n\n* update config reference\n\n* additional documentation on celery configs [#755]\n\n* add celery.toml reference\n\n* define optional celery config and overrides\n\n* reword config links\n\n* link to lowercase settings\n\n* v1.6.2 Release Checklist (#881)\n\n* updates changelog\n\n* update ERD\n\n* use star\n\n* splits IMAGE_NAME into COMPOSE_SERVICE_NAME for docker compose services as naming has diverged (#884)\n\n* belated changelog push (#885)\n\n* Move root-level docker files into docker/ subdir (#877)\n\n* Move root-level docker files into docker/ subdir\n\n* move all of the compose files\n\n* remove root aux compose files, refactor away the no-db and worker compose files\n\n* unify the app and worker dockerfiles into a single file that leverages build stages\n\n* move python scripts into a subdir, fix paths in compose integration files\n\n* fix the script tests\n\n* use python sleep instead of system sleep, fix script paths\n\n* remove the analytics_id that accicentally got committed\n\n* updated changelog\n\n* move the sample sql data to a subdir of docker/ so it can be mounted\n\n* update the teardown command and fix the integration files\n\n* fix more path typos\n\n* more desperate tweaks\n\n* fix mysql/mariadb/mongo tests\n\n* added an additional build step if mssql not there, all tests passing\n\n* fix an accidental lowercasing\n\n* Apply suggestions from code review\n\nCo-authored-by: Paul Sanders \u003cpsanders1@gmail.com\u003e\n\n* Update CHANGELOG.md\n\nCo-authored-by: Paul Sanders \u003cpsanders1@gmail.com\u003e\n\nCo-authored-by: Paul Sanders \u003cpsanders1@gmail.com\u003e\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* 745 - [Datastore Management] Keep datastore cards in 1/3 screen pattern (#873)\n\n* 745 - [Datastore Management] Keep datastore cards in 1/3 screen pattern\n\n* 745 - [Datastore Management] Keep datastore cards in 1/3 screen pattern\n\nRefactored the Datastore Management column border layout design to match Figma.\n\n* Updated CHANGELOG.md file\n\n* 45 - [Datastore Management] Keep datastore cards in 1/3 screen pattern\n\n* 745 - [Datastore Management] Keep datastore cards in 1/3 screen pattern\n\n* Fixed import statement\n\n* 793 upgrade password hash (#876)\n\n* Update config.py\n\n* Update crypto imports to fideslib and remove legacy file and tests\n\n* Update Identity salt\n\n* checkpoint, lowercasing all of the things\n\n* update more config values to lowercase\n\n* fix linting errors, fix config validators\n\n* update the configs to use lowercase keys\n\n* lowercase the allowed keys\n\n* Update create_test_data.py\n\n* bump fideslib version\n\n* Lowercase config variables\n\n* Fix linting issues\n\n* Fix some test failures\n\n* Fix application fixtures\n\n* Remove old celery config options\n\n* Sort script imports\n\n* Update changelog\n\nCo-authored-by: Thomas \u003cthomas.lapiana+github@pm.me\u003e\n\n* Feat: Vault for secrets (#869)\n\n* Replace config/gh secrets with secrets from vault\n\n* sorting and update changelog\n\n* fix path for importing test helpers\n\n* more sorting\n\n* run ci again with empty commit\n\n* update unsafe_pr_checks\n\n* allow for no vault vars for unit tests, adds back back for saas config toml vars\n\n* check for client\n\n* sort\n\n* init client to None\n\n* fix imports\n\n* fix pytest markers to better indicate which tests rely on actual secrets\n\n* Bump pandas from 1.3.3 to 1.4.3 (#896)\n\nBumps [pandas](https://github.com/pandas-dev/pandas) from 1.3.3 to 1.4.3.\n- [Release notes](https://github.com/pandas-dev/pandas/releases)\n- [Changelog](https://github.com/pandas-dev/pandas/blob/main/RELEASE.md)\n- [Commits](https://github.com/pandas-dev/pandas/compare/v1.3.3...v1.4.3)\n\n---\nupdated-dependencies:\n- dependency-name: pandas\n  dependency-type: direct:production\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Add dependabot label to dependabot PRs (#898)\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* [SaaS Connector] Salesforce (erasure) (#888)\n\n* 252 saas connector sendgrid (#883)\n\n* initial sendgrid saas connector integration. access only, contacts only\n\n* erasure (update) support for sendgrid\n\n* remove unused imports\n\n* update sendgrid test fixture to expect 404 response status code because of ignore_errors enhancement\n\n* Fixing import order and cleaning up the retry logic for consistency\n\n* add sendgrid env var support to makefile and unsafe_pr_checks config. remove DELETE endpoint per PR comments\n\n* Added delete endpoint for contacts\n\n* Fixing data_path for contacts endpoint\n\n* Reverting search query to improve performance and avoid server timeouts\n\n* Updated delete endpoint request, used request instead of SaaSRequest in tests\n\n* updated imports after check suggestion\n\n* Updated code after review\n\n* Removed unused variables, imports\n\n* Restoring Makefile\n\n* Fixed import cryptographic_util error\n\n* Misc fixes\n\n* Updated Changelog file\n\n* Updated Changelog for unreleased section and pulled main\n\n* Updated Changelog and added Sendgrid in unreleased section\n\n* Updated Changelog and added Sendgrid in unreleased section with link\n\n* Updated Changelog and added Sendgrid in added section after Adam's suggestion\n\nCo-authored-by: Adam Sachs \u003cadam@Adams-MacBook-Pro.local\u003e\nCo-authored-by: Adam Sachs \u003cadam@Adams-MBP.attlocal.net\u003e\nCo-authored-by: Adrian Galvan \u003cadriang430@gmail.com\u003e\nCo-authored-by: Hamza W \u003chamza@Hamzas-MacBook-Pro.local\u003e\nCo-authored-by: Adrian Galvan \u003cadrian@ethyca.com\u003e\n\n* 747 - Users should be able to click on the full field of a dropdown-type filter to open up the dropdown (#903)\n\n* 747 - Users should be able to click on the full field of a dropdown-type filter to open up the dropdown\n\n* Disabled eslint import/extensions rule for certain files with an alias path in the import statement\n\n* Updated CHANGELOG.md file\n\n* Resolved jest unit tests from failing\n\n* Removed eslint-disable import/extensions declarations\n\n* Bump types-redis from 4.3.2 to 4.3.4 (#895)\n\nBumps [types-redis](https://github.com/python/typeshed) from 4.3.2 to 4.3.4.\n- [Release notes](https://github.com/python/typeshed/releases)\n- [Commits](https://github.com/python/typeshed/commits)\n\n---\nupdated-dependencies:\n- dependency-name: types-redis\n  dependency-type: direct:development\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Adds saas config base info to connection config responses (#904)\n\n* adds saas config base info to connection config responses\n\n* update changelog\n\n* add properties to base class\n\n* use diff naming to avoid recursion\n\n* update saas_config prop for unit tests\n\n* another occurance of saas config in unit test\n\n* use pydantic alias for type field\n\n* sort\n\n* revert change to saas type naming\n\n* unused import\n\n* init parent class so that props are avail on self\n\n* revert to using props\n\n* Adding privacy_request_id placeholder (#911)\n\n* Bump mypy from 0.961 to 0.971 (#914)\n\nBumps [mypy](https://github.com/python/mypy) from 0.961 to 0.971.\n- [Release notes](https://github.com/python/mypy/releases)\n- [Commits](https://github.com/python/mypy/compare/v0.961...v0.971)\n\n---\nupdated-dependencies:\n- dependency-name: mypy\n  dependency-type: direct:development\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump pydash from 5.0.2 to 5.1.0 (#920)\n\nBumps [pydash](https://github.com/dgilland/pydash) from 5.0.2 to 5.1.0.\n- [Release notes](https://github.com/dgilland/pydash/releases)\n- [Changelog](https://github.com/dgilland/pydash/blob/develop/CHANGELOG.rst)\n- [Commits](https://github.com/dgilland/pydash/compare/v5.0.2...v5.1.0)\n\n---\nupdated-dependencies:\n- dependency-name: pydash\n  dependency-type: direct:production\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Update boto3 requirement from ~=1.18.14 to ~=1.24.34 (#917)\n\nUpdates the requirements on [boto3](https://github.com/boto/boto3) to permit the latest version.\n- [Release notes](https://github.com/boto/boto3/releases)\n- [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)\n- [Commits](https://github.com/boto/boto3/compare/1.18.14...1.24.34)\n\n---\nupdated-dependencies:\n- dependency-name: boto3\n  dependency-type: direct:production\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump dask from 2022.6.1 to 2022.7.0 (#915)\n\nBumps [dask](https://github.com/dask/dask) from 2022.6.1 to 2022.7.0.\n- [Release notes](https://github.com/dask/dask/releases)\n- [Changelog](https://github.com/dask/dask/blob/main/docs/release-procedure.md)\n- [Commits](https://github.com/dask/dask/compare/2022.6.1...2022.7.0)\n\n---\nupdated-dependencies:\n- dependency-name: dask\n  dependency-type: direct:production\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Fix integration set up scripts for postgres and mariadb - casing has changed for config variables. (#921)\n\n* zendesk and salesforce connection docs (#908)\n\n* Adobe Campaign access and erasure (#905)\n\n* Updated tutorial to match latest fidesdemo (#772)\n\n* Correct build arg variable name (#925)\n\n* Correct build arg variable name\n\n* Update CHANGELOG\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Default `FIDESOPS__ADMIN_UI__ENABLED` to `True` (#936)\n\n* serve AdminUI by default\n\n* updates changelog\n\n* Update python docker base image from slim-buster to slim-bullseye (#928)\n\n* Update python docker base image from slim-buster to slim-bullseye\n\n* Update CHANGELOG\n\n* Remove ipython from dev-requirements.txt\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\nCo-authored-by: Sean Preston \u003csean@ethyca.com\u003e\n\n* Update boto3 requirement from ~=1.24.34 to ~=1.24.36 (#939)\n\nUpdates the requirements on [boto3](https://github.com/boto/boto3) to permit the latest version.\n- [Release notes](https://github.com/boto/boto3/releases)\n- [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)\n- [Commits](https://github.com/boto/boto3/compare/1.24.34...1.24.36)\n\n---\nupdated-dependencies:\n- dependency-name: boto3\n  dependency-type: direct:production\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump faker from 13.15.0 to 13.15.1 (#941)\n\nBumps [faker](https://github.com/joke2k/faker) from 13.15.0 to 13.15.1.\n- [Release notes](https://github.com/joke2k/faker/releases)\n- [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md)\n- [Commits](https://github.com/joke2k/faker/compare/v13.15.0...v13.15.1)\n\n---\nupdated-dependencies:\n- dependency-name: faker\n  dependency-type: direct:development\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump types-ujson from 5.2.0 to 5.4.0 (#947)\n\nBumps [types-ujson](https://github.com/python/typeshed) from 5.2.0 to 5.4.0.\n- [Release notes](https://github.com/python/typeshed/releases)\n- [Commits](https://github.com/python/typeshed/commits)\n\n---\nupdated-dependencies:\n- dependency-name: types-ujson\n  dependency-type: direct:development\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Add db vs saas to connection type api (#937)\n\n* add db vs saas to connection type api\n\n* adds changelog line\n\n* mypy\n\n* fix test\n\n* format\n\n* more tests\n\n* formatting\n\n* adds system type query param\n\n* format\n\n* adjust test saas search\n\n* correct query param\n\n* Use Nox as the build tool instead of Make (#919)\n\n* Create noxfile.py\n\n* update the dockerfile with more stages\n\n* add GitPython as a dev requirement (used by nox builds)\n\n* add the noxfiles, all docker builds work\n\n* fix the \"make compose-build\" command\n\n* remove the worker docker stage due to it being redundant with prod\n\n* create a unified compose file for integrations\n\n* update the password prompt to be accurate\n\n* copy/pasta run_infrastructure into the noxfiles dir so it can be called directly via python\n\n* add create_user and seed_test_data to the nox utils\n\n* docs commands work\n\n* get the generic dev command working\n\n* add db commands to utils\n\n* clean up mypy configuration in pyproject.toml and remove config from setup.cfg\n\n* simplify some of the CI targets and start updating the pytest targets\n\n* cleanup run_infrastructure\n\n* update ci_suite and other CI nox targets\n\n* add nox as a dev-requirement\n\n* get dev commands working, sans quickstart\n\n* tweak to the compose_down constant\n\n* get the new pylint target passing\n\n* remove analytics_id\n\n* updated the changelog\n\n* Apply suggestions from code review\n\nCo-authored-by: Paul Sanders \u003cpsanders1@gmail.com\u003e\n\n* fix the xenon command\n\n* add a few tweaks to the worker dev command\n\n* fix an import issue\n\n* update dockerignore\n\n* move mssql to the bottom of the datastore list\n\nCo-authored-by: Paul Sanders \u003cpsanders1@gmail.com\u003e\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* 863 - Retry a DSR (FE) (#938)\n\n* Update docs docker base image from slim-buster to slim-bullseye (#949)\n\n* Changed Debian base image\n\n* Update the python version to be the same as used in the fidesops app image\n\nCo-authored-by: Paul Sanders \u003cpsanders1@gmail.com\u003e\n\n* Updated changelog\n\nCo-authored-by: Dave Quinlan \u003cdave@ethyca.com\u003e\nCo-authored-by: Paul Sanders \u003cpsanders1@gmail.com\u003e\n\n* Experimenting with fixes for the failing MSSQL CI tests (#918)\n\n* Change docker password environment variable to MSSQL_SA_PASSWORD\n\n* Revert password environment variable name\n\n* Set MSSQL user to root in docker-compose\n\n* Revert setting root user in docker-compose\n\n* Change environment variable name from REQUIRE_MSSQL to SKIP_MSSQL_INSTALLATION\n\n* Add healthcheck to mssql compose file\n\n* Modify healthcheck command\n\n* Revert healthcheck\n\n* Try for more loging information\n\n* Another try for logging\n\n* Another try for logging\n\n* Try running only mssql tests to avoid timeout\n\n* Revert mssql only flag\n\n* Extend time out to try to get logs\n\n* Revert extra logging and extended timeout\n\n* Set network mode to host\n\n* Make mssql run on its own\n\n* Remove network from docker-compose and only run mssql in integration\n\n* Increase integration test logging\n\n* Revert mssql only\n\n* Use cache for docker\n\n* Fix workflow error\n\n* Fix workflow error\n\n* Fix target\n\n* Fix make traget\n\n* Fix make traget\n\n* Revert cache\n\n* Verify that mssql is running from pytest fixture\n\n* Add restart to mssql\n\n* Revert rester in docker-compose\n\n* Revert wait for mssql in pytest fixture\n\n* Wait for mssql to be ready before adding test data\n\n* Add count of retries\n\n* Update CHANGELOG\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Delete custom GitHub issue templates (#955)\n\nWe've defined generic issue templates for all Ethyca repos here: https://github.com/ethyca/.github/tree/main/.github/ISSUE_TEMPLATE\n\nRemoving the templates from this repo allows it to automatically pull in the organization templates for consistency.\n\n* Bump sqlalchemy-redshift from 0.8.8 to 0.8.10 (#940)\n\nBumps [sqlalchemy-redshift](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift) from 0.8.8 to 0.8.10.\n- [Release notes](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/releases)\n- [Changelog](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/blob/main/CHANGES.rst)\n- [Commits](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/compare/0.8.8...0.8.10)\n\n---\nupdated-dependencies:\n- dependency-name: sqlalchemy-redshift\n  dependency-type: direct:production\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Move tests into an \"ops\" subdir (#935)\n\n* Create __init__.py\n\n* move all of the test files down one dir into `ops`\n\n* update test paths\n\n* update paths where needed\n\n* update the changelog\n\n* Dispatch a repository event on new published releases (#945)\n\n* Add release dispatch event action\n\n* Update `CHANGELOG.md`\n\n* Reduce # of clients connected to the application db [#810] (#944)\n\n* Reduce number of open connections:\n\n- Limit task concurrency to two per worker.\n- Create one Engine per celery process which opens up a connection pool.  Create one Session per celery process and use that session across privacy requests.\n- Close the session after the privacy request has finished executing.  This just resets the session and returns connections back to the pool. It can be reused.\n- Remove unnecessary places where session is closed manually because the session is being used as a context manager and is already closed through that.\n- Pass the same Session that the privacy request is using through to TaskResources to be re-used to create ExecutionLogs instead of opening up a new Session.\n- Don't close the session when passing it into the Execution Log, wait until the entire privacy request is complete/exited.\n\n* Define \"self\" for run_privacy_task - it's the task itself.\n\nFor mypy's benefits, define that the session is a context manager.\n\n* Make a session non-optional for graph_task.run_access_request, graph_task.run_erasure, and for instantiating taskResources\n\n* Use missing db fixture.\n\n* Add missing db resource.\n\n* Update test to reflect new behavior that disabling a datasource while a request is in progress can cause related collections to be skipped once the current session is expired and the connection config has the most recent state.\n\nBecause the same Session that is being used to run the PrivacyRequest is now being used for ExecutionLogs, the process of saving an ExecutionLog runs a session.commit() which expires the Session and causes the ConnectionConfig to have the most recent state the next time it is accessed.\n\n* Update CHANGELOG.\n\n* enable worker by default in our dockerfile (#958)\n\n* add extra steps to make clean (#767)\n\n* Push `dev` image on pushes to `main` (#956)\n\n* Update publish_to_dockerhub.yml\n\n* add a dev step and use nox\n\n* update the changelog\n\n* Move Client Code into an `ops` subdir (#964)\n\n* Move Client Code into an `ops` subdir\n\n* move all of the files\n\n* update the dockerfile\n\n* update package.json\n\n* update codepaths for workflow tests\n\n* Update the changelog and docs references\n\n* Update .github/dependabot.yaml\n\n* Bump gitpython from 3.1 to 3.1.27 (#971)\n\nBumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.1 to 3.1.27.\n- [Release notes](https://github.com/gitpython-developers/GitPython/releases)\n- [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES)\n- [Commits](https://github.com/gitpython-developers/GitPython/compare/3.1.0...3.1.27)\n\n---\nupdated-dependencies:\n- dependency-name: gitpython\n  dependency-type: direct:development\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Fix the `nox -s dev` command not spinning up the webserver (#959)\n\n* Update dev_nox.py\n\n* spin up the app before running a container shell\n\n* [#927, #929, #930] sendgrid, adobe, outreach connector docs (#951)\n\n* Bump sqlalchemy-utils from 0.37.8 to 0.38.3 (#968)\n\nBumps [sqlalchemy-utils](https://github.com/kvesteri/sqlalchemy-utils) from 0.37.8 to 0.38.3.\n- [Release notes](https://github.com/kvesteri/sqlalchemy-utils/releases)\n- [Changelog](https://github.com/kvesteri/sqlalchemy-utils/blob/master/CHANGES.rst)\n- [Commits](https://github.com/kvesteri/sqlalchemy-utils/compare/0.37.8...0.38.3)\n\n---\nupdated-dependencies:\n- dependency-name: sqlalchemy-utils\n  dependency-type: direct:production\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump pyodbc from 4.0.32 to 4.0.34 (#980)\n\nBumps [pyodbc](https://github.com/mkleehammer/pyodbc) from 4.0.32 to 4.0.34.\n- [Release notes](https://github.com/mkleehammer/pyodbc/releases)\n- [Commits](https://github.com/mkleehammer/pyodbc/compare/4.0.32...4.0.34)\n\n---\nupdated-dependencies:\n- dependency-name: pyodbc\n  dependency-type: direct:production\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump dask from 2022.7.0 to 2022.7.1 (#967)\n\nBumps [dask](https://github.com/dask/dask) from 2022.7.0 to 2022.7.1.\n- [Release notes](https://github.com/dask/dask/releases)\n- [Changelog](https://github.com/dask/dask/blob/main/docs/release-procedure.md)\n- [Commits](https://github.com/dask/dask/compare/2022.7.0...2022.7.1)\n\n---\nupdated-dependencies:\n- dependency-name: dask\n  dependency-type: direct:production\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump alembic from 1.8.0 to 1.8.1 (#989)\n\nBumps [alembic](https://github.com/sqlalchemy/alembic) from 1.8.0 to 1.8.1.\n- [Release notes](https://github.com/sqlalchemy/alembic/releases)\n- [Changelog](https://github.com/sqlalchemy/alembic/blob/main/CHANGES)\n- [Commits](https://github.com/sqlalchemy/alembic/commits)\n\n---\nupdated-dependencies:\n- dependency-name: alembic\n  dependency-type: direct:production\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* fix erroneous values in the Outreach config (#988)\n\n* Add documentation for new nox commands (#981)\n\n* Add documentation for new nox commands\n\n* changelog\n\n* missed make to nox edits\n\n* review edits\n\n* update `nox -s dev` to not open a shell, add it as a posarg option\n\n* Apply suggestions from code review\n\n* remove the analytics id\n\n* update typo\n\n* remove extra numbering\n\nCo-authored-by: Thomas \u003cthomas.lapiana+github@pm.me\u003e\n\n* Bump types-toml from 0.10.7 to 0.10.8 (#998)\n\nBumps [types-toml](https://github.com/python/typeshed) from 0.10.7 to 0.10.8.\n- [Release notes](https://github.com/python/typeshed/releases)\n- [Commits](https://github.com/python/typeshed/commits)\n\n---\nupdated-dependencies:\n- dependency-name: types-toml\n  dependency-type: direct:development\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump fideslog from 1.2.1 to 1.2.2 (#996)\n\nBumps [fideslog](https://github.com/ethyca/fideslog) from 1.2.1 to 1.2.2.\n- [Release notes](https://github.com/ethyca/fideslog/releases)\n- [Commits](https://github.com/ethyca/fideslog/compare/v1.2.1...v1.2.2)\n\n---\nupdated-dependencies:\n- dependency-name: fideslog\n  dependency-type: direct:production\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Added Choose your connection feature (#987)\n\n* Added Choose your connection feature\n\n760-Add a Connection - Select a connector to configure (front end)\n866-Add a Connection - Front End layout structure\n\n* Updated CHANGELOG.md file\n\n* Fix lint issues\n\n* Fix build issue\n\n* Hide overflow\n\nCo-authored-by: Andrew Jackson \u003candrew.c.j1995@gmail.com\u003e\n\n* saas request overrides (#986)\n\n* initial cut of saas request overrides. include mailchimp as an example and test case. minor refactor of some of the saas request execution to enable smoother override\n\n* fix rebase issue by moving saas override tests into ops subdir\n\n* import path updates to resolve conflicts caused by rebase\n\n* add session parameter into graph task calls to fix saas override integration tests caused by rebase\n\n* update changelog\n\n* tweaks to saas connector overrides and associated tests\n\n* expose override factory register as module variable for clenaer decorator calls\n\nCo-authored-by: Adam Sachs \u003cadam@Adams-MacBook-Pro.local\u003e\n\n* Update OAuth strategy to be able to perform local testing (#962)\n\n* Update boto3 requirement from ~=1.24.36 to ~=1.24.42 (#1001)\n\nUpdates the requirements on [boto3](https://github.com/boto/boto3) to permit the latest version.\n- [Release notes](https://github.com/boto/boto3/releases)\n- [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)\n- [Commits](https://github.com/boto/boto3/compare/1.24.36...1.24.42)\n\n---\nupdated-dependencies:\n- dependency-name: boto3\n  dependency-type: direct:production\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump fastapi[all] from 0.78.0 to 0.79.0 (#1002)\n\nBumps [fastapi[all]](https://github.com/tiangolo/fastapi) from 0.78.0 to 0.79.0.\n- [Release notes](https://github.com/tiangolo/fastapi/releases)\n- [Commits](https://github.com/tiangolo/fastapi/compare/0.78.0...0.79.0)\n\n---\nupdated-dependencies:\n- dependency-name: fastapi[all]\n  dependency-type: direct:production\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* update config reference to use lowercase values (#952)\n\n* [#909] update config reference to use lowercase values\n\n* changelog\n\n* missed uppercase variables\n\n* update usages of False to false in connector docs\n\n* Added Auth0 Connector (#991)\n\n* add pagination back to connection types endpoints (#1019)\n\n* add pagination back to connection types endpoints\n\n* check for pagination in system_type search\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Subject Request Events and Logs Section (#1018)\n\n* Fix small issue with eslint config\n\n* Get initial drawer working\n\n* Add jest config to eslintignore\n\n* Fix small css issue\n\n* Refactor components and get initial functionality\n\n* Fix small logic error\n\n* Fix couple of bugs and format code\n\n* Conditinally display error tag\n\n* Fix issues with merge\n\n* Format and lint\n\n* Sort imports\n\n* Update to new solution\n\n* Format code\n\n* Update changelog\n\nCo-authored-by: Sean Preston \u003csean@ethyca.com\u003e\n\n* First draft of OAuth documentation (#963)\n\n* Send Errored Requests / Reprocessed Requests Info to FidesLog [#754] (#993)\n\n* Add a method to format a representation of the graph for caching in Redis and a separate method to build a summary of the differences in the graph when a privacy request is rerun to fideslog.\n\n- Adds FieldAddress.from_string method.\n\n* Add methods to cache a representation of the access graph when it is built and a separate method to retrieve it from the cache.\n\n- Also add a method to build a \"rerun_access_graph\" AnalyticsEvent for fideslog where applicable.\n\n* When running the access portion of the privacy request, log stats about a rerun and then cache the current access graph.\n\n- Give a different prefix when we're caching the access graph to not get mixed up with access request results.\n\n* Show skipped new edges that are directly upstream of completed nodes instead. These edges are intentionally dropped from the new graph on rerun, so want to surface this count.\n\n* Log if a privacy request fails during the \"erasure\" step of privacy request execution.  Even though the access step is not rerun here, compare the previously cached access graph with the access graph that would have been run to determine what data has changed.\n\n* Add missing session variables.\n\n* Send an event to Fideslog when privacy request execution fails.\n\n* Add missed session - bad merge.\n\n* Update changelog.\n\n* Update docstrings.\n\n* Remove copy/paste comment.\n\n* Respond to CR comments.\n\n* Currently AnalyticsEvent.local_host cannot be None.\n\n* Update the compose file and workflows to expect an already-built image (#966)\n\n* Update the compose file and workflows to be in line with fidesctl\n\n* update the compose file to look for a specific image\n\n* rename files and update the safe PR checks\n\n* add check_migrations to the ci checks and nox\n\n* fix the failing PR checks\n\n* fix CI failures\n\n* update the unsafe checks workflow\n\n* update run_infrastructure to use the compose service name\n\n* remove the makefile and the old run_infra script\n\n* bump pylint version, pin isort, fix issues\n\n* update pytest setup path\n\n* update the changelog\n\n* make OPS_TEST_DIR a constant\n\n* fix nox missing vars\n\n* specify that the nox imports are relative imports\n\n* remove relative import paths\n\n* run isort\n\nCo-authored-by: Sean Preston \u003csean@ethyca.com\u003e\n\n* Create `AuditLog` on privacy request approval (#1038)\n\n* Create approval audit logs\n\n* Add tests\n\n* disable a pylint lint\n\n* Update changelog\n\n* Update privacy_request fixture\n\n* Updating Salesforce to use OAuth2 authentication code flow (#1039)\n\n* Removing saas_config.toml (#1043)\n\n* Bump types-pyyaml from 6.0.9 to 6.0.11 (#1047)\n\nBumps [types-pyyaml](https://github.com/python/typeshed) from 6.0.9 to 6.0.11.\n- [Release notes](https://github.com/python/typeshed/releases)\n- [Commits](https://github.com/python/typeshed/commits)\n\n---\nupdated-dependencies:\n- dependency-name: types-pyyaml\n  dependency-type: direct:development\n  update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump docker/build-push-action from 2 to 3 (#1044)\n\nBumps [docker/build-push-action](https://github.com/docker/build-push-action) from 2 to 3.\n- [Release notes](https://github.com/docker/build-push-action/releases)\n- [Commits](https://github.com/docker/build-push-action/compare/v2...v3)\n\n---\nupdated-dependencies:\n- dependency-name: docker/build-push-action\n  dependency-type: direct:production\n  update-type: version-update:semver-major\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Bump dask from 2022.7.1 to 2022.8.0 (#1046)\n\nBumps [dask](https://github.com/dask/dask) from 2022.7.1 to 2022.8.0.\n- [Release notes](https://github.com/dask/dask/releases)\n- [Changelog](https://github.com/dask/dask/blob/main/docs/release-procedure.md)\n- [Commits](https://github.com/dask/dask/compare/2022.7.1...2022.8.0)\n\n---\nupdated-dependencies:\n- dependency-name: dask\n  dependency-type: direct:production\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* Update boto3 requirement from ~=1.24.42 to ~=1.24.46 (#1045)\n\nUpdates the requirements on [boto3](https://github.com/boto/boto3) to permit the latest version.\n- [Release notes](https://github.com/boto/boto3/releases)\n- [C…\n\n* Get _all_ automated checks passing (#1122)\n\n* Update README.md\n\n* rename references to ops config\n\n* more updates, FE is broken\n\n* more code fixes, still some missing configs to fix\n\n* fix isort and pylint\n\n* get the webserver spinning up again (sans UI)\n\n* remove legacy pr workflows\n\n* fix the UI build\n\n* clean up pr workflows\n\n* fix isort\n\n* fix ctl tests\n\n* remove erroneous test file\n\n* fix some ops test issues\n\n* fix more ops test issues\n\n* add new ops migrations\n\n* fix some merge conflicts\n\n* Fix merge conflict in tsconfig\n\n* Fix merge conflicts in datasets-classify.cy.ts\n\n* Configure Cypress to be able to log in\n\n* Standardizie on one NavBar\n\n* Standardize on using the Layout component\n\n* Remove unneeded files\n\n* Update cypress tests\n\n* Log in before cypress tests\n\n* Run prettier\n\n* Use Layout for individual subject requests\n\n* more test fixes\n\n* clean up more merge conflicts\n\n* add notification settings\n\n* update the view config CLI command to accept subsections\n\n* fix the celery tests\n\n* reactivate the registry load for saas connectors\n\n* reactivate the webserver scheduler\n\n* remove devcontainer setup\n\nCo-authored-by: Allison King \u003callisonjuliaking@gmail.com\u003e\n\n* update quickstart URLs\n\n* fix the quickstart\n\n* fix some static checks\n\n* Fix postgres example data\n\n* update the postgres example sql\n\n* start the UI up in the background\n\n* fix docs merge conflicts\n\n* remove the analytics_id\n\n* Make nox test name sessions more logical (#1135)\n\n* Make nox test name sessions more logical\n\n* consolidate the nox test sessions\n\n* Fix privacy center conflicts (#1133)\n\n* Resolve conflicts and upgrade next\n\n* Resolve merge conflicts\n\n* Fix missing required prop\n\n* Update postman collection /policy --\u003e /dsr/policy\n\n* Make local docker UI better (#1140)\n\n* Instead of rebuilding node_modules, just make docker-compose not overwrite them during volume mounting\n\n* Increase  count for starting up fides, likely needed for M1 macs\n\n* Disable various email sends by default in unit testing with autouse fixtures. (#1153)\n\n* add vault environment variables to docker-compose container (#1151)\n\nCo-authored-by: Adam Sachs \u003cadam@Adams-MacBook-Pro.local\u003e\n\n* FIdesConfig instead of the EmailConfigRequest resource is being passed into create_or_update_email_config. (#1146)\n\n* 1149 - Update all fidesops/fidesctl logos to use the default Fides logo (#1158)\n\n* 1149 - Update all fidesops/fidesctl logos to use the default Fides logo\n\nUpdated logo.svg file to the new Fides logo\n\n* Updated CHANGELOG.md file\n\n* ui/package: restore copy-export command (#1166)\n\n* Make `optional-requirements.txt` a source of version truth (#1171)\n\n* Correctly `isort` `versioneer` in files\n\n* Make `optional-requirements.txt` a source of truth\n\n* Update `CHANGELOG.md`\n\n* Include `optional_requirements.txt` in the package\n\n* Specify an encoding when opening requirements files\n\nQuiets the pylint warning\n\n* Remove superfluous variables\n\n* Match dependency names more strictly, add docstring\n\n* [Unified Fides] Address Remaining Failing Fidesops-Related Test Failures (#1178)\n\n* Delete fidesctl user routes in favor of fidesops user routes.\n\nThey are practically copies of each other except fidesops user logout route has improved logic such as allowing the user to logout with a malformed or expired token.  The fidesctl route was taking priority.\n\n* Use new CONFIG variable in test_saas_queryconfig\n\n* Adjust fidesops health endpoint test.  We're using the fidesctl health endpoint now and they surface a version number while fidesops did not.\n\n* Add missing CONFIG variable in existing execution tests.\n\nFix incorrect timescale test name.\n\n* Add missing timescale secrets to integration_test_config.toml.\n\n* Only show parameters inside queries if \"dev mode\" is True, not if \"test mode\" is True. (#1162)\n\n* 1177 - ESLint: Parsing error: Cannot read file tsconfig.json file (#1181)\n\n* 1177 - ESLint: Parsing error: Cannot read file tsconfig.json file\n\n- Resolved parsing error: Cannot read file '/users/ccalhoun/documents/github/fides/tsconfig.json'.eslint\n- Updated NPM caniuse-lite version dependency\n- Sorted package.json file\n\n* Updated CHANGELOG.md file\n\n* 1191 duplicate key in fides admin UI package.json file (#1192)\n\n* 1191-Duplicate key in fides admin ui package.json file\n\n* Updated CHANGELOG.md file\n\n* [Unified Fides] Reduce Idle Health Check Connections (#1182)\n\n* Stop creating new engines as part of the process of running health checks which cause us to have too many idle connections opened against the application database.\nUse the same engine that is being shared across the \"ops\" API endpoints.\n\n- Remove unused get_db_for_health_check\n\n* Remove the ctl get_db used in the endpoints in favor of the ops get_db.\n\n* Mock get_db_health needs two parameters.\n\n* Re-add SaaS template update script to startup hook (#1213)\n\n* Re-add SaaS template update script to startup hook\n\nAlso update dependency getter that returns the shared db session directly\nto be importable from outside classes.\n\n* Use a context manager for db session\n\nCo-authored-by: Adam Sachs \u003cadam@Adams-MacBook-Pro.local\u003e\n\n* Square access and erasure (#1193)\n\n* Fidesops merge to unified fides 2 (#1214)\n\n* Bump pre-commit from 2.9.3 to 2.20.0 (#853) (#1373)\n\nBumps [pre-commit](https://github.com/pre-commit/pre-commit) from 2.9.3 to 2.20.0.\n- [Release notes](https://github.com/pre-commit/pre-commit/releases)\n- [Changelog](https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md)\n- [Commits](https://github.com/pre-commit/pre-commit/compare/v2.9.3...v2.20.0)\n\n---\nupdated-dependencies:\n- dependency-name: pre-commit\n  dependency-type: direct:development\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\n\n* 1016 frontend ability for users to manually enter pii to an in progress subject request (#1377)\n\n*  On successful connector creation the empty yaml text input displays a validation error\n\n* Updated caniuse-lite NPM dependency\n\n* Added min/max character limitation validation for Manual Webhook DSR Customization\n\n* Prevented Chakra Divider component opacity from being overridden at runtime. User could not see visible horizontal line on UI before.\n\n* Updated the Manual Processing detail form to not be submitted until the form is dirty.\n\n* Removed form dirty check on ManualProcessingDetail component\n\n* Update URL when user is navigating to DSR Customization screen when creating a Manual Webhook\n\n* Enable retries on saas connectors for failures at the http request level (#1376)\n\n* Add decorator to send method that retries throttles\n\n* Fix black/mypy\n\n* Fix pylint\n\n* Add tests for authenticated client\n\n* Small fixes and typos\n\n* Update CHANGELOG.md\n\n* Small changes. Update retry logic to not retry general exceptions\n\nCo-authored-by: Eduardo Armendariz \u003ceduardo@ethyca.com\u003e\n\n* Add Consent Request API (#1387)\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Add new template for consent requets emails (#1405)\n\n* Add new template for consent requets emails\n\n* Remove analytics id\n\n* Update subject\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Add authenticated route to get consent preferences (#1402)\n\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\n\n* Backend: Updating or Deleting Access Webhooks [#1388][#1389] (#1394)\n\n* If a manual webhook is deleted or disabled, check if there are any remaining active manual webhooks configured. If not, queue any Privacy Requests stuck in \"requires_input\" for processing.\n\n* In the \"view_uploaded_manual_webhook_data\", load cached webhook data for a privacy request in strict mode. If it fails (no data saved, extra field saved, field missing), return checked=True, so the user knows they need to reupload data for this webhook before it can be submitted.\n\nReturn the data in non-strict mode, so we just show the overlap between the data saved and the fields defined.\n\n* Update changelog.\n\n* Move queue_requires_input_requests to the connection_endpoints where this is the only module it's being called - both where you update and delete a connection config.\n\n* Clarify docstring.\n\n* Braze Connector: Access Endpoints (#1248)\n\n* [#1393] Update Fidesops config with sane defaults where necessary (#1395)\n\n* add sane defaults\n\n* make subsections of config with complete defaults optional\n\n* lowercase database.enabled, set defaults for optional configs\n\n* update return type\n\n* updates changelog\n\n* make PORT an env var\n\n* cast env var to int\n\n* remove unnecessary unpinned dependency\n\n* bump fideslib version\n\n* bump fideslib to 3.1.4\n\n* add defaults for the non optional config subclasses\n\n* set empty dict to default for config subclasses that require some fields\n\n* use .get() in assemble URL for correct error message, correct comment\n\n* update jwt_key type annotation\n\n* Explain Privacy Request Execution [#1397] (#1396)\n\nCo-authored-by: Cole \u003ccole@ethyca.com\u003e\n\n* Update docker command on privacy center step 4 (#1410)\n\nremoving the typo ` . at the end of the command so that the command works.\n\n* 1319 consent UI api integration (#1407)\n\n* Add consent UI back in\n\n* Finish initial integration with consent api\n\n* WIP consent page\n\n* Get initial consent updating working\n\n* Improve button look and feel\n\n* Add untracked files\n\n* Format VerificationForm.tsx\n\n* Remove comments\n\n* Rename Privacy modal variables\n\n* Rename variable\n\n* 1401 admin UI persist redux store to localstorage (#1409)\n\n* 1401 - Admin UI: Persist Redux store to localStorage\n\n* Resolved React memory leak when user attempts to logout via the Subject Requests landing page\n\n* Resolved UI unit test failure\n\n* Rollback previous change\n\n* Removed blacklist property from Redux store config\n\n* Refactored due to recommended code review feedback\n\n* update footer links (#1406)\n\n* update footer links\n\n* changelog\n\n* Update CHANGELOG.md\n\nCo-authored-by: Paul Sanders \u003cpaul@ethyca.com\u003e\n\nCo-authored-by: Sean Preston \u003csean@ethyca.com\u003e\nCo-authored-by: Paul Sanders \u003cpaul@ethyca.com\u003e\n\n* firebase auth integration (#1399)\n\n* Initial implementation of firebase auth connector\n\nIntroduces firebase_admin sdk as a fidesops dependency\nLeverages request overrides to define custom functionality\n\n* Update changelog\n\n* Minimize unused client config in firebase auth config\n\n* Add user.photo_url as a supported Firebase Auth field\n\nImprove readability in access implementation\n\n* Fix field name to be more consistent with python SDK properties\n\n* Add clarifying comment to unused function docstring\n\n* Add photo_url field to firebase auth dataset\n\n* Properly test provider data in firebase auth integration.\n\nAlso include test coverage for delete function, even thought it's\nnot being invoked by default saas config.\n\n* Only add fields to result if popluated\n\n* Fix empty attribute logic within provider data block\n\n* Ignore pylint error for too many variables\n\n* fix up provider data check\n\nCo-authored-by: Adam Sachs \u003cadam@Adams-MacBook-Pro.local\u003e\n\n* Add Braze connector registry entry (#1418)\n\n* Braze SVG added\n\n* Fixing change log message\n\nCo-authored-by: Adrian Galvan \u003cadrian@ethyca.com\u003e\nCo-authored-by: Adam Sachs \u003cadam@Adams-MacBook-Pro.local\u003e\n\n* Removing PII fields from conversations collection (#1386)\n\n* Fidesops v1.8.1 Release Checklist (#1420)\n\n* updates changelog\n\n* add UI testing steps to release checklist\n\n* DX Improvements from `v1.8.1` release (#1421)\n\n* stop swallowing integrity errors on dataset update\n\n* dont attach logging middleware if analytics opt_out is true\n\n* handle diff correctly\n\n* disable consider-using-f-string as we need this for logging statements to work effectively in fideslog\n\n* Updated CHANGELOG.md file from prior merge\n\n* get the server up and running\n\n* fix test collection\n\n* fix isort and black\n\n* update dataset annotations\n\n* Merge latest fidesops UI into unified ui\n\n* Fixing post UI merge issues\n\n* Fix formatting and import issues\n\n* get all unit tests passing\n\n* Fix cypress failure 🤞\n\n* Remove auth.slice.test.ts\n\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\nCo-authored-by: Eduardo Armendariz \u003ceduardo.armendariz13@gmail.com\u003e\nCo-authored-by: Eduardo Armendariz \u003ceduardo@ethyca.com\u003e\nCo-authored-by: Paul Sanders \u003cpaul@ethyca.com\u003e\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\nCo-authored-by: Dawn Pattison \u003cpattisdr@users.noreply.github.com\u003e\nCo-authored-by: Noonari \u003csadaqatullah.noonari@gmail.com\u003e\nCo-authored-by: Sean Preston \u003csean@ethyca.com\u003e\nCo-authored-by: Cole \u003ccole@ethyca.com\u003e\nCo-authored-by: shawnplusplus \u003c46225246+shawnplusplus@users.noreply.github.com\u003e\nCo-authored-by: Andrew Jackson \u003candrew.c.j1995@gmail.com\u003e\nCo-authored-by: Cole Isaac \u003c82131455+conceptualshark@users.noreply.github.com\u003e\nCo-authored-by: Adam Sachs \u003cadam@ethyca.com\u003e\nCo-authored-by: Adam Sachs \u003cadam@Adams-MacBook-Pro.local\u003e\nCo-authored-by: Adrian Galvan \u003cadrian@ethyca.com\u003e\nCo-authored-by: Thomas \u003cthomas.lapiana+github@pm.me\u003e\n\n* Remove SaaS type enum (#1197)\n\n* Remove SaaS type enum and dynamically reference registered types\n\nUpdate tests to dynamically compare results rather than looking for static values.\n\n* Remove references to specific custom SaaSType\n\nSince we no longer have an enum for SaaSType, we don't need to have a specific\n'custom' type - instead, users can simply create their own type dynamically.\nThere is also now no more invalid 'type' value for saas configs, so\nwe remove the test for that functionality.\n\n* Remove unused import\n\n* update changelog\n\nCo-authored-by: Adam Sachs \u003cadam@Adams-MacBook-Pro.local\u003e\n\n* Create rate limiter that interacts with Redis (#1189)\n\n* Rate limiter implementation copied from fidesops repo\n\n* Change enum values to not be plural\n\n* Add test using rate limiter from multiple threads\n\n* Update function names and test parameters\n\n* Update error messages to include better details\n\n* Update changelog\n\n* Add an additional comment to test\n\n* fix typo in test name\n\n* Fix typo in test comment\n\n* Fix one more typo\n\n* Run black one more time\n\nCo-authored-by: Eduardo Armendariz \u003ceduardo@ethyca.com\u003e\n\n* update to the type index\n\n* format and lint UI\n\n* remove duplicates from the types index\n\n* fix MenuButton typing issue\n\n* fix references to clients/ctl\n\n* readd the prod-export command\n\n* remove mentions of unified-fides branch in workflows\n\n* merge alembic heads\n\n* fix typo\n\n* fix static checks\n\n* fix dataset parsing error\n\n* fix fides annotation coverage\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\nCo-authored-by: Dawn Pattison \u003cpattisdr@users.noreply.github.com\u003e\nCo-authored-by: Cole Isaac \u003c82131455+conceptualshark@users.noreply.github.com\u003e\nCo-authored-by: Adrian Galvan \u003cadriang430@gmail.com\u003e\nCo-authored-by: Dave Quinlan \u003c83430497+daveqnet@users.noreply.github.com\u003e\nCo-authored-by: Paul Sanders \u003cpsanders1@gmail.com\u003e\nCo-authored-by: Paul Sanders \u003cpau@ethyca.com\u003e\nCo-authored-by: Sean Preston \u003csean@ethyca.com\u003e\nCo-authored-by: Catherine Smith \u003ceastandwestwind@gmail.com\u003e\nCo-authored-by: chriscalhoun1974 \u003c68459950+chriscalhoun1974@users.noreply.github.com\u003e\nCo-authored-by: Dave Quinlan \u003cdave@ethyca.com\u003e\nCo-authored-by: Neville Samuell \u003cneville@ethyca.com\u003e\nCo-authored-by: Phil Salant \u003cPSalant726@users.noreply.github.com\u003e\nCo-authored-by: Andrew Jackson \u003candrew.c.j1995@gmail.com\u003e\nCo-authored-by: Adam Sachs \u003cadam@ethyca.com\u003e\nCo-authored-by: Adam Sachs \u003cadam@Adams-MacBook-Pro.local\u003e\nCo-authored-by: HamzaWaseemOnBench \u003c104363089+HamzaWaseemOnBench@users.noreply.github.com\u003e\nCo-authored-by: Adrian Galvan \u003cadrian@ethyca.com\u003e\nCo-authored-by: Noonari \u003csadaqatullah.noonari@gmail.com\u003e\nCo-authored-by: Robert Keyser \u003c39230492+RobertKeyser@users.noreply.github.com\u003e\nCo-authored-by: Eduardo Armendariz \u003ceduardo.armendariz13@gmail.com\u003e\nCo-authored-by: Kelsey Thomas \u003c101993653+Kelsey-Ethyca@users.noreply.github.com\u003e\nCo-authored-by: Cole Garbo \u003ccolegarbo@Ethycas-MacBook-Pro.local\u003e\nCo-authored-by: Cole \u003ccole@ethyca.com\u003e\nCo-authored-by: Paul Sanders \u003cpaul@ethyca.com\u003e\nCo-authored-by: Hamza W \u003chamza@Hamzas-MacBook-Pro.local\u003e\nCo-authored-by: Cillian \u003c1268052+cilliankieran@users.noreply.github.com\u003e\nCo-authored-by: Christopher Calhoun \u003cchris@ethyca.com\u003e\nCo-authored-by: Allison King \u003callisonjuliaking@gmail.com\u003e\nCo-authored-by: Adam Sachs \u003cadam@Adams-MBP.attlocal.net\u003e\nCo-authored-by: HamzaWaseemOnBench \u003chwaseem@onbench.com\u003e\nCo-authored-by: Steve Murphy \u003csteven.d.murphy@gmail.com\u003e\nCo-authored-by: Sebastian Sangervasi \u003c2236777+ssangervasi@users.noreply.github.com\u003e\nCo-authored-by: Zeeshan-Ethyca \u003c109582532+Zeeshan-Ethyca@users.noreply.github.com\u003e\nCo-authored-by: Eduardo Armendariz \u003ceduardo@ethyca.com\u003e\nCo-authored-by: shawnplusplus \u003c46225246+shawnplusplus@users.noreply.github.com\u003e",
"Tags": [],
"RuleID": "generic-api-key",
"Fingerprint": "5a485387d8af247ec6479e4115088cbbb8394d77:.fides/fides.toml:generic-api-key:33"
}

detect-secrets

{
    "category": "UNVERIFIED",
    "filename": ".fides/fides.toml",
    "lines": {
        "37": "oauth_root_client_secret = \"fidesadminsecret\""
    },
    "secrets": "fidesadminsecret",
    "types": [
        "Secret Keyword"
    ]
},
{
    "category": "UNVERIFIED",
    "filename": ".fides/fides.toml",
    "lines": {
        "38": "drp_jwt_secret = \"secret\""
    },
    "secrets": "secret",
    "types": [
        "Secret Keyword"
    ]
},
{
    "category": "UNVERIFIED",
    "filename": ".fides/fides.toml",
    "lines": {
        "40": "root_password = \"Testpassword1!\""
    },
    "secrets": "Testpassword1!",
    "types": [
        "Secret Keyword"
    ]
}

daveqnet commented 1 year ago

This morning: testing with real, recently revoked secrets.

An AWS access key
A Mailchimp API key
A Mailgun API key
A Stripe API key

First up, detect-secrets. It catches the AWS access key 🥳 but fails to catch the API keys 😢.

~/projects/github/ethyca/fides main ❯ git log -1 | head -n 3
commit af4ba105f421c8a020a231ecf3e1b64f6a8d43ea
Author: Paul Sanders <paul@ethyca.com>
Date:   Sun Oct 23 20:04:24 2022 -0700
~/projects/github/ethyca/fides main ❯ detect-secrets scan > .secrets.baseline
~/projects/github/ethyca/fides main ?1 ❯ code .pre-commit-config.yaml                                                                                                                                                                                                                  5s
~/projects/github/ethyca/fides main ?1 ❯ cat .pre-commit-config.yaml
minimum_pre_commit_version: "2"

repos:
-   repo: https://github.com/Yelp/detect-secrets
    rev: v1.4.0
    hooks:
    -   id: detect-secrets
        args: ['--baseline', '.secrets.baseline']
        exclude: package.lock.json
~/projects/github/ethyca/fides main !1 ?1 ❯ pre-commit install
pre-commit installed at .git/hooks/pre-commit
~/projects/github/ethyca/fides main !1 ?1 ❯ touch TESTSECRETS.md
~/projects/github/ethyca/fides main !1 ?2 ❯ code TESTSECRETS.md 
~/projects/github/ethyca/fides main !1 ?2 ❯ head -n 2 TESTSECRETS.md
### Test Secrets for Secrets Detection Pre-Commit Hooks
All recently revoked: an AWS key, a Mailchip API key, a Mailgun API key, and a Stripe API key.
~/projects/github/ethyca/fides main !1 ?2 ❯ git add . 
~/projects/github/ethyca/fides main +3 ❯ git commit -m "Will detect-secrets catch these secrets pre-commit?"
Detect secrets...........................................................Failed
- hook id: detect-secrets
- exit code: 1

ERROR: Potential secrets about to be committed to git repo!

Secret Type: AWS Access Key
Location:    TESTSECRETS.md:4

Possible mitigations:
  - For information about putting your secrets in a safer place, please ask in
    #security
  - Mark false positives with an inline `pragma: allowlist secret`
    comment

If a secret has already been committed, visit
https://help.github.com/articles/removing-sensitive-data-from-a-repository

~/projects/github/ethyca/fides main +3 ❯ echo "Okay, it caught the AWS key. I will remove it and check the others."
Okay, it caught the AWS key. I will remove it and check the others.
~/projects/github/ethyca/fides main +3 ❯ code TESTSECRETS.md 
~/projects/github/ethyca/fides main +3 ❯ git add .
~/projects/github/ethyca/fides main +3 ❯ git commit -m "Will detect-secrets catch the non-AWS secrets pre-commit?"
Detect secrets...........................................................Passed
[main fe12acea] Will detect-secrets catch the non-AWS secrets pre-commit?
 3 files changed, 2485 insertions(+), 35 deletions(-)
 create mode 100644 .secrets.baseline
 create mode 100644 TESTSECRETS.md
~/projects/github/ethyca/fides main ⇡1 ❯ echo "No, it did not. :("
No, it did not. :(

daveqnet commented 1 year ago

The exact same test using gitleaks. It performs better than detect-secrets, but still misses two API keys.

~/projects/github/ethyca/fides main ❯ git log -1 | head -n 3 
commit af4ba105f421c8a020a231ecf3e1b64f6a8d43ea
Author: Paul Sanders <paul@ethyca.com>
Date:   Sun Oct 23 20:04:24 2022 -0700
~/projects/github/ethyca/fides main ❯ code .pre-commit-config.yaml 
~/projects/github/ethyca/fides main ❯ cat .pre-commit-config.yaml 
minimum_pre_commit_version: "2"

repos:
  - repo: https://github.com/zricethezav/gitleaks
    rev: v8.15.0
    hooks:
      - id: gitleaks
~/projects/github/ethyca/fides main !1 ❯ pre-commit install
pre-commit installed at .git/hooks/pre-commit
~/projects/github/ethyca/fides main !1 ❯ touch TESTSECRETS.md
~/projects/github/ethyca/fides main !1 ?1 ❯ code TESTSECRETS.md 
~/projects/github/ethyca/fides main !1 ?1 ❯ git add .
~/projects/github/ethyca/fides main +2 ❯ git commit -m "Will gitleaks detect these secrets pre-commit?"
[INFO] Initializing environment for https://github.com/zricethezav/gitleaks.
[INFO] Installing environment for https://github.com/zricethezav/gitleaks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Detect hardcoded secrets.................................................Failed
- hook id: gitleaks
- exit code: 1

○
    │╲
    │ ○
    ○ ░
    ░    gitleaks

Finding:     ...RIPE_API_KEY”, “REDACTED”
Secret:      REDACTED
RuleID:      stripe-access-token
Entropy:     4.443569
File:        TESTSECRETS.md
Line:        10
Fingerprint: TESTSECRETS.md:stripe-access-token:10

Finding:     ...CCESS_KEY_ID”, “REDACTED”
Secret:      REDACTED
RuleID:      aws-access-token
Entropy:     3.621928
File:        TESTSECRETS.md
Line:        4
Fingerprint: TESTSECRETS.md:aws-access-token:4

10:36AM INF 1 commits scanned.
10:36AM INF scan completed in 66.1ms
10:36AM WRN leaks found: 2

~/projects/github/ethyca/fides main +2 ❯ echo "Good, it caught both the AWS and the Stripe keys. I will remove both and re-test."                                                                                                                                                      7s
Good, it caught both the AWS and the Stripe keys. I will remove both and re-test.
~/projects/github/ethyca/fides main +2 ❯ code TESTSECRETS.md 
~/projects/github/ethyca/fides main +2 ❯ git add .
~/projects/github/ethyca/fides main +2 ❯ git commit -m "Testing again, will the Mailgun and Mailchimp keys be caught?"
Detect hardcoded secrets.................................................Passed
[main f318c6f4] Testing again, will the Mailgun and Mailchimp keys be caught?
 2 files changed, 12 insertions(+), 35 deletions(-)
 create mode 100644 TESTSECRETS.md
~/projects/github/ethyca/fides main ⇡1 ❯ echo "No. :("
No. :(

daveqnet commented 1 year ago

Out of morbid curiosity I checked ggshield to see if a commercial tool performs better. The answer: no. It definitely looks a bit fancier but only catches the AWS key. Also, it took 13s to complete.

~/projects/github/ethyca/fides main ❯ git log -1 | head -n 3
commit 6e76b1ba0e30919414b883766add4957ebc64114
Author: Paul Sanders <paul@ethyca.com>
Date:   Mon Oct 24 05:34:12 2022 -0700
~/projects/github/ethyca/fides main ❯ code .pre-commit-config.yaml 
~/projects/github/ethyca/fides main ❯ cat .pre-commit-config.yaml 
minimum_pre_commit_version: "2"

repos:
  - repo: https://github.com/gitguardian/ggshield
    rev: v1.13.6
    hooks:
      - id: ggshield
        language_version: python3
        stages: [commit]
~/projects/github/ethyca/fides main !1 ❯ pre-commit install
pre-commit installed at .git/hooks/pre-commit
~/projects/github/ethyca/fides main !1 ❯ touch TESTSECRETS.md
~/projects/github/ethyca/fides main !1 ?1 ❯ code TESTSECRETS.md 
~/projects/github/ethyca/fides main !1 ?1 ❯ head -n 2 TESTSECRETS.md 
### Test Secrets for Secrets Detection Pre-Commit Hooks
All recently revoked: an AWS key, a Mailchip API key, a Mailgun API key, and a Stripe API key.
~/projects/github/ethyca/fides main !1 ?1 ❯ git add .
~/projects/github/ethyca/fides main +2 ❯ git commit -m "Will ggshield perform any better?"
[INFO] Initializing environment for https://github.com/gitguardian/ggshield.
[INFO] Installing environment for https://github.com/gitguardian/ggshield.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
GitGuardian Shield (pre-commit)..........................................Failed
- hook id: ggshield
- exit code: 1

secrets-engine-version: 2.77.0

🛡️  ⚔️  🛡️  1 incident has been found in file TESTSECRETS.md

>>> Incident 1(Secrets detection): AWS Keys (Validity: Invalid)  (Ignore with SHA: 99ea8e51c7a9f9387590b8491a75b2d2bd537dc823406ff19e0744dad7f9a948) (1 occurrence)
  2 | All recent…
  3 | …
  4 | …KEY_ID”, “AKIA************5KS6"…
                 |_____client_id____|
  5 | …RET_ID”, “VGlVuXp**************************1ZLSh9E”…
                 |_____________client_secret____________|
  6 | MAILCHIMP_…
  7 | MAILCHIMP_…

ThomasLaPiana commented 1 year ago

@daveqnet I'm enjoying this thread and your findings! Thanks for putting in this work

daveqnet commented 1 year ago

Hehehe, thanks @ThomasLaPiana! Documenting heavily as I go is my style, I never remember things otherwise! 😆

Trying to slowly reach a conclusion this week. 🤞

daveqnet-alt commented 1 year ago

A very quick closer look at the pre-commit hooks themselves:

gitleaks

Including something like the following in your local repo's .pre-commit-config.yaml...

repos:
  - repo: https://github.com/zricethezav/gitleaks
    rev: v8.15.0
    hooks:
      - id: gitleaks

... links to the following in https://github.com/zricethezav/gitleaks/blob/master/.pre-commit-hooks.yaml

- id: gitleaks
  name: Detect hardcoded secrets
  description: Detect hardcoded secrets using Gitleaks
  entry: gitleaks protect --verbose --redact --staged
  language: golang
  pass_filenames: false

Quoting from the Gitleaks docs:

The protect command is used to uncommitted changes in a git repo... When running protect on a git repository, gitleaks will parse the output of a git diff command... You can set the --staged flag to check for changes in commits that have been git added. The --staged flag should be used when running Gitleaks as a pre-commit.

The --redact flag redacts secrets from logs and stdout.

Obviously this is a Go hook.

detect-secrets

detect-secrets's starter .pre-commit-config.yaml looks like the following

repos:
-   repo: https://github.com/Yelp/detect-secrets
    rev: v1.4.0
    hooks:
    -   id: detect-secrets
        args: ['--baseline', '.secrets.baseline']
        exclude: package.lock.json

There's a bit more going on here in the pre-commit config e.g. args and excludes (help). I assume package.lock.json (not package-lock.json?) is being excluded because an npm dep tree woud never contain a secret. Including --baseline .secrets.baseline again just emphasizes the need for a baseline file to be maintained somewhere in the repo.

Anyway, it links to the following in https://github.com/Yelp/detect-secrets/blob/master/.pre-commit-hooks.yaml

-   id: detect-secrets
    name: Detect secrets
    description: Detects high entropy strings that are likely to be passwords.
    entry: detect-secrets-hook
    language: python
    # for backward compatibility
    files: .*

detect-secrets-hook can be used with a bunch of different flags beyond just --baseline. More info here: https://github.com/Yelp/detect-secrets/blob/master/.pre-commit-hooks.yaml

Obviously this is a Python hook.

One advantage detect-secrets may have over gitleaks is that any dev working on the fides repo will already have Python already installed. The same is not true of Go.

ethyca / fides