Improved privacy request error handling

galvana commented 1 week ago

Closes LA-102 and LA-68

Description Of Changes

This PR contains miscellaneous improvements to the privacy request flow.

Code Changes

Main change

Fixed issue where database sessions were no longer available after long-running access, erasure, or consent requests

Bug fixes

Fixed bug where datasets from disabled integrations were still used in the traversal
Fixed bug where privacy request did not transition from approved to in progress
Fixed bug where execution log timestamps were incorrect

Misc improvements

Updates the privacy request detail page to poll for the latest results as the request is processing
Adds a new Dataset traversal timeline event before the actual request starts
Removed the extra click between the activity timeline to the event logs
Adds the privacy request ID to all privacy request logs

Steps to Confirm

Set the following values in fides.toml
```
[execution]
use_dsr_3_0 = true
```

[celery] task_always_eager = false

2. Start the test environment with `nox -s dev -- postgres`, if this fails you might need to run `nox -s build`
3. Connect to the `postgres_example` database and run this SQL script [postgres_sample.txt](https://github.com/user-attachments/files/17807816/postgres_sample.txt)
4. Create this dataset 
[postgres_dataset.txt](https://github.com/user-attachments/files/17808004/postgres_dataset.txt)
5. Create a new system with a Postgres integration to the `postgres_example` database. After saving attach the dataset from the previous step

host: host.docker.internal port: 6432 username: postgres password: postgres database: postgres_example


6. Submit a privacy request via the Privacy Center for `jane@example.com`
7. Go the privacy request detail page for this request and approve it. You should see the page auto-update as the request processes
8. Go back to the integration and disable it, then submit another request. The Postgres dataset should not appear in the activity timeline

### Pre-Merge Checklist

* [ ] Issue requirements met
* [ ] All CI pipelines succeeded
* [ ] `CHANGELOG.md` updated
* Followup issues:
  * [ ] Followup issues created (include link)
  * [ ] No followup issues
* Database migrations:
  * [ ] Ensure that your downrev is up to date with the latest revision on `main`
  * [ ] Ensure that your `downgrade()` migration is correct and works
    * [ ] If a downgrade migration is not possible for this change, please call this out in the PR description!
  * [ ] No migrations
* Documentation:
  * [ ] Documentation complete, [PR opened in fidesdocs](https://github.com/ethyca/fidesdocs/pulls)
  * [ ] Documentation [issue created in fidesdocs](https://github.com/ethyca/fidesdocs/issues/new/choose)
  * [ ] If there are any new client scopes created as part of the pull request, remember to update public-facing documentation that references our scope registry
  * [ ] No documentation updates required

[LA-102]: https://ethyca.atlassian.net/browse/LA-102?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

[LA-68]: https://ethyca.atlassian.net/browse/LA-68?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

vercel[bot] commented 1 week ago

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

cypress[bot] commented 1 week ago

fides Run #11114

Run Properties: Passed #11114 • 514567e1b2 ℹ️: Merge 9baf5e4931813ad690114a9e4fc8501efd380d9e into 0a7a2b2feb03717bbc5297eb0bc9...

Project	`fides`
Branch Review	`refs/pull/5500/merge`
Run status	`Passed #11114`
Run duration	`00m 39s`
Commit	`514567e1b2 ℹ️: Merge 9baf5e4931813ad690114a9e4fc8501efd380d9e into 0a7a2b2feb03717bbc5297eb0bc9...`
Committer	`Adrian Galvan`
View all properties for this run ↗︎

Test results
Failures	`0`
Flaky	`0`
Pending	`0`
Skipped	`0`
Passing	`4`
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.
View all changes introduced in this branch ↗︎

codecov[bot] commented 1 week ago

Codecov Report

Attention: Patch coverage is 97.56098% with 1 line in your changes missing coverage. Please review.

Project coverage is 85.19%. Comparing base (1c8b2d7) to head (2a1c2cc).

Files with missing lines	Patch %	Lines
.../service/privacy_request/request_runner_service.py	88.88%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #5500 +/- ## ========================================== + Coverage 85.18% 85.19% +0.01% ========================================== Files 387 387 Lines 24265 24285 +20 Branches 2646 2649 +3 ========================================== + Hits 20669 20689 +20 Misses 3042 3042 Partials 554 554 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

cypress[bot] commented 3 days ago

fides Run #11115

Run Properties: Passed #11115 • 97edac0ea8: Improved privacy request error handling (#5500)

Project	`fides`
Branch Review	`main`
Run status	`Passed #11115`
Run duration	`00m 39s`
Commit	`97edac0ea8: Improved privacy request error handling (#5500)`
Committer	`Adrian Galvan`
View all properties for this run ↗︎

Test results
Failures	`0`
Flaky	`0`
Pending	`0`
Skipped	`0`
Passing	`4`
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.
View all changes introduced in this branch ↗︎

ethyca / fides