Handle soft-deleted datasets in Elasticsearch - Githubissues

tsdataclinic / scout

Scout is a data discovery tool to explore open data portals worldwide.

https://scout.tsdataclinic.com

Apache License 2.0

33 stars 12 forks source link

Handle soft-deleted datasets in Elasticsearch #351

Closed jps327 closed 1 year ago

jps327 commented 1 year ago

Summary

Remove soft-deleted datasets from Elasticsearch
Prevent re-adding soft-deleted datasets to Elasticsearch on successive portal syncs
Recover soft-deleted datasets when they are seen in Socrata again so they can get added back to Elasticsearch on that portal sync

Screenshots or Videos (if applicable)

n/a

Related Issues

Closes #343

Test Plan

Verify soft-deleted datasets are removed from Elasticsearch
Verify soft-deleted datasets don't get reinserted to Elasticsearch on successive portal syncs
Verify that a previously soft-deleted dataset gets recovered (in both postgres and elasticsearch) when it is seen in Socrata again

Checklist Before Requesting a Review

[x] I have performed a self-review of my code
[x] My code follows the Style Guidelines and Best Practices outlined in the project wiki
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made changes to the documentation, if applicable
[x] My change generates no new warnings or failed tests
[ ] If it is a core feature, I have added thorough tests
[ ] I have implemented analytics, if applicable