Implement (small) data lifecycle (backup/archive ES)

raft-tech / TANF-app

Repo for development of a new TANF Data Reporting System

Other

16 stars 3 forks source link

Implement (small) data lifecycle (backup/archive ES) #3004

Closed andrew-jameson closed 2 months ago

andrew-jameson commented 4 months ago

Description: Pivoted on #2978, below is the new plan to deliver feature. We first need to implement a data lifecycle to have a clear space for the new incoming data and this ticket will be to capture the groundwork and orchestration needed. This will deliver reparsing feature.

Acceptance Criteria:

[ ] App can successfully deploy even with conflicting ES document indices
[ ] App can archive/delete live data set
[ ] Testing Checklist has been run and all tests pass
[ ] README is updated, if necessary

Tasks:

[x] Build queryset for certain date block aka prior submission period, drop relevant PG data
[x] Drop old ES, deploy new ES documents (re-index)
[x] Implement data lifecycle: backup/archive ES & PG to new S3 [ STRETCH ]
[x] Run Testing Checklist and confirm all tests pass

Notes:

Supporting Documentation:

Open Questions:

robgendron commented 3 months ago

Some unexpected work, may result with additional tickets. 2 points remaining with this initial ticket.

robgendron commented 3 months ago

Follow-on tickets being created. 3 point remaining on the initial.

ADPennington commented 3 months ago

https://github.com/raft-tech/TANF-app/issues/2978#issuecomment-2113309306

robgendron commented 3 months ago

Developing strategy to work with data team on this. Given constraints, estimating to take over a month to deliver feature (After August 3).

ADPennington commented 3 months ago

here are my notes from the post-standup meeting on 7/3. @andrew-jameson @robgendron feel free to add on to anything i missed or that needs clarity cc: @lfrohlich

data team will start manual resubmissions for FY23 next week for states and territories only. in advance we will send a notification to states and territories to help keep TDP user activity at a minimum.
reparsing PR is currently in raft review, but this is still basically an MVP. dev team doesn't have enough prod-like test files to work with, so data team will plan to share some after the manual resubmission process starts next week.
performance is also an unknown with network traffic. @andrew-jameson plans to join data team next week to observe a few resubmissions to help track anything noteworthy from the logs.
end-of-month for reparsing feature to prod is optimistic (it may just be an MVP). but hopefully the manual effort will help inform how to get a more mature version of it scoped.

robgendron commented 2 months ago

Update: Testing currently

robgendron commented 2 months ago

Update: Scheduling time with Alex to review.