Rewrite WMRC ETL updates

agrc / planning-queue

A place to manage issues that do not have a dedicated repository or code... yet.

MIT License

0 stars 0 forks source link

Rewrite WMRC ETL updates #269

Open nathankota opened 7 months ago

nathankota commented 7 months ago

Benefit

Improve the ETL processes for the WMRC recycling map/app data from google sheets and salesforce, and to calculate the precision rates more efficiently than Arcade expressions in the app.

Acceptance Criteria

[ ] google sheet ETL (used oil collection data)
[ ] salesforce ETL (solid waste data)
[ ] precision rates calculations

Notes

Another possible task is to open a GitHub repo for the WMRC project overall, mostly to capture and track issues.

Risks

No response

Issue Reference

refs #221

jacobdadams commented 5 months ago

Sprint 3

Get connected to DEQ salesforce sandbox (Radiation data are restricted, which constrains our access to the sandbox. Jake will ding Barry. )
Write Salesforce ETL
- Create feature service for summary stats
- Calc summary stats
- Update skid w/Salesforce ETL

jacobdadams commented 5 months ago

Still waiting on Barry Blackington & Tom to get access to their salesforce org

steveoh commented 4 months ago

Still waiting on Barry Blackington & Tom to get access to their salesforce org

Do you know when the last reminder was given to Barry?

nathankota commented 4 months ago

Sprint 4

ding Barry & Tom

nathankota commented 2 months ago

Sprint 2

[x] connect app
[x] update palletjack to account for REST limitations
[x] validate access to correct data

nathankota commented 1 month ago

Q4 Sprint 4

[x] validate report/queries by comparing extracted data with Stevie's report

nathankota commented 1 month ago

Sprint 5

[ ] Finalize ETL component
- terraform, gcp function, etc.

From meeting with Stevie:

[x] Make sure update is adding any new locations found on sheet
[x] Add MRF facility category symbology
[x] De-dupe facility ids
- Try to use LastModifiedDate field
- Report duplicates in status email
[x] Add Rachel Boyer to email list