cityofaustin / atd-data-tech

Austin Transportation Data & Technology Services
17 stars 2 forks source link

VZD: Clear any old Hasura events (tech debt) #3592

Closed sergiogcx closed 4 years ago

sergiogcx commented 4 years ago

Events can take up a large amount of data in Postgres, it could make Postres slower and this can also bloat the size of the backups.

If possible, it might be a good idea to create backups of the event logs separately, then remove them from Hasura entirely.

sergiogcx commented 4 years ago

I found this article in Hasura:

https://hasura.io/docs/1.0/graphql/manual/event-triggers/clean-up.html

sergiogcx commented 4 years ago

We may need a separate DAG that runs every week or every few days at night. It may also be possible to export the logs, but I don't really see much value in that idea because we can always run an event as needed. For production log history may be more valuable than in staging, that is one thing to consider.

sergiogcx commented 4 years ago

I've created a PR for this purpose: https://github.com/cityofaustin/atd-airflow/pull/26

sergiogcx commented 4 years ago

It's working by the way:

Before Execution: 2020-08-20_21-18-00.png

Execution: 2020-08-20_21-20-49.png

After: 2020-08-20_21-21-02.png