ETL Superset logs so we can understand who is using it and how!
There are a few sources of superset information
1) the superset database and
2) the Cloud logs produced by Cloud Run.
3) the Cloud Monitoring Metrics of the Cloud Run instance.
I think the database will provide valuable information like users, queries and logs. This comment has some more information. The Cloud Logs will likely have everything in the logs table plus some additional Cloud Run operational information which could be helpful for monitoring the performance of specific operations. The Monitoring metrics information would be helpful for monitoring overall performance of the deployment. Monitoring metrics are currently retained for 6 weeks so we'd need to archive them.
I could see a few different phases for this work:
Phase I
Connect the superset db to superset and build a few simple dashboards off the raw tables.
ETL Superset logs so we can understand who is using it and how!
There are a few sources of superset information 1) the superset database and 2) the Cloud logs produced by Cloud Run. 3) the Cloud Monitoring Metrics of the Cloud Run instance.
I think the database will provide valuable information like users, queries and logs. This comment has some more information. The Cloud Logs will likely have everything in the logs table plus some additional Cloud Run operational information which could be helpful for monitoring the performance of specific operations. The Monitoring metrics information would be helpful for monitoring overall performance of the deployment. Monitoring metrics are currently retained for 6 weeks so we'd need to archive them.
I could see a few different phases for this work:
Phase I
Phase II
Phase II