search

mozilla / glam

Mozilla's primary interactive dashboard for examining the distribution of telemetry values.
https://glam.telemetry.mozilla.org
Mozilla Public License 2.0
20 stars 23 forks source link

Move GLAM ETL Legacy to moz-fx-data-glam-prod-fca7 #2809

Closed edugfilho closed 4 months ago

edugfilho commented 5 months ago

GLAM ETL Legacy has been running on moz-fx-data-shared-prod. Recently some of its tasks started timing out due to resource contention and I had to pause it. (https://bugzilla.mozilla.org/show_bug.cgi?id=1890351) We want to move it to moz-fx-data-glam-prod-fca7 which will allow it to run in its own resource pool and not compete with other ETLs. Another assumption we have is that the glam-prod resource pool is a more appropriate one for this type of workload, which is more computationally intensive instead of volume-heavy, once it is on-demand and bills per bytes scanned instead of computing power used.

For more details on the bigger picture around this see https://mozilla-hub.atlassian.net/browse/DSRE-1599

edugfilho commented 4 months ago

The DAG has been migrated. I'm currently fixing bugs from the PRs above and backfilling data so I can resume the DAG

edugfilho commented 4 months ago

The bugs have been fixed and the DAG is running on glam-prod, catching up. GLAM data won't be updated until the DAG is fully caught up. Currently finishing May 11

edugfilho commented 4 months ago

The DAG has finally run end-to-end on the new billing project with no issues. GLAM is up-to-date!