Change fxci metric export to process all data from yesterday, rather than from 10 minutes ago

mozilla / docker-etl

Collection of dockerized ETL jobs managed by data engineering.

Mozilla Public License 2.0

16 stars 14 forks source link

Closed ahal closed 1 month ago

ahal commented 1 month ago

Checklist for reviewer:

[ ] Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title)
[ ] Scan the PR and verify that no changes (particularly to .circleci/config.yml) will cause environment variables (particularly credentials) to be exposed in test logs
[ ] Ensure the container image will be using permissions granted to telemetry-airflow responsibly.

ahal commented 1 month ago

It wouldn't hurt, but the impression I got is that they wanted us to self-serve this as much as possible.

ahal commented 1 month ago

@akkomar hi, could you land this so it triggers the image push please?

Feel free to also review if you like, we're not entirely sure how hands on/off you and your team want to be for changes like this.

akkomar commented 1 month ago

We discussed this on Slack and decided to merge as-is for now, and @ahal will look into simplifying as described in https://github.com/mozilla/docker-etl/pull/253#discussion_r1705849802.

ahal commented 1 month ago

Thanks for your help!

For posterity, here's the bug I filed to track the improvements: https://bugzilla.mozilla.org/show_bug.cgi?id=1912067

Even more than the simplification, the ability to easily re-populate older days makes this worth implementing