mozilla-services / mozilla-pipeline-schemas

Schemas for Mozilla's data ingestion pipeline and data lake outputs
https://protosaur.dev/mps-deploys/
Other
46 stars 91 forks source link

Bug 1827238 - Add dlsource to messaging_system.onboarding_v1.attribution #773

Closed lelilia closed 1 year ago

lelilia commented 1 year ago

Bug 1827238

Adding dlsource to messaging_system.onboarding_v1.attribution

I recently added dlsource to a lot of telemetry schemas as well (see #768), but this looks to be the only other schema outside of telemetry that uses dlsource and dltoken

It might be more elegant to move the attribution part out of the templates/include/telemetry/environment.1.schema.json into a common template that can be used here as well

Checklist for reviewer:

For glean changes: - [ ] Update templates/include/glean/CHANGELOG.md

For modifications to schemas in restricted namespaces (see CODEOWNERS):

lelilia commented 1 year ago

ok then I will not merge but append this PR tomorrow

dataops-ci-bot commented 1 year ago

Integration report for "refactored attribution to common include template"

bq_schema_51c39d9c-4eec69b6.diff

Click to expand! ```diff diff --new-file --exclude '*.txt' /app/integration/51c39d9c/messaging-system.onboarding.1.bq /app/integration/4eec69b6/messaging-system.onboarding.1.bq 22c22,28 < "description": "Unique token created at Firefox download time, see bug 1757451", --- > "description": "Identifier that indicates where installations of Firefox originate, see bug 1827233", > "mode": "NULLABLE", > "name": "dlsource", > "type": "STRING" > }, > { > "description": "Unique token created at Firefox download time, see bug 1677497", 28c34 < "description": "Funnel experiment parameters, see bug 1567339", --- > "description": "funnel experiment parameters, see bug 1567339", 46c52 < "description": "Derived user agent, see bug 1595063", --- > "description": "derived user agent, see bug 1595063", 52c58 < "description": "Funnel experiment parameters, see bug 1567339", --- > "description": "funnel experiment parameters, see bug 1567339", diff --new-file --exclude '*.txt' /app/integration/51c39d9c/telemetry.crash.4.bq /app/integration/4eec69b6/telemetry.crash.4.bq 1734,1739d1733 < "description": "Optional, contains a string describing the reason the crash occurred", < "mode": "NULLABLE", < "name": "shutdown_reason", < "type": "STRING" < }, < { diff --new-file --exclude '*.txt' /app/integration/51c39d9c/telemetry.third-party-modules.4.bq /app/integration/4eec69b6/telemetry.third-party-modules.4.bq 1501,1506d1500 < "description": "The debug ID (or breakpad ID) of the module.", < "mode": "NULLABLE", < "name": "debug_id", < "type": "STRING" < }, < { ```

compact_schema_51c39d9c-4eec69b6.diff

Click to expand! ```diff diff --new-file --exclude '*.bq' /app/integration/51c39d9c/messaging-system.onboarding.1.txt /app/integration/4eec69b6/messaging-system.onboarding.1.txt 3a4 > root.attribution.dlsource STRING diff --new-file --exclude '*.bq' /app/integration/51c39d9c/telemetry.crash.4.txt /app/integration/4eec69b6/telemetry.crash.4.txt 250d249 < root.payload.metadata.shutdown_reason STRING diff --new-file --exclude '*.bq' /app/integration/51c39d9c/telemetry.third-party-modules.4.txt /app/integration/4eec69b6/telemetry.third-party-modules.4.txt 211d210 < root.payload.modules.[].debug_id STRING ```
lelilia commented 1 year ago

@scholtzan please have a quick look again to see if the refactoring is up to code or if there is some documentation or anything missing now thank you

sean-rose commented 1 year ago

FYI, I believe this is causing the probe_scraper.mozilla_schema_generator task to fail.