bug 1910670: GLAM create extract tables individually

[edugfilho]

Checklist for reviewer:

• [ ] Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title).
• [ ] If the PR comes from a fork, trigger integration CI tests by running the Push to upstream workflow and provide the <username>:<branch> of the fork as parameter. The parameter will also show up in the logs of the manual-trigger-required-for-fork CI task together with more detailed instructions.
• [ ] If adding a new field to a query, ensure that the schema and dependent downstream schemas have been updated.
• [ ] When adding a new derived dataset, ensure that data is not available already (fully or partially) and recommend extending an existing dataset in favor of creating new ones. Data can be available in the bigquery-etl repository, looker-hub or in looker-spoke-default.

For modifications to schemas in restricted namespaces (see CODEOWNERS):

• [ ] Follow the change control procedure

┆Issue is synchronized with this Jira Task

[dataops-ci-bot]

Integration report for "GLAM proper extract tables"

sql.diff

Click to expand!

```diff diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_generated_funnels.py /tmp/workspace/generated-sql/dags/bqetl_generated_funnels.py --- /tmp/workspace/main-generated-sql/dags/bqetl_generated_funnels.py 2024-07-30 19:06:05.000000000 +0000 +++ /tmp/workspace/generated-sql/dags/bqetl_generated_funnels.py 2024-07-30 19:22:35.000000000 +0000 @@ -288,6 +288,21 @@ ) ) + monitor_frontend_derived__monitor_dashboard_user_journey_funnels__v1 = bigquery_etl_query( + task_id="monitor_frontend_derived__monitor_dashboard_user_journey_funnels__v1", + destination_table="monitor_dashboard_user_journey_funnels_v1", + dataset_id="monitor_frontend_derived", + project_id="moz-fx-data-shared-prod", + owner="ksiegler@mozilla.org", + email=[ + "ascholtz@mozilla.com", + "ksiegler@mozilla.org", + "telemetry-alerts@mozilla.com", + ], + date_partition_parameter="submission_date", + depends_on_past=False, + ) + accounts_frontend_derived__email_first_reg_login_funnels_by_service__v1.set_upstream( wait_for_copy_deduplicate_all ) @@ -345,3 +360,7 @@ firefox_accounts_derived__registration_funnels_legacy_events__v1.set_upstream( wait_for_firefox_accounts_derived__fxa_stdout_events__v1 ) + + monitor_frontend_derived__monitor_dashboard_user_journey_funnels__v1.set_upstream( + wait_for_copy_deduplicate_all + ) Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/mobile_attribution_clients: schema.yaml Only in /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_client_probe_counts_extract_v1: query.sql Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_beta_v1: query.sql Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_beta_v1: schema.yaml Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_nightly_v1: query.sql Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_nightly_v1: schema.yaml Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_release_v1: query.sql Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_release_v1: schema.yaml diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml 2024-07-30 18:59:59.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml 2024-07-30 19:11:47.000000000 +0000 @@ -1,49 +1,49 @@ fields: -- mode: NULLABLE - name: submission_date +- name: submission_date type: DATE -- mode: NULLABLE - name: source + mode: NULLABLE +- name: source type: STRING -- mode: NULLABLE - name: event_type + mode: NULLABLE +- name: event_type type: STRING -- mode: NULLABLE - name: form_factor + mode: NULLABLE +- name: form_factor type: STRING -- mode: NULLABLE - name: country + mode: NULLABLE +- name: country type: STRING -- mode: NULLABLE - name: subdivision1 + mode: NULLABLE +- name: subdivision1 type: STRING -- mode: NULLABLE - name: advertiser + mode: NULLABLE +- name: advertiser type: STRING -- mode: NULLABLE - name: release_channel + mode: NULLABLE +- name: release_channel type: STRING -- mode: NULLABLE - name: position + mode: NULLABLE +- name: position type: INTEGER -- mode: NULLABLE - name: provider + mode: NULLABLE +- name: provider type: STRING -- mode: NULLABLE - name: match_type + mode: NULLABLE +- name: match_type type: STRING -- mode: NULLABLE - name: normalized_os + mode: NULLABLE +- name: normalized_os type: STRING -- mode: NULLABLE - name: suggest_data_sharing_enabled + mode: NULLABLE +- name: suggest_data_sharing_enabled type: BOOLEAN -- mode: NULLABLE - name: event_count + mode: NULLABLE +- name: event_count type: INTEGER -- mode: NULLABLE - name: user_count + mode: NULLABLE +- name: user_count type: INTEGER -- mode: NULLABLE - name: query_type + mode: NULLABLE +- name: query_type type: STRING + mode: NULLABLE diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml 2024-07-30 18:59:59.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml 2024-07-30 19:11:47.000000000 +0000 @@ -1,40 +1,40 @@ fields: -- mode: NULLABLE - name: submission_date +- name: submission_date type: DATE -- mode: NULLABLE - name: form_factor + mode: NULLABLE +- name: form_factor type: STRING -- mode: NULLABLE - name: country + mode: NULLABLE +- name: country type: STRING -- mode: NULLABLE - name: advertiser + mode: NULLABLE +- name: advertiser type: STRING -- mode: NULLABLE - name: normalized_os + mode: NULLABLE +- name: normalized_os type: STRING -- mode: NULLABLE - name: release_channel + mode: NULLABLE +- name: release_channel type: STRING -- mode: NULLABLE - name: position + mode: NULLABLE +- name: position type: INTEGER -- mode: NULLABLE - name: provider + mode: NULLABLE +- name: provider type: STRING -- mode: NULLABLE - name: match_type + mode: NULLABLE +- name: match_type type: STRING -- mode: NULLABLE - name: suggest_data_sharing_enabled + mode: NULLABLE +- name: suggest_data_sharing_enabled type: BOOLEAN -- mode: NULLABLE - name: impression_count + mode: NULLABLE +- name: impression_count type: INTEGER -- mode: NULLABLE - name: click_count + mode: NULLABLE +- name: click_count type: INTEGER -- mode: NULLABLE - name: query_type + mode: NULLABLE +- name: query_type type: STRING + mode: NULLABLE diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/attribution_clients/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/attribution_clients/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/attribution_clients/schema.yaml 2024-07-30 19:00:45.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/attribution_clients/schema.yaml 2024-07-30 19:09:34.000000000 +0000 @@ -17,9 +17,6 @@ description: 'The source of a profile installation. ' -- name: submission_timestamp - type: TIMESTAMP - mode: NULLABLE - name: adjust_ad_group type: STRING mode: NULLABLE @@ -35,9 +32,6 @@ - name: adjust_attribution_timestamp type: TIMESTAMP mode: NULLABLE -- name: submission_timestamp_1 - type: TIMESTAMP - mode: NULLABLE - name: play_store_attribution_campaign type: STRING mode: NULLABLE @@ -50,9 +44,6 @@ - name: play_store_attribution_timestamp type: TIMESTAMP mode: NULLABLE -- name: submission_timestamp_2 - type: TIMESTAMP - mode: NULLABLE - name: meta_attribution_app type: STRING mode: NULLABLE diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml 2024-07-30 18:59:59.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml 2024-07-30 19:09:29.000000000 +0000 @@ -26,6 +26,9 @@ - name: adjust_network type: STRING mode: NULLABLE +- name: install_source + type: STRING + mode: NULLABLE - name: retained_week_2 type: BOOLEAN mode: NULLABLE diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml 2024-07-30 18:59:59.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml 2024-07-30 19:09:30.000000000 +0000 @@ -48,6 +48,10 @@ description: 'The type of source of a client installation. ' +- name: install_source + type: STRING + mode: NULLABLE + description: null - name: new_profiles type: INTEGER mode: NULLABLE diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/use_counters/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/use_counters/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/use_counters/schema.yaml 2024-07-30 19:00:45.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/use_counters/schema.yaml 2024-07-30 19:11:30.000000000 +0000 @@ -1,11 +1,11 @@ fields: - name: normalized_app_id - mode: NULLABLE type: STRING + mode: NULLABLE description: App ID of the channel data was received from - name: normalized_channel - mode: NULLABLE type: STRING + mode: NULLABLE description: Normalized channel name - name: additional_properties type: STRING diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_ios/attribution_clients/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_ios/attribution_clients/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_ios/attribution_clients/schema.yaml 2024-07-30 19:00:43.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_ios/attribution_clients/schema.yaml 2024-07-30 19:12:24.000000000 +0000 @@ -11,9 +11,6 @@ type: INTEGER mode: NULLABLE description: null -- name: submission_timestamp - type: TIMESTAMP - mode: NULLABLE - name: adjust_ad_group type: STRING mode: NULLABLE diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/focus_android/use_counters/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/focus_android/use_counters/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/focus_android/use_counters/schema.yaml 2024-07-30 19:00:44.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/focus_android/use_counters/schema.yaml 2024-07-30 19:16:17.000000000 +0000 @@ -1,11 +1,11 @@ fields: - name: normalized_app_id - mode: NULLABLE type: STRING + mode: NULLABLE description: App ID of the channel data was received from - name: normalized_channel - mode: NULLABLE type: STRING + mode: NULLABLE description: Normalized channel name - name: additional_properties type: STRING diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/google_search_console/search_impressions_by_page/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/google_search_console/search_impressions_by_page/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/google_search_console/search_impressions_by_page/schema.yaml 2024-07-30 18:59:59.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/google_search_console/search_impressions_by_page/schema.yaml 2024-07-30 19:11:32.000000000 +0000 @@ -6,9 +6,10 @@ - name: site_url type: STRING mode: NULLABLE - description: |- - For domain properties, this will be `sc-domain:` followed by the domain name. - For URL-prefix properties, it will be the full URL of the property definition. + description: 'For domain properties, this will be `sc-domain:` followed by the domain + name. + + For URL-prefix properties, it will be the full URL of the property definition.' - name: site_domain_name type: STRING mode: NULLABLE @@ -16,103 +17,110 @@ - name: page_url type: STRING mode: NULLABLE - description: |- - The final page URL linked by a search result after any skip redirects. - This will be null for anonymized Discover impressions. + description: 'The final page URL linked by a search result after any skip redirects. + + This will be null for anonymized Discover impressions.' - name: page_domain_name type: STRING mode: NULLABLE - description: |- - Domain name of the page URL. - This will be null for anonymized Discover impressions. + description: 'Domain name of the page URL. + + This will be null for anonymized Discover impressions.' - name: page_path type: STRING mode: NULLABLE - description: |- - The path part of the page URL. - This will be null for anonymized Discover impressions. + description: 'The path part of the page URL. + + This will be null for anonymized Discover impressions.' - name: localized_site_code type: STRING mode: NULLABLE - description: |- - Localized site code such as `en-US` or `de` found in the first segment of the page URL path (if any). - This will be null for anonymized Discover impressions. + description: 'Localized site code such as `en-US` or `de` found in the first segment + of the page URL path (if any). + + This will be null for anonymized Discover impressions.' - name: localized_site type: STRING mode: NULLABLE - description: |- - Description of the localized site language and/or country based on `localized_site_code` (if any). - This will be null for anonymized Discover impressions. + description: 'Description of the localized site language and/or country based on + `localized_site_code` (if any). + + This will be null for anonymized Discover impressions.' - name: localized_site_language_code type: STRING mode: NULLABLE - description: |- - Localized site language code in ISO-639-alpha-2 format found in the first segment of the page URL path (if any). - This will be null for anonymized Discover impressions. + description: 'Localized site language code in ISO-639-alpha-2 format found in the + first segment of the page URL path (if any). + + This will be null for anonymized Discover impressions.' - name: localized_site_language type: STRING mode: NULLABLE - description: |- - Localized site language based on `localized_site_language_code` (if any). - This will be null for anonymized Discover impressions. + description: 'Localized site language based on `localized_site_language_code` (if + any). + + This will be null for anonymized Discover impressions.' - name: query type: STRING mode: NULLABLE - description: |- - The search query. - This will be null for anonymized search impressions, and all Discover and Google News search impressions. + description: 'The search query. + + This will be null for anonymized search impressions, and all Discover and Google + News search impressions.' - name: query_type type: STRING mode: NULLABLE - description: |- - Type of search query: - * Anonymized: Query was redacted by Google to protect the users' privacy. - * Brand: Query contained one or more Mozilla brand keywords. - * Non-Brand: Query didn't contain any Mozilla brand keywords. - * Unknown: Query couldn't be classified. - This will be null for all Discover and Google News search impressions. + description: "Type of search query:\n * Anonymized: Query was redacted by Google\ + \ to protect the users' privacy.\n * Brand: Query contained one or more Mozilla\ + \ brand keywords.\n * Non-Brand: Query didn't contain any Mozilla brand keywords.\n\ + \ * Unknown: Query couldn't be classified.\nThis will be null for all Discover\ + \ and Google News search impressions." - name: is_anonymized type: BOOLEAN mode: NULLABLE - description: |- - Whether Google has anonymized the search impression to protect the users' privacy. + description: 'Whether Google has anonymized the search impression to protect the + users'' privacy. + The `query` field will be null for anonymized search impressions. - The `country_code`, `page_url`, and related fields will be null for anonymized Discover impressions. + + The `country_code`, `page_url`, and related fields will be null for anonymized + Discover impressions.' - name: has_good_page_experience type: BOOLEAN mode: NULLABLE - description: |- - Whether Google Search considers the page to be providing a good page experience. - This will be null when the source data wasn't exported directly to BigQuery by Google. + description: 'Whether Google Search considers the page to be providing a good page + experience. + + This will be null when the source data wasn''t exported directly to BigQuery by + Google.' - name: search_type type: STRING mode: NULLABLE - description: |- - Where the link was seen by the user: - * Web: In Google Search's default "All" tab. - * Image: In Google Search's "Images" tab. - * Video: In Google Search's "Videos" tab. - * News: In Google Search's "News" tab. - * Discover: In Google's Discover feed. - * Google News: On news.google.com or in the Google News app on Android and iOS. + description: "Where the link was seen by the user:\n * Web: In Google Search's\ + \ default \"All\" tab.\n * Image: In Google Search's \"Images\" tab.\n * Video:\ + \ In Google Search's \"Videos\" tab.\n * News: In Google Search's \"News\" tab.\n\ + \ * Discover: In Google's Discover feed.\n * Google News: On news.google.com\ + \ or in the Google News app on Android and iOS." - name: search_appearance type: STRING mode: NULLABLE - description: |- - How the search result appeared (e.g. normal result, translated result, video). - This will be null when the source data wasn't exported directly to BigQuery by Google. + description: 'How the search result appeared (e.g. normal result, translated result, + video). + + This will be null when the source data wasn''t exported directly to BigQuery by + Google.' - name: user_country_code type: STRING mode: NULLABLE - description: |- - Country from which the user was searching, in ISO-3166-1-alpha-3 format. - This will be null for anonymized Discover impressions. + description: 'Country from which the user was searching, in ISO-3166-1-alpha-3 format. + + This will be null for anonymized Discover impressions.' - name: user_country type: STRING mode: NULLABLE - description: |- - Country from which the user was searching. - This will be null for anonymized Discover impressions. + description: 'Country from which the user was searching. + + This will be null for anonymized Discover impressions.' - name: user_region type: STRING mode: NULLABLE @@ -124,13 +132,15 @@ - name: device_type type: STRING mode: NULLABLE - description: |- - The type of device on which the user was searching: Desktop, Mobile, or Tablet. - This will be null for Discover impressions. + description: 'The type of device on which the user was searching: Desktop, Mobile, + or Tablet. + + This will be null for Discover impressions.' - name: impressions type: INTEGER mode: NULLABLE - description: The number of times that search results with a link to the page were shown to a user. + description: The number of times that search results with a link to the page were + shown to a user. - name: clicks type: INTEGER mode: NULLABLE @@ -138,6 +148,7 @@ - name: average_position type: FLOAT mode: NULLABLE - description: |- - The average position of the page in the search results, where `1` is the topmost position. - This will be null for Discover and Google News search impressions. + description: 'The average position of the page in the search results, where `1` + is the topmost position. + + This will be null for Discover and Google News search impressions.' diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/google_search_console/search_impressions_by_site/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/google_search_console/search_impressions_by_site/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/google_search_console/search_impressions_by_site/schema.yaml 2024-07-30 18:59:59.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/google_search_console/search_impressions_by_site/schema.yaml 2024-07-30 19:11:32.000000000 +0000 @@ -6,9 +6,10 @@ - name: site_url type: STRING mode: NULLABLE - description: |- - For domain properties, this will be `sc-domain:` followed by the domain name. - For URL-prefix properties, it will be the full URL of the property definition. + description: 'For domain properties, this will be `sc-domain:` followed by the domain + name. + + For URL-prefix properties, it will be the full URL of the property definition.' - name: site_domain_name type: STRING mode: NULLABLE @@ -20,27 +21,23 @@ - name: query_type type: STRING mode: NULLABLE - description: |- - Type of search query: - * Anonymized: Query was redacted by Google to protect the users' privacy. - * Brand: Query contained one or more Mozilla brand keywords. - * Non-Brand: Query didn't contain any Mozilla brand keywords. - * Unknown: Query couldn't be classified. + description: "Type of search query:\n * Anonymized: Query was redacted by Google\ + \ to protect the users' privacy.\n * Brand: Query contained one or more Mozilla\ + \ brand keywords.\n * Non-Brand: Query didn't contain any Mozilla brand keywords.\n\ + \ * Unknown: Query couldn't be classified." - name: is_anonymized type: BOOLEAN mode: NULLABLE - description: |- - Whether Google has anonymized the search impression to protect the users' privacy. - The `query` field will be null for anonymized search impressions. + description: 'Whether Google has anonymized the search impression to protect the + users'' privacy. + + The `query` field will be null for anonymized search impressions.' - name: search_type type: STRING mode: NULLABLE - description: |- - Where the link was seen by the user: - * Web: In Google Search's default "All" tab. - * Image: In Google Search's "Images" tab. - * Video: In Google Search's "Videos" tab. - * News: In Google Search's "News" tab. + description: "Where the link was seen by the user:\n * Web: In Google Search's\ + \ default \"All\" tab.\n * Image: In Google Search's \"Images\" tab.\n * Video:\ + \ In Google Search's \"Videos\" tab.\n * News: In Google Search's \"News\" tab." - name: user_country_code type: STRING mode: NULLABLE @@ -60,17 +57,20 @@ - name: device_type type: STRING mode: NULLABLE - description: |- - The type of device on which the user was searching: Desktop, Mobile, or Tablet. + description: 'The type of device on which the user was searching: Desktop, Mobile, + or Tablet.' - name: impressions type: INTEGER mode: NULLABLE - description: The number of times that search results with at least one link to the site were shown to a user. + description: The number of times that search results with at least one link to the + site were shown to a user. - name: clicks type: INTEGER mode: NULLABLE - description: The number of times a user clicked at least one search result link to the site. + description: The number of times a user clicked at least one search result link + to the site. - name: average_top_position type: FLOAT mode: NULLABLE - description: The average top position of the site in the search results, where `1` is the topmost position. + description: The average top position of the site in the search results, where `1` + is the topmost position. diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/klar_android/use_counters/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/klar_android/use_counters/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/klar_android/use_counters/schema.yaml 2024-07-30 19:00:43.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/klar_android/use_counters/schema.yaml 2024-07-30 19:15:20.000000000 +0000 @@ -1,11 +1,11 @@ fields: - name: normalized_app_id - mode: NULLABLE type: STRING + mode: NULLABLE description: App ID of the channel data was received from - name: normalized_channel - mode: NULLABLE type: STRING + mode: NULLABLE description: Normalized channel name - name: additional_properties type: STRING diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/monitor_frontend_derived/event_monitoring_live_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitor_frontend_derived/event_monitoring_live_v1/metadata.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/monitor_frontend_derived/event_monitoring_live_v1/metadata.yaml 2024-07-30 19:01:20.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitor_frontend_derived/event_monitoring_live_v1/metadata.yaml 2024-07-30 19:17:20.000000000 +0000 @@ -1,10 +1,19 @@ +friendly_name: Event Monitoring Live +description: |- + Materialized view of experimentation related events + coming from monitor_frontend. +owners: +- ascholtz@mozilla.com +- akomar@mozilla.com +labels: + materialized_view: true + owner1: ascholtz + owner2: akomar +bigquery: null workgroup_access: - role: roles/bigquery.dataViewer members: - workgroup:mozilla-confidential - - workgroup:dataops-managed/external-fides - -# Generated by bigquery_etl.dependency references: materialized_view.sql: - moz-fx-data-shared-prod.monitor_frontend_live.events_v1 diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/monitor_frontend_derived/monitor_dashboard_user_journey_funnels_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitor_frontend_derived/monitor_dashboard_user_journey_funnels_v1/metadata.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/monitor_frontend_derived/monitor_dashboard_user_journey_funnels_v1/metadata.yaml 2024-07-30 19:01:20.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitor_frontend_derived/monitor_dashboard_user_journey_funnels_v1/metadata.yaml 2024-07-30 19:17:20.000000000 +0000 @@ -1,10 +1,26 @@ +friendly_name: Monitor Dashboard User Journey Funnels +description: |- + Please provide a description for the query +owners: +- ksiegler@mozilla.org +labels: + incremental: true + dag: bqetl_generated_funnels + owner1: ksiegler +scheduling: + dag_name: bqetl_generated_funnels +bigquery: + time_partitioning: + type: day + field: submission_date + require_partition_filter: false + expiration_days: null + range_partitioning: null + clustering: null workgroup_access: - role: roles/bigquery.dataViewer members: - workgroup:mozilla-confidential - - workgroup:dataops-managed/external-fides - -# Generated by bigquery_etl.dependency references: query.sql: - mozdata.monitor_frontend.events_unnested diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix/geckoview_version/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix/geckoview_version/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix/geckoview_version/schema.yaml 2024-07-30 18:59:59.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix/geckoview_version/schema.yaml 2024-07-30 19:13:11.000000000 +0000 @@ -1,7 +1,13 @@ fields: -- type: DATETIME - name: build_hour -- type: INTEGER - name: geckoview_major_version -- type: INTEGER - name: n_pings +- name: build_hour + type: DATETIME + mode: NULLABLE + description: null +- name: geckoview_major_version + type: INTEGER + mode: NULLABLE + description: null +- name: n_pings + type: INTEGER + mode: NULLABLE + description: null diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry/eng_workflow_build_parquet/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/eng_workflow_build_parquet/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry/eng_workflow_build_parquet/schema.yaml 2024-07-30 19:00:46.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/eng_workflow_build_parquet/schema.yaml 2024-07-30 19:12:48.000000000 +0000 @@ -28,36 +28,48 @@ - name: artifact type: BOOLEAN mode: NULLABLE + description: true if --enable-artifact-builds - name: ccache type: BOOLEAN mode: NULLABLE + description: true if ccache is in use (--with-ccache) - name: compiler type: STRING mode: NULLABLE + description: The compiler type in use (CC_TYPE) - name: debug type: BOOLEAN mode: NULLABLE + description: true if build is debug (--enable-debug) - name: icecream type: BOOLEAN mode: NULLABLE + description: true if icecream in use - name: opt type: BOOLEAN mode: NULLABLE + description: true if build is optimized (--enable-optimize) - name: sccache type: BOOLEAN mode: NULLABLE + description: true if ccache in use is sccache + description: Selected build options - name: client_id type: STRING mode: NULLABLE + description: A UUID to uniquely identify a client - name: command type: STRING mode: NULLABLE + description: The mach command that was invoked - name: duration_ms type: FLOAT mode: NULLABLE + description: Command duration in milliseconds - name: success type: BOOLEAN mode: NULLABLE + description: true if the command succeeded - name: system type: RECORD mode: NULLABLE @@ -65,30 +77,40 @@ - name: os type: STRING mode: NULLABLE + description: Operating system - name: cpu_brand type: STRING mode: NULLABLE + description: CPU brand string from CPUID - name: drive_is_ssd type: BOOLEAN mode: NULLABLE + description: true if the source directory is on a solid-state disk - name: logical_cores type: INTEGER mode: NULLABLE + description: Number of logical CPU cores present - name: memory_gb type: INTEGER mode: NULLABLE + description: System memory in GB - name: physical_cores type: INTEGER mode: NULLABLE + description: Number of physical CPU cores present - name: virtual_machine type: BOOLEAN mode: NULLABLE + description: true if the OS appears to be running in a virtual machine - name: time type: STRING mode: NULLABLE + description: Time at which this event happened - name: exception type: STRING mode: NULLABLE + description: If a Python exception was encountered during the execution of the command, + this value contains the result of calling `repr` on the exception object. - name: file_types_changed type: RECORD mode: REPEATED @@ -96,9 +118,11 @@ - name: count type: INTEGER mode: NULLABLE + description: Count of changed files with this extension - name: ext type: STRING mode: NULLABLE + description: File extension - name: build_attrs type: RECORD mode: NULLABLE @@ -106,6 +130,9 @@ - name: clobber type: BOOLEAN mode: NULLABLE + description: true if the build was a clobber/full build - name: cpu_percent type: FLOAT mode: NULLABLE + description: cpu utilization observed during a build + description: Attributes characterizing a build diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry/mobile_attribution_clients/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/mobile_attribution_clients/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry/mobile_attribution_clients/schema.yaml 1970-01-01 00:00:00.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/mobile_attribution_clients/schema.yaml 2024-07-30 19:12:55.000000000 +0000 @@ -0,0 +1,55 @@ +fields: +- name: submission_date + type: DATE + mode: NULLABLE +- name: client_id + type: STRING + mode: NULLABLE +- name: sample_id + type: INTEGER + mode: NULLABLE +- name: paid_vs_organic + type: STRING + mode: NULLABLE +- name: is_suspicious_device_client + type: BOOLEAN + mode: NULLABLE +- name: adjust_ad_group + type: STRING + mode: NULLABLE +- name: adjust_campaign + type: STRING + mode: NULLABLE +- name: adjust_creative + type: STRING + mode: NULLABLE +- name: adjust_network + type: STRING + mode: NULLABLE +- name: adjust_attribution_timestamp + type: TIMESTAMP + mode: NULLABLE +- name: play_store_attribution_campaign + type: STRING + mode: NULLABLE +- name: play_store_attribution_medium + type: STRING + mode: NULLABLE +- name: play_store_attribution_source + type: STRING + mode: NULLABLE +- name: play_store_attribution_timestamp + type: TIMESTAMP + mode: NULLABLE +- name: meta_attribution_app + type: STRING + mode: NULLABLE +- name: meta_attribution_timestamp + type: TIMESTAMP + mode: NULLABLE +- name: install_source + type: STRING + mode: NULLABLE +- name: product_name + type: STRING + mode: NULLABLE diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_client_probe_counts_extract_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_client_probe_counts_extract_v1/metadata.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_client_probe_counts_extract_v1/metadata.yaml 2024-07-30 19:02:04.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_client_probe_counts_extract_v1/metadata.yaml 2024-07-30 19:17:21.000000000 +0000 @@ -10,7 +10,4 @@ - role: roles/bigquery.dataViewer members: - workgroup:mozilla-confidential -references: - query.sql: - - moz-fx-data-shared-prod.telemetry_derived.client_probe_counts - - moz-fx-data-shared-prod.telemetry_derived.glam_sample_counts_v1 +references: {} diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_client_probe_counts_extract_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_client_probe_counts_extract_v1/query.sql --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_client_probe_counts_extract_v1/query.sql 2024-07-30 19:00:00.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_client_probe_counts_extract_v1/query.sql 1970-01-01 00:00:00.000000000 +0000 @@ -1,106 +0,0 @@ -WITH final_probe_extract AS ( - SELECT - app_version, - COALESCE(os, "*") AS os, - COALESCE(app_build_id, "*") AS app_build_id, - process, - metric, - REPLACE(KEY, r"\x00", "") AS KEY, - client_agg_type, - metric_type, - total_users, - -- Using MAX instead of COALESCE since this is not in the GROUP BY. - MAX(IF(agg_type = "histogram", mozfun.glam.histogram_cast_json(aggregates), NULL)) AS histogram, - MAX( - IF(agg_type = "histogram", mozfun.glam.histogram_cast_json(non_norm_aggregates), NULL) - ) AS non_norm_histogram - FROM - `moz-fx-data-shared-prod.telemetry_derived.client_probe_counts` - WHERE - channel = @channel - AND app_version IS NOT NULL - AND total_users > 375 - AND CHAR_LENGTH(KEY) <= 200 - GROUP BY - channel, - app_version, - app_build_id, - os, - metric, - metric_type, - KEY, - process, - client_agg_type, - total_users -), -glam_sample_counts AS ( - SELECT - fsc1.os, - fsc1.app_version, - fsc1.app_build_id, - fsc1.metric, - fsc1.key, - fsc1.process, - fsc1.agg_type, - CASE - WHEN fsc1.agg_type IN ('max', 'min', 'sum', 'avg') - AND fsc2.agg_type = 'count' - THEN fsc2.total_sample - ELSE fsc1.total_sample - END AS total_sample - FROM - `moz-fx-data-shared-prod.telemetry_derived.glam_sample_counts_v1` fsc1 - INNER JOIN - `moz-fx-data-shared-prod.telemetry_derived.glam_sample_counts_v1` fsc2 - ON fsc1.os = fsc2.os - AND fsc1.app_build_id = fsc2.app_build_id - AND fsc1.app_version = fsc2.app_version - AND fsc1.metric = fsc2.metric - AND fsc1.key = fsc2.key - AND fsc1.channel = fsc2.channel - AND fsc1.process = fsc2.process - WHERE - fsc1.channel = @channel -) -SELECT - cp.app_version, - cp.os, - cp.app_build_id, - cp.process, - cp.metric, - cp.key, - cp.client_agg_type, - cp.metric_type, - total_users, - histogram, - CASE - WHEN client_agg_type = '' - THEN 0 - ELSE CAST(total_sample AS BIGNUMERIC) - END AS total_sample, - non_norm_histogram -FROM - final_probe_extract cp -LEFT JOIN - glam_sample_counts sc - ON sc.os = cp.os - AND sc.app_build_id = cp.app_build_id - AND sc.app_version = cp.app_version - AND sc.metric = cp.metric - AND sc.key = cp.key - AND sc.process = cp.process - AND total_sample IS NOT NULL - AND (sc.agg_type = cp.client_agg_type OR cp.client_agg_type = '') -GROUP BY - cp.app_version, - cp.os, - cp.app_build_id, - cp.process, - cp.metric, - cp.key, - cp.client_agg_type, - cp.metric_type, - total_users, - total_sample, - histogram, - non_norm_histogram diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_beta_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_beta_v1/metadata.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_beta_v1/metadata.yaml 2024-07-30 19:02:04.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_beta_v1/metadata.yaml 2024-07-30 19:17:20.000000000 +0000 @@ -11,4 +11,7 @@ - role: roles/bigquery.dataViewer members: - workgroup:mozilla-confidential -references: {} +references: + query.sql: + - moz-fx-data-shared-prod.telemetry_derived.client_probe_counts + - moz-fx-data-shared-prod.telemetry_derived.glam_sample_counts_v1 diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_beta_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_beta_v1/query.sql --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_beta_v1/query.sql 1970-01-01 00:00:00.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_beta_v1/query.sql 2024-07-30 19:01:02.000000000 +0000 @@ -0,0 +1,106 @@ +WITH final_probe_extract AS ( + SELECT + app_version, + COALESCE(os, "*") AS os, + COALESCE(app_build_id, "*") AS app_build_id, + process, + metric, + REPLACE(KEY, r"\x00", "") AS KEY, + client_agg_type, + metric_type, + total_users, + -- Using MAX instead of COALESCE since this is not in the GROUP BY. + MAX(IF(agg_type = "histogram", mozfun.glam.histogram_cast_json(aggregates), NULL)) AS histogram, + MAX( + IF(agg_type = "histogram", mozfun.glam.histogram_cast_json(non_norm_aggregates), NULL) + ) AS non_norm_histogram + FROM + `moz-fx-data-shared-prod.telemetry_derived.client_probe_counts` + WHERE + channel = "beta" + AND app_version IS NOT NULL + AND total_users > 375 + AND CHAR_LENGTH(KEY) <= 200 + GROUP BY + channel, + app_version, + app_build_id, + os, + metric, + metric_type, + KEY, + process, + client_agg_type, + total_users +), +glam_sample_counts AS ( + SELECT + fsc1.os, + fsc1.app_version, + fsc1.app_build_id, + fsc1.metric, + fsc1.key, + fsc1.process, + fsc1.agg_type, + CASE + WHEN fsc1.agg_type IN ('max', 'min', 'sum', 'avg') + AND fsc2.agg_type = 'count' + THEN fsc2.total_sample + ELSE fsc1.total_sample + END AS total_sample + FROM + `moz-fx-data-shared-prod.telemetry_derived.glam_sample_counts_v1` fsc1 + INNER JOIN + `moz-fx-data-shared-prod.telemetry_derived.glam_sample_counts_v1` fsc2 + ON fsc1.os = fsc2.os + AND fsc1.app_build_id = fsc2.app_build_id + AND fsc1.app_version = fsc2.app_version + AND fsc1.metric = fsc2.metric + AND fsc1.key = fsc2.key + AND fsc1.channel = fsc2.channel + AND fsc1.process = fsc2.process + WHERE + fsc1.channel = @channel +) +SELECT + cp.app_version, + cp.os, + cp.app_build_id, + cp.process, + cp.metric, + cp.key, + cp.client_agg_type, + cp.metric_type, + total_users, + histogram, + CASE + WHEN client_agg_type = '' + THEN 0 + ELSE CAST(total_sample AS BIGNUMERIC) + END AS total_sample, + non_norm_histogram +FROM + final_probe_extract cp +LEFT JOIN + glam_sample_counts sc + ON sc.os = cp.os + AND sc.app_build_id = cp.app_build_id + AND sc.app_version = cp.app_version + AND sc.metric = cp.metric + AND sc.key = cp.key + AND sc.process = cp.process + AND total_sample IS NOT NULL + AND (sc.agg_type = cp.client_agg_type OR cp.client_agg_type = '') +GROUP BY + cp.app_version, + cp.os, + cp.app_build_id, + cp.process, + cp.metric, + cp.key, + cp.client_agg_type, + cp.metric_type, + total_users, + total_sample, + histogram, + non_norm_histogram diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_beta_v1/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_beta_v1/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_beta_v1/schema.yaml 1970-01-01 00:00:00.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_beta_v1/schema.yaml 2024-07-30 19:01:02.000000000 +0000 @@ -0,0 +1,37 @@ +fields: +- name: app_version + type: INTEGER + mode: NULLABLE +- name: os + type: STRING + mode: NULLABLE +- name: app_build_id + type: STRING + mode: NULLABLE +- name: process + type: STRING + mode: NULLABLE +- name: metric + type: STRING + mode: NULLABLE +- name: key + type: STRING + mode: NULLABLE +- name: client_agg_type + type: STRING + mode: NULLABLE +- name: metric_type + type: STRING + mode: NULLABLE +- name: total_users + type: INTEGER + mode: NULLABLE +- name: histogram + type: STRING + mode: NULLABLE +- name: total_sample + type: BIGNUMERIC + mode: NULLABLE +- name: non_norm_histogram + type: STRING + mode: NULLABLE diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_nightly_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_nightly_v1/metadata.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_nightly_v1/metadata.yaml 2024-07-30 19:02:04.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_nightly_v1/metadata.yaml 2024-07-30 19:17:20.000000000 +0000 @@ -11,4 +11,7 @@ - role: roles/bigquery.dataViewer members: - workgroup:mozilla-confidential -references: {} +references: + query.sql: + - moz-fx-data-shared-prod.telemetry_derived.client_probe_counts + - moz-fx-data-shared-prod.telemetry_derived.glam_sample_counts_v1 diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_nightly_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_nightly_v1/query.sql --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_nightly_v1/query.sql 1970-01-01 00:00:00.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_nightly_v1/query.sql 2024-07-30 19:01:02.000000000 +0000 @@ -0,0 +1,106 @@ +WITH final_probe_extract AS ( + SELECT + app_version, + COALESCE(os, "*") AS os, + COALESCE(app_build_id, "*") AS app_build_id, + process, + metric, + REPLACE(KEY, r"\x00", "") AS KEY, + client_agg_type, + metric_type, + total_users, + -- Using MAX instead of COALESCE since this is not in the GROUP BY. + MAX(IF(agg_type = "histogram", mozfun.glam.histogram_cast_json(aggregates), NULL)) AS histogram, + MAX( + IF(agg_type = "histogram", mozfun.glam.histogram_cast_json(non_norm_aggregates), NULL) + ) AS non_norm_histogram + FROM + `moz-fx-data-shared-prod.telemetry_derived.client_probe_counts` + WHERE + channel = "nightly" + AND app_version IS NOT NULL + AND total_users > 375 + AND CHAR_LENGTH(KEY) <= 200 + GROUP BY + channel, + app_version, + app_build_id, + os, + metric, + metric_type, + KEY, + process, + client_agg_type, + total_users +), +glam_sample_counts AS ( + SELECT + fsc1.os, + fsc1.app_version, + fsc1.app_build_id, + fsc1.metric, + fsc1.key, + fsc1.process, + fsc1.agg_type, + CASE + WHEN fsc1.agg_type IN ('max', 'min', 'sum', 'avg') + AND fsc2.agg_type = 'count' + THEN fsc2.total_sample + ELSE fsc1.total_sample + END AS total_sample + FROM + `moz-fx-data-shared-prod.telemetry_derived.glam_sample_counts_v1` fsc1 + INNER JOIN + `moz-fx-data-shared-prod.telemetry_derived.glam_sample_counts_v1` fsc2 + ON fsc1.os = fsc2.os + AND fsc1.app_build_id = fsc2.app_build_id + AND fsc1.app_version = fsc2.app_version + AND fsc1.metric = fsc2.metric + AND fsc1.key = fsc2.key + AND fsc1.channel = fsc2.channel + AND fsc1.process = fsc2.process + WHERE + fsc1.channel = @channel +) +SELECT + cp.app_version, + cp.os, + cp.app_build_id, + cp.process, + cp.metric, + cp.key, + cp.client_agg_type, + cp.metric_type, + total_users, + histogram, + CASE + WHEN client_agg_type = '' + THEN 0 + ELSE CAST(total_sample AS BIGNUMERIC) + END AS total_sample, + non_norm_histogram +FROM + final_probe_extract cp +LEFT JOIN + glam_sample_counts sc + ON sc.os = cp.os + AND sc.app_build_id = cp.app_build_id + AND sc.app_version = cp.app_version + AND sc.metric = cp.metric + AND sc.key = cp.key + AND sc.process = cp.process + AND total_sample IS NOT NULL + AND (sc.agg_type = cp.client_agg_type OR cp.client_agg_type = '') +GROUP BY + cp.app_version, + cp.os, + cp.app_build_id, + cp.process, + cp.metric, + cp.key, + cp.client_agg_type, + cp.metric_type, + total_users, + total_sample, + histogram, + non_norm_histogram diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_nightly_v1/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_nightly_v1/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_nightly_v1/schema.yaml 1970-01-01 00:00:00.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_nightly_v1/schema.yaml 2024-07-30 19:01:02.000000000 +0000 @@ -0,0 +1,37 @@ +fields: +- name: app_version + type: INTEGER + mode: NULLABLE +- name: os + type: STRING + mode: NULLABLE +- name: app_build_id + type: STRING + mode: NULLABLE +- name: process + type: STRING + mode: NULLABLE +- name: metric + type: STRING + mode: NULLABLE +- name: key + type: STRING + mode: NULLABLE +- name: client_agg_type + type: STRING + mode: NULLABLE +- name: metric_type + type: STRING + mode: NULLABLE +- name: total_users + type: INTEGER + mode: NULLABLE +- name: histogram + type: STRING + mode: NULLABLE +- name: total_sample + type: BIGNUMERIC + mode: NULLABLE +- name: non_norm_histogram + type: STRING + mode: NULLABLE diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_release_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_release_v1/metadata.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_release_v1/metadata.yaml 2024-07-30 19:02:04.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_release_v1/metadata.yaml 2024-07-30 19:17:21.000000000 +0000 @@ -11,4 +11,7 @@ - role: roles/bigquery.dataViewer members: - workgroup:mozilla-confidential -references: {} +references: + query.sql: + - moz-fx-data-shared-prod.telemetry_derived.client_probe_counts + - moz-fx-data-shared-prod.telemetry_derived.glam_sample_counts_v1 diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_release_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_release_v1/query.sql --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_release_v1/query.sql 1970-01-01 00:00:00.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_release_v1/query.sql 2024-07-30 19:01:03.000000000 +0000 @@ -0,0 +1,106 @@ +WITH final_probe_extract AS ( + SELECT + app_version, + COALESCE(os, "*") AS os, + COALESCE(app_build_id, "*") AS app_build_id, + process, + metric, + REPLACE(KEY, r"\x00", "") AS KEY, + client_agg_type, + metric_type, + total_users, + -- Using MAX instead of COALESCE since this is not in the GROUP BY. + MAX(IF(agg_type = "histogram", mozfun.glam.histogram_cast_json(aggregates), NULL)) AS histogram, + MAX( + IF(agg_type = "histogram", mozfun.glam.histogram_cast_json(non_norm_aggregates), NULL) + ) AS non_norm_histogram + FROM + `moz-fx-data-shared-prod.telemetry_derived.client_probe_counts` + WHERE + channel = "release" + AND app_version IS NOT NULL + AND total_users > 375 + AND CHAR_LENGTH(KEY) <= 200 + GROUP BY + channel, + app_version, + app_build_id, + os, + metric, + metric_type, + KEY, + process, + client_agg_type, + total_users +), +glam_sample_counts AS ( + SELECT + fsc1.os, + fsc1.app_version, + fsc1.app_build_id, + fsc1.metric, + fsc1.key, + fsc1.process, + fsc1.agg_type, + CASE + WHEN fsc1.agg_type IN ('max', 'min', 'sum', 'avg') + AND fsc2.agg_type = 'count' + THEN fsc2.total_sample + ELSE fsc1.total_sample + END AS total_sample + FROM + `moz-fx-data-shared-prod.telemetry_derived.glam_sample_counts_v1` fsc1 + INNER JOIN + `moz-fx-data-shared-prod.telemetry_derived.glam_sample_counts_v1` fsc2 + ON fsc1.os = fsc2.os + AND fsc1.app_build_id = fsc2.app_build_id + AND fsc1.app_version = fsc2.app_version + AND fsc1.metric = fsc2.metric + AND fsc1.key = fsc2.key + AND fsc1.channel = fsc2.channel + AND fsc1.process = fsc2.process + WHERE + fsc1.channel = @channel +) +SELECT + cp.app_version, + cp.os, + cp.app_build_id, + cp.process, + cp.metric, + cp.key, + cp.client_agg_type, + cp.metric_type, + total_users, + histogram, + CASE + WHEN client_agg_type = '' + THEN 0 + ELSE CAST(total_sample AS BIGNUMERIC) + END AS total_sample, + non_norm_histogram +FROM + final_probe_extract cp +LEFT JOIN + glam_sample_counts sc + ON sc.os = cp.os + AND sc.app_build_id = cp.app_build_id + AND sc.app_version = cp.app_version + AND sc.metric = cp.metric + AND sc.key = cp.key + AND sc.process = cp.process + AND total_sample IS NOT NULL + AND (sc.agg_type = cp.client_agg_type OR cp.client_agg_type = '') +GROUP BY + cp.app_version, + cp.os, + cp.app_build_id, + cp.process, + cp.metric, + cp.key, + cp.client_agg_type, + cp.metric_type, + total_users, + total_sample, + histogram, + non_norm_histogram diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_release_v1/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_release_v1/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_release_v1/schema.yaml 1970-01-01 00:00:00.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_release_v1/schema.yaml 2024-07-30 19:01:03.000000000 +0000 @@ -0,0 +1,37 @@ +fields: +- name: app_version + type: INTEGER + mode: NULLABLE +- name: os + type: STRING + mode: NULLABLE +- name: app_build_id + type: STRING + mode: NULLABLE +- name: process + type: STRING + mode: NULLABLE +- name: metric + type: STRING + mode: NULLABLE +- name: key + type: STRING + mode: NULLABLE +- name: client_agg_type + type: STRING + mode: NULLABLE +- name: metric_type + type: STRING + mode: NULLABLE +- name: total_users + type: INTEGER + mode: NULLABLE +- name: histogram + type: STRING + mode: NULLABLE +- name: total_sample + type: BIGNUMERIC + mode: NULLABLE +- name: non_norm_histogram + type: STRING + mode: NULLABLE ```

Link to full diff

[edugfilho]

We can also delete the glam_client_probe_counts_extract_v1 directory right?

Yes. I'm clearly not putting enough attention to this whole thing.

[dataops-ci-bot]

Integration report for "Remove glam_client_probe_counts_extract_v1 dir"

sql.diff

Click to expand!

```diff diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_generated_funnels.py /tmp/workspace/generated-sql/dags/bqetl_generated_funnels.py --- /tmp/workspace/main-generated-sql/dags/bqetl_generated_funnels.py 2024-07-30 19:56:26.000000000 +0000 +++ /tmp/workspace/generated-sql/dags/bqetl_generated_funnels.py 2024-07-30 20:15:29.000000000 +0000 @@ -288,6 +288,21 @@ ) ) + monitor_frontend_derived__monitor_dashboard_user_journey_funnels__v1 = bigquery_etl_query( + task_id="monitor_frontend_derived__monitor_dashboard_user_journey_funnels__v1", + destination_table="monitor_dashboard_user_journey_funnels_v1", + dataset_id="monitor_frontend_derived", + project_id="moz-fx-data-shared-prod", + owner="ksiegler@mozilla.org", + email=[ + "ascholtz@mozilla.com", + "ksiegler@mozilla.org", + "telemetry-alerts@mozilla.com", + ], + date_partition_parameter="submission_date", + depends_on_past=False, + ) + accounts_frontend_derived__email_first_reg_login_funnels_by_service__v1.set_upstream( wait_for_copy_deduplicate_all ) @@ -345,3 +360,7 @@ firefox_accounts_derived__registration_funnels_legacy_events__v1.set_upstream( wait_for_firefox_accounts_derived__fxa_stdout_events__v1 ) + + monitor_frontend_derived__monitor_dashboard_user_journey_funnels__v1.set_upstream( + wait_for_copy_deduplicate_all + ) Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/mobile_attribution_clients: schema.yaml Only in /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived: glam_client_probe_counts_extract_v1 Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_beta_v1: query.sql Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_beta_v1: schema.yaml Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_nightly_v1: query.sql Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_nightly_v1: schema.yaml Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_release_v1: query.sql Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/glam_extract_firefox_release_v1: schema.yaml diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml 2024-07-30 19:50:23.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml 2024-07-30 19:59:35.000000000 +0000 @@ -1,49 +1,49 @@ fields: -- mode: NULLABLE - name: submission_date +- name: submission_date type: DATE -- mode: NULLABLE - name: source + mode: NULLABLE +- name: source type: STRING -- mode: NULLABLE - name: event_type + mode: NULLABLE +- name: event_type type: STRING -- mode: NULLABLE - name: form_factor + mode: NULLABLE +- name: form_factor type: STRING -- mode: NULLABLE - name: country + mode: NULLABLE +- name: country type: STRING -- mode: NULLABLE - name: subdivision1 + mode: NULLABLE +- name: subdivision1 type: STRING -- mode: NULLABLE - name: advertiser + mode: NULLABLE +- name: advertiser type: STRING -- mode: NULLABLE - name: release_channel + mode: NULLABLE +- name: release_channel type: STRING -- mode: NULLABLE - name: position + mode: NULLABLE +- name: position type: INTEGER -- mode: NULLABLE - name: provider + mode: NULLABLE +- name: provider type: STRING -- mode: NULLABLE - name: match_type + mode: NULLABLE +- name: match_type type: STRING -- mode: NULLABLE - name: normalized_os + mode: NULLABLE +- name: normalized_os type: STRING -- mode: NULLABLE - name: suggest_data_sharing_enabled + mode: NULLABLE +- name: suggest_data_sharing_enabled type: BOOLEAN -- mode: NULLABLE - name: event_count + mode: NULLABLE +- name: event_count type: INTEGER -- mode: NULLABLE - name: user_count + mode: NULLABLE +- name: user_count type: INTEGER -- mode: NULLABLE - name: query_type + mode: NULLABLE +- name: query_type type: STRING + mode: NULLABLE diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml 2024-07-30 19:50:23.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml 2024-07-30 19:59:35.000000000 +0000 @@ -1,40 +1,40 @@ fields: -- mode: NULLABLE - name: submission_date +- name: submission_date type: DATE -- mode: NULLABLE - name: form_factor + mode: NULLABLE +- name: form_factor type: STRING -- mode: NULLABLE - name: country + mode: NULLABLE +- name: country type: STRING -- mode: NULLABLE - name: advertiser + mode: NULLABLE +- name: advertiser type: STRING -- mode: NULLABLE - name: normalized_os + mode: NULLABLE +- name: normalized_os type: STRING -- mode: NULLABLE - name: release_channel + mode: NULLABLE +- name: release_channel type: STRING -- mode: NULLABLE - name: position + mode: NULLABLE +- name: position type: INTEGER -- mode: NULLABLE - name: provider + mode: NULLABLE +- name: provider type: STRING -- mode: NULLABLE - name: match_type + mode: NULLABLE +- name: match_type type: STRING -- mode: NULLABLE - name: suggest_data_sharing_enabled + mode: NULLABLE +- name: suggest_data_sharing_enabled type: BOOLEAN -- mode: NULLABLE - name: impression_count + mode: NULLABLE +- name: impression_count type: INTEGER -- mode: NULLABLE - name: click_count + mode: NULLABLE +- name: click_count type: INTEGER -- mode: NULLABLE - name: query_type + mode: NULLABLE +- name: query_type type: STRING + mode: NULLABLE diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/attribution_clients/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/attribution_clients/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/attribution_clients/schema.yaml 2024-07-30 19:50:59.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/attribution_clients/schema.yaml 2024-07-30 20:01:09.000000000 +0000 @@ -17,9 +17,6 @@ description: 'The source of a profile installation. ' -- name: submission_timestamp - type: TIMESTAMP - mode: NULLABLE - name: adjust_ad_group type: STRING mode: NULLABLE @@ -35,9 +32,6 @@ - name: adjust_attribution_timestamp type: TIMESTAMP mode: NULLABLE -- name: submission_timestamp_1 - type: TIMESTAMP - mode: NULLABLE - name: play_store_attribution_campaign type: STRING mode: NULLABLE @@ -50,9 +44,6 @@ - name: play_store_attribution_timestamp type: TIMESTAMP mode: NULLABLE -- name: submission_timestamp_2 - type: TIMESTAMP - mode: NULLABLE - name: meta_attribution_app type: STRING mode: NULLABLE diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml 2024-07-30 19:50:23.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml 2024-07-30 20:01:04.000000000 +0000 @@ -26,6 +26,9 @@ - name: adjust_network type: STRING mode: NULLABLE +- name: install_source + type: STRING + mode: NULLABLE - name: retained_week_2 type: BOOLEAN mode: NULLABLE diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml 2024-07-30 19:50:23.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml 2024-07-30 20:01:05.000000000 +0000 @@ -48,6 +48,10 @@ description: 'The type of source of a client installation. ' +- name: install_source + type: STRING + mode: NULLABLE + description: null - name: new_profiles type: INTEGER mode: NULLABLE diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_ios/attribution_clients/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_ios/attribution_clients/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_ios/attribution_clients/schema.yaml 2024-07-30 19:51:00.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_ios/attribution_clients/schema.yaml 2024-07-30 19:59:45.000000000 +0000 @@ -11,9 +11,6 @@ type: INTEGER mode: NULLABLE description: null -- name: submission_timestamp - type: TIMESTAMP - mode: NULLABLE - name: adjust_ad_group type: STRING mode: NULLABLE diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/google_search_console/search_impressions_by_page/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/google_search_console/search_impressions_by_page/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/google_search_console/search_impressions_by_page/schema.yaml 2024-07-30 19:50:24.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/google_search_console/search_impressions_by_page/schema.yaml 2024-07-30 19:59:49.000000000 +0000 @@ -6,9 +6,10 @@ - name: site_url type: STRING mode: NULLABLE - description: |- - For domain properties, this will be `sc-domain:` followed by the domain name. - For URL-prefix properties, it will be the full URL of the property definition. + description: 'For domain properties, this will be `sc-domain:` followed by the domain + name. + + For URL-prefix properties, it will be the full URL of the property definition.' - name: site_domain_name type: STRING mode: NULLABLE @@ -16,103 +17,110 @@ - name: page_url type: STRING mode: NULLABLE - description: |- - The final page URL linked by a search result after any skip redirects. - This will be null for anonymized Discover impressions. + description: 'The final page URL linked by a search result after any skip redirects. + + This will be null for anonymized Discover impressions.' - name: page_domain_name type: STRING mode: NULLABLE - description: |- - Domain name of the page URL. - This will be null for anonymized Discover impressions. + description: 'Domain name of the page URL. + + This will be null for anonymized Discover impressions.' - name: page_path type: STRING mode: NULLABLE - description: |- - The path part of the page URL. - This will be null for anonymized Discover impressions. + description: 'The path part of the page URL. + + This will be null for anonymized Discover impressions.' - name: localized_site_code type: STRING mode: NULLABLE - description: |- - Localized site code such as `en-US` or `de` found in the first segment of the page URL path (if any). - This will be null for anonymized Discover impressions. + description: 'Localized site code such as `en-US` or `de` found in the first segment + of the page URL path (if any). + + This will be null for anonymized Discover impressions.' - name: localized_site type: STRING mode: NULLABLE - description: |- - Description of the localized site language and/or country based on `localized_site_code` (if any). - This will be null for anonymized Discover impressions. + description: 'Description of the localized site language and/or country based on + `localized_site_code` (if any). + + This will be null for anonymized Discover impressions.' - name: localized_site_language_code type: STRING mode: NULLABLE - description: |- - Localized site language code in ISO-639-alpha-2 format found in the first segment of the page URL path (if any). - This will be null for anonymized Discover impressions. + description: 'Localized site language code in ISO-639-alpha-2 format found in the + first segment of the page URL path (if any). + + This will be null for anonymized Discover impressions.' - name: localized_site_language type: STRING mode: NULLABLE - description: |- - Localized site language based on `localized_site_language_code` (if any). - This will be null for anonymized Discover impressions. + description: 'Localized site language based on `localized_site_language_code` (if + any). + + This will be null for anonymized Discover impressions.' - name: query type: STRING mode: NULLABLE - description: |- - The search query. - This will be null for anonymized search impressions, and all Discover and Google News search impressions. + description: 'The search query. + + This will be null for anonymized search impressions, and all Discover and Google + News search impressions.' - name: query_type type: STRING mode: NULLABLE - description: |- - Type of search query: - * Anonymized: Query was redacted by Google to protect the users' privacy. - * Brand: Query contained one or more Mozilla brand keywords. - * Non-Brand: Query didn't contain any Mozilla brand keywords. - * Unknown: Query couldn't be classified. - This will be null for all Discover and Google News search impressions. + description: "Type of search query:\n * Anonymized: Query was redacted by Google\ + \ to protect the users' privacy.\n * Brand: Query contained one or more Mozilla\ + \ brand keywords.\n * Non-Brand: Query didn't contain any Mozilla brand keywords.\n\ + \ * Unknown: Query couldn't be classified.\nThis will be null for all Discover\ + \ and Google News search impressions." - name: is_anonymized type: BOOLEAN mode: NULLABLE - description: |- - Whether Google has anonymized the search impression to protect the users' privacy. + description: 'Whether Google has anonymized the search impression to protect the + users'' privacy. + The `query` field will be null for anonymized search impressions. - The `country_code`, `page_url`, and related fields will be null for anonymized Discover impressions. + + The `country_code`, `page_url`, and related fields will be null for anonymized + Discover impressions.' - name: has_good_page_experience type: BOOLEAN mode: NULLABLE - description: |- - Whether Google Search considers the page to be providing a good page experience. - This will be null when the source data wasn't exported directly to BigQuery by Google. + description: 'Whether Google Search considers the page to be providing a good page + experience. + + This will be null when the source data wasn''t exported directly to BigQuery by + Google.' - name: search_type type: STRING mode: NULLABLE - description: |- - Where the link was seen by the user: - * Web: In Google Search's default "All" tab. - * Image: In Google Search's "Images" tab. - * Video: In Google Search's "Videos" tab. - * News: In Google Search's "News" tab. - * Discover: In Google's Discover feed. - * Google News: On news.google.com or in the Google News app on Android and iOS. + description: "Where the link was seen by the user:\n * Web: In Google Search's\ + \ default \"All\" tab.\n * Image: In Google Search's \"Images\" tab.\n * Video:\ + \ In Google Search's \"Videos\" tab.\n * News: In Google Search's \"News\" tab.\n\ + \ * Discover: In Google's Discover feed.\n * Google News: On news.google.com\ + \ or in the Google News app on Android and iOS." - name: search_appearance type: STRING mode: NULLABLE - description: |- - How the search result appeared (e.g. normal result, translated result, video). - This will be null when the source data wasn't exported directly to BigQuery by Google. + description: 'How the search result appeared (e.g. normal result, translated result, + video). + + This will be null when the source data wasn''t exported directly to BigQuery by + Google.' - name: user_country_code type: STRING mode: NULLABLE - description: |- - Country from which the user was searching, in ISO-3166-1-alpha-3 format. - This will be null for anonymized Discover impressions. + description: 'Country from which the user was searching, in ISO-3166-1-alpha-3 format. + + This will be null for anonymized Discover impressions.' - name: user_country type: STRING mode: NULLABLE - description: |- - Country from which the user was searching. - This will be null for anonymized Discover impressions. + description: 'Country from which the user was searching. + + This will be null for anonymized Discover impressions.' - name: user_region type: STRING mode: NULLABLE @@ -124,13 +132,15 @@ - name: device_type type: STRING mode: NULLABLE - description: |- - The type of device on which the user was searching: Desktop, Mobile, or Tablet. - This will be null for Discover impressions. + description: 'The type of device on which the user was searching: Desktop, Mobile, + or Tablet. + + This will be null for Discover impressions.' - name: impressions type: INTEGER mode: NULLABLE - description: The number of times that search results with a link to the page were shown to a user. + description: The number of times that search results with a link to the page were + shown to a user. - name: clicks type: INTEGER mode: NULLABLE @@ -138,6 +148,7 @@ - name: average_position type: FLOAT mode: NULLABLE - description: |- - The average position of the page in the search results, where `1` is the topmost position. - This will be null for Discover and Google News search impressions. + description: 'The average position of the page in the search results, where `1` + is the topmost position. + + This will be null for Discover and Google News search impressions.' diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/google_search_console/search_impressions_by_site/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/google_search_console/search_impressions_by_site/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/google_search_console/search_impressions_by_site/schema.yaml 2024-07-30 19:50:24.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/google_search_console/search_impressions_by_site/schema.yaml 2024-07-30 19:59:48.000000000 +0000 @@ -6,9 +6,10 @@ - name: site_url type: STRING mode: NULLABLE - description: |- - For domain properties, this will be `sc-domain:` followed by the domain name. - For URL-prefix properties, it will be the full URL of the property definition. + description: 'For domain properties, this will be `sc-domain:` followed by the domain + name. + + For URL-prefix properties, it will be the full URL of the property definition.' - name: site_domain_name type: STRING mode: NULLABLE @@ -20,27 +21,23 @@ - name: query_type type: STRING mode: NULLABLE - description: |- - Type of search query: - * Anonymized: Query was redacted by Google to protect the users' privacy. - * Brand: Query contained one or more Mozilla brand keywords. - * Non-Brand: Query didn't contain any Mozilla brand keywords. - * Unknown: Query couldn't be classified. + description: "Type of search query:\n * Anonymized: Query was redacted by Google\ + \ to protect the users' privacy.\n * Brand: Query contained one or more Mozilla\ + \ brand keywords.\n * Non-Brand: Query didn't contain any Mozilla brand keywords.\n\ + \ * Unknown: Query couldn't be classified." - name: is_anonymized type: BOOLEAN mode: NULLABLE - description: |- - Whether Google has anonymized the search impression to protect the users' privacy. - The `query` field will be null for anonymized search impressions. + description: 'Whether Google has anonymized the search impression to protect the + users'' privacy. + + The `query` field will be null for anonymized search impressions.' - name: search_type type: STRING mode: NULLABLE - description: |- - Where the link was seen by the user: - * Web: In Google Search's default "All" tab. - * Image: In Google Search's "Images" tab. - * Video: In Google Search's "Videos" tab. - * News: In Google Search's "News" tab. + description: "Where the link was seen by the user:\n * Web: In Google Search's\ + \ default \"All\" tab.\n * Image: In Google Search's \"Images\" tab.\n * Video:\ + \ In Google Search's \"Videos\" tab.\n * News: In Google Search's \"News\" tab." - name: user_country_code type: STRING mode: NULLABLE @@ -60,17 +57,20 @@ - name: device_type type: STRING mode: NULLABLE - description: |- - The type of device on which the user was searching: Desktop, Mobile, or Tablet. + description: 'The type of device on which the user was searching: Desktop, Mobile, + or Tablet.' - name: impressions type: INTEGER mode: NULLABLE - description: The number of times that search results with at least one link to the site were shown to a user. + description: The number of times that search results with at least one link to the + site were shown to a user. - name: clicks type: INTEGER mode: NULLABLE - description: The number of times a user clicked at least one search result link to the site. + description: The number of times a user clicked at least one search result link + to the site. - name: average_top_position type: FLOAT mode: NULLABLE - description: The average top position of the site in the search results, where `1` is the topmost position. + description: The average top position of the site in the search results, where `1` + is the topmost position. diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/monitor_frontend_derived/event_monitoring_live_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitor_frontend_derived/event_monitoring_live_v1/metadata.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/monitor_frontend_derived/event_monitoring_live_v1/metadata.yaml 2024-07-30 19:51:29.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitor_frontend_derived/event_monitoring_live_v1/metadata.yaml 2024-07-30 20:10:04.000000000 +0000 @@ -1,10 +1,19 @@ +friendly_name: Event Monitoring Live +description: |- + Materialized view of experimentation related events + coming from monitor_frontend. +owners: +- ascholtz@mozilla.com +- akomar@mozilla.com +labels: + materialized_view: true + owner1: ascholtz + owner2: akomar +bigquery: null workgroup_access: - role: roles/bigquery.dataViewer members: - workgroup:mozilla-confidential - - workgroup:dataops-managed/external-fides - -# Generated by bigquery_etl.dependency references: materialized_view.sql: - moz-fx-data-shared-prod.monitor_frontend_live.events_v1 diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/monitor_frontend_derived/monitor_dashboard_user_journey_funnels_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitor_frontend_derived/monitor_dashboard_user_journey_funnels_v1/metadata.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/monitor_frontend_derived/monitor_dashboard_user_journey_funnels_v1/metadata.yaml 2024-07-30 19:51:30.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitor_frontend_derived/monitor_dashboard_user_journey_funnels_v1/metadata.yaml 2024-07-30 20:10:04.000000000 +0000 @@ -1,10 +1,26 @@ +friendly_name: Monitor Dashboard User Journey Funnels +description: |- + Please provide a description for the query +owners: +- ksiegler@mozilla.org +labels: + incremental: true + dag: bqetl_generated_funnels + owner1: ksiegler +scheduling: + dag_name: bqetl_generated_funnels +bigquery: + time_partitioning: + type: day + field: submission_date + require_partition_filter: false + expiration_days: null + range_partitioning: null + clustering: null workgroup_access: - role: roles/bigquery.dataViewer members: - workgroup:mozilla-confidential - - workgroup:dataops-managed/external-fides - -# Generated by bigquery_etl.dependency references: query.sql: - mozdata.monitor_frontend.events_unnested diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix/geckoview_version/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix/geckoview_version/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix/geckoview_version/schema.yaml 2024-07-30 19:50:24.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix/geckoview_version/schema.yaml 2024-07-30 20:00:17.000000000 +0000 @@ -1,7 +1,13 @@ fields: -- type: DATETIME - name: build_hour -- type: INTEGER - name: geckoview_major_version -- type: INTEGER - name: n_pings +- name: build_hour + type: DATETIME + mode: NULLABLE + description: null +- name: geckoview_major_version + type: INTEGER + mode: NULLABLE + description: null +- name: n_pings + type: INTEGER + mode: NULLABLE + description: null diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry/clients_daily/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/clients_daily/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry/clients_daily/schema.yaml 2024-07-30 19:51:00.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/clients_daily/schema.yaml 2024-07-30 20:00:32.000000000 +0000 @@ -5,21 +5,27 @@ - name: submission_date type: DATE mode: NULLABLE + description: null - name: client_id type: STRING mode: NULLABLE + description: null - name: aborts_content_sum type: INTEGER mode: NULLABLE + description: null - name: aborts_gmplugin_sum type: INTEGER mode: NULLABLE + description: null - name: aborts_plugin_sum type: INTEGER mode: NULLABLE + description: null - name: active_addons_count_mean type: FLOAT mode: NULLABLE + description: null - name: active_addons type: RECORD mode: REPEATED @@ -27,69 +33,92 @@ - name: addon_id type: STRING mode: NULLABLE + description: null - name: blocklisted type: BOOLEAN mode: NULLABLE + description: null - name: name type: STRING mode: NULLABLE + description: null - name: user_disabled type: BOOLEAN mode: NULLABLE + description: null - name: app_disabled type: BOOLEAN mode: NULLABLE + description: null - name: version type: STRING mode: NULLABLE + description: null - name: scope type: INTEGER mode: NULLABLE + description: null - name: type type: STRING mode: NULLABLE + description: null - name: foreign_install type: BOOLEAN mode: NULLABLE + description: null - name: has_binary_components type: BOOLEAN mode: NULLABLE + description: null - name: install_day type: INTEGER mode: NULLABLE + description: null - name: update_day type: INTEGER mode: NULLABLE + description: null - name: signed_state type: INTEGER mode: NULLABLE + description: null - name: is_system type: BOOLEAN mode: NULLABLE + description: null - name: is_web_extension type: BOOLEAN mode: NULLABLE + description: null - name: multiprocess_compatible type: BOOLEAN mode: NULLABLE + description: null + description: null - name: active_hours_sum type: FLOAT mode: NULLABLE + description: null - name: addon_compatibility_check_enabled type: BOOLEAN mode: NULLABLE + description: null - name: app_build_id type: STRING mode: NULLABLE + description: null - name: app_display_version type: STRING mode: NULLABLE + description: null - name: app_name type: STRING mode: NULLABLE + description: null - name: app_version type: STRING mode: NULLABLE + description: null - name: attribution type: RECORD mode: NULLABLE @@ -97,15 +126,19 @@ - name: source type: STRING mode: NULLABLE + description: null - name: medium type: STRING mode: NULLABLE + description: null - name: campaign type: STRING mode: NULLABLE + description: null - name: content type: STRING mode: NULLABLE + description: null - name: experiment type: STRING mode: NULLABLE @@ -121,123 +154,163 @@ - name: ua type: STRING mode: NULLABLE + description: null - name: blocklist_enabled type: BOOLEAN mode: NULLABLE + description: null - name: channel type: STRING mode: NULLABLE + description: null - name: client_clock_skew_mean type: FLOAT mode: NULLABLE + description: null - name: client_submission_latency_mean type: FLOAT mode: NULLABLE + description: null - name: cpu_cores type: INTEGER mode: NULLABLE + description: null - name: cpu_count type: INTEGER mode: NULLABLE + description: null - name: cpu_family type: INTEGER mode: NULLABLE + description: null - name: cpu_l2_cache_kb type: INTEGER mode: NULLABLE + description: null - name: cpu_l3_cache_kb type: INTEGER mode: NULLABLE + description: null - name: cpu_model type: INTEGER mode: NULLABLE + description: null - name: cpu_speed_mhz type: INTEGER mode: NULLABLE + description: null - name: cpu_stepping type: INTEGER mode: NULLABLE + description: null - name: cpu_vendor type: STRING mode: NULLABLE + description: null - name: crashes_detected_content_sum type: INTEGER mode: NULLABLE + description: null - name: crashes_detected_gmplugin_sum type: INTEGER mode: NULLABLE + description: null - name: crashes_detected_plugin_sum type: INTEGER mode: NULLABLE + description: null - name: crash_submit_attempt_content_sum type: INTEGER mode: NULLABLE + description: null - name: crash_submit_attempt_main_sum type: INTEGER mode: NULLABLE + description: null - name: crash_submit_attempt_plugin_sum type: INTEGER mode: NULLABLE + description: null - name: crash_submit_success_content_sum type: INTEGER mode: NULLABLE + description: null - name: crash_submit_success_main_sum type: INTEGER mode: NULLABLE + description: null - name: crash_submit_success_plugin_sum type: INTEGER mode: NULLABLE + description: null - name: default_search_engine type: STRING mode: NULLABLE + description: null - name: default_search_engine_data_load_path type: STRING mode: NULLABLE + description: null - name: default_search_engine_data_name type: STRING mode: NULLABLE + description: null - name: default_search_engine_data_origin type: STRING mode: NULLABLE + description: null - name: default_search_engine_data_submission_url type: STRING mode: NULLABLE + description: null - name: devtools_toolbox_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: distribution_id type: STRING mode: NULLABLE + description: null - name: e10s_enabled type: BOOLEAN mode: NULLABLE + description: null - name: env_build_arch type: STRING mode: NULLABLE + description: null - name: env_build_id type: STRING mode: NULLABLE + description: null - name: env_build_version type: STRING mode: NULLABLE + description: null - name: environment_settings_intl_accept_languages type: STRING mode: REPEATED + description: null - name: environment_settings_intl_app_locales type: STRING mode: REPEATED + description: null - name: environment_settings_intl_available_locales type: STRING mode: REPEATED + description: null - name: environment_settings_intl_requested_locales type: STRING mode: REPEATED + description: null - name: environment_settings_intl_system_locales type: STRING mode: REPEATED + description: null - name: environment_settings_intl_regional_prefs_locales type: STRING mode: REPEATED + description: null - name: experiments type: RECORD mode: REPEATED @@ -245,264 +318,352 @@ - name: key type: STRING mode: NULLABLE + description: null - name: value type: STRING mode: NULLABLE + description: null + description: null - name: first_paint_mean type: FLOAT mode: NULLABLE + description: null - name: flash_version type: STRING mode: NULLABLE + description: null - name: country type: STRING mode: NULLABLE + description: null - name: city type: STRING mode: NULLABLE + description: null - name: geo_subdivision1 type: STRING mode: NULLABLE + description: null - name: geo_subdivision2 type: STRING mode: NULLABLE + description: null - name: isp_name type: STRING mode: NULLABLE + description: null - name: isp_organization type: STRING mode: NULLABLE + description: null - name: gfx_features_advanced_layers_status type: STRING mode: NULLABLE + description: null - name: gfx_features_d2d_status type: STRING mode: NULLABLE + description: null - name: gfx_features_d3d11_status type: STRING mode: NULLABLE + description: null - name: gfx_features_gpu_process_status type: STRING mode: NULLABLE + description: null - name: histogram_parent_devtools_aboutdebugging_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_animationinspector_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_browserconsole_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_canvasdebugger_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_computedview_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_custom_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_dom_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_eyedropper_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_fontinspector_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_inspector_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_jsbrowserdebugger_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_jsdebugger_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_jsprofiler_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_layoutview_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_memory_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_menu_eyedropper_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_netmonitor_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_options_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_paintflashing_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_picker_eyedropper_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_responsive_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_ruleview_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_scratchpad_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_scratchpad_window_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_shadereditor_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_storage_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_styleeditor_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_webaudioeditor_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_webconsole_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: histogram_parent_devtools_webide_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: install_year type: INTEGER mode: NULLABLE + description: null - name: is_default_browser type: BOOLEAN mode: NULLABLE + description: null - name: is_wow64 type: BOOLEAN mode: NULLABLE + description: null - name: locale type: STRING mode: NULLABLE + description: null - name: memory_mb type: INTEGER mode: NULLABLE + description: null - name: normalized_channel type: STRING mode: NULLABLE + description: null - name: normalized_os_version type: STRING mode: NULLABLE + description: null - name: os type: STRING mode: NULLABLE + description: null - name: os_service_pack_major type: INTEGER mode: NULLABLE + description: null - name: os_service_pack_minor type: INTEGER mode: NULLABLE + description: null - name: os_version type: STRING mode: NULLABLE + description: null - name: pings_aggregated_by_this_row type: INTEGER mode: NULLABLE + description: null - name: places_bookmarks_count_mean type: FLOAT mode: NULLABLE + description: null - name: places_pages_count_mean type: FLOAT mode: NULLABLE + description: null - name: plugin_hangs_sum type: INTEGER mode: NULLABLE + description: null - name: plugins_infobar_allow_sum type: INTEGER mode: NULLABLE + description: null - name: plugins_infobar_block_sum type: INTEGER mode: NULLABLE + description: null - name: plugins_infobar_shown_sum type: INTEGER mode: NULLABLE + description: null - name: plugins_notification_shown_sum type: INTEGER mode: NULLABLE + description: null - name: previous_build_id type: STRING mode: NULLABLE + description: null - name: profile_age_in_days type: INTEGER mode: NULLABLE + description: null - name: profile_creation_date type: STRING mode: NULLABLE + description: null - name: push_api_notify_sum type: INTEGER mode: NULLABLE + description: null - name: sample_id type: INTEGER mode: NULLABLE + description: null - name: sandbox_effective_content_process_level type: INTEGER mode: NULLABLE + description: null - name: scalar_combined_webrtc_nicer_stun_retransmits_sum type: INTEGER mode: NULLABLE + description: null - name: scalar_combined_webrtc_nicer_turn_401s_sum type: INTEGER mode: NULLABLE + description: null - name: scalar_combined_webrtc_nicer_turn_403s_sum type: INTEGER mode: NULLABLE + description: null - name: scalar_combined_webrtc_nicer_turn_438s_sum type: INTEGER mode: NULLABLE + description: null - name: scalar_content_navigator_storage_estimate_count_sum type: INTEGER mode: NULLABLE + description: null - name: scalar_content_navigator_storage_persist_count_sum type: INTEGER mode: NULLABLE + description: null - name: scalar_parent_aushelper_websense_reg_version type: STRING mode: NULLABLE + description: null - name: scalar_parent_browser_engagement_max_concurrent_tab_count_max type: INTEGER mode: NULLABLE + description: null - name: scalar_parent_browser_engagement_max_concurrent_window_count_max type: INTEGER mode: NULLABLE + description: null - name: scalar_parent_browser_engagement_tab_open_event_count_sum type: INTEGER mode: NULLABLE + description: null - name: scalar_parent_browser_engagement_total_uri_count_sum type: INTEGER mode: NULLABLE + description: null - name: scalar_parent_browser_engagement_unfiltered_uri_count_sum type: INTEGER mode: NULLABLE + description: null - name: scalar_parent_browser_engagement_unique_domains_count_max type: INTEGER mode: NULLABLE + description: null - name: scalar_parent_browser_engagement_unique_domains_count_mean type: FLOAT mode: NULLABLE + description: null - name: scalar_parent_browser_engagement_window_open_event_count_sum type: INTEGER mode: NULLABLE + description: null - name: scalar_parent_devtools_accessibility_node_inspected_count_sum type: INTEGER mode: NULLABLE + description: null - name: scalar_parent_devtools_accessibility_opened_count_sum type: INTEGER mode: NULLABLE + description: null - name: scalar_parent_devtools_accessibility_picker_used_count_sum type: INTEGER mode: NULLABLE + description: null - name: scalar_parent_devtools_accessibility_select_accessible_for_node_sum type: RECORD mode: REPEATED @@ -510,159 +671,211 @@ - name: key type: STRING mode: NULLABLE + description: null - name: value type: INTEGER mode: NULLABLE + description: null + description: null - name: scalar_parent_devtools_accessibility_service_enabled_count_sum type: INTEGER mode: NULLABLE + description: null - name: scalar_parent_devtools_copy_full_css_selector_opened_sum type: INTEGER mode: NULLABLE + description: null - name: scalar_parent_devtools_copy_unique_css_selector_opened_sum type: INTEGER mode: NULLABLE + description: null - name: scalar_parent_devtools_toolbar_eyedropper_opened_sum type: INTEGER mode: NULLABLE + description: null - name: scalar_parent_navigator_storage_estimate_count_sum type: INTEGER mode: NULLABLE + description: null - name: scalar_parent_navigator_storage_persist_count_sum type: INTEGER mode: NULLABLE + description: null - name: scalar_parent_storage_sync_api_usage_extensions_using_sum type: INTEGER mode: NULLABLE + description: null - name: search_cohort type: STRING mode: NULLABLE + description: null - name: search_count_abouthome type: INTEGER mode: NULLABLE + description: null - name: search_count_contextmenu type: INTEGER mode: NULLABLE + description: null - name: search_count_newtab type: INTEGER mode: NULLABLE + description: null - name: search_count_searchbar type: INTEGER mode: NULLABLE + description: null - name: search_count_system type: INTEGER mode: NULLABLE + description: null - name: search_count_urlbar type: INTEGER mode: NULLABLE + description: null - name: search_count_all type: INTEGER mode: NULLABLE + description: null - name: search_count_tagged_sap type: INTEGER mode: NULLABLE + description: null - name: search_count_tagged_follow_on type: INTEGER mode: NULLABLE + description: null - name: search_count_organic type: INTEGER mode: NULLABLE + description: null - name: search_count_urlbar_handoff type: INTEGER mode: NULLABLE - name: session_restored_mean type: FLOAT mode: NULLABLE + description: null - name: sessions_started_on_this_day type: INTEGER mode: NULLABLE + description: null - name: shutdown_kill_sum type: INTEGER mode: NULLABLE + description: null - name: subsession_hours_sum type: NUMERIC mode: NULLABLE + description: null - name: ssl_handshake_result_failure_sum type: INTEGER mode: NULLABLE + description: null - name: ssl_handshake_result_success_sum type: INTEGER mode: NULLABLE + description: null - name: sync_configured type: BOOLEAN mode: NULLABLE + description: null - name: sync_count_desktop_mean type: FLOAT mode: NULLABLE + description: null - name: sync_count_mobile_mean type: FLOAT mode: NULLABLE + description: null - name: sync_count_desktop_sum type: INTEGER mode: NULLABLE + description: null - name: sync_count_mobile_sum type: INTEGER mode: NULLABLE + description: null - name: telemetry_enabled type: BOOLEAN mode: NULLABLE + description: null - name: timezone_offset type: INTEGER mode: NULLABLE + description: null - name: update_auto_download type: BOOLEAN mode: NULLABLE + description: null - name: update_channel type: STRING mode: NULLABLE + description: null - name: update_enabled type: BOOLEAN mode: NULLABLE + description: null - name: vendor type: STRING mode: NULLABLE + description: null - name: web_notification_shown_sum type: INTEGER mode: NULLABLE + description: null - name: windows_build_number type: INTEGER mode: NULLABLE + description: null - name: windows_ubr type: INTEGER mode: NULLABLE + description: null - name: fxa_configured type: BOOLEAN mode: NULLABLE + description: null - name: trackers_blocked_sum type: INTEGER mode: NULLABLE + description: null - name: submission_timestamp_min type: TIMESTAMP mode: NULLABLE + description: null - name: ad_clicks_count_all type: INTEGER mode: NULLABLE + description: null - name: search_with_ads_count_all type: INTEGER mode: NULLABLE + description: null - name: scalar_parent_urlbar_impression_autofill_about_sum type: INTEGER mode: NULLABLE + description: null - name: scalar_parent_urlbar_impression_autofill_adaptive_sum type: INTEGER mode: NULLABLE + description: null - name: scalar_parent_urlbar_impression_autofill_origin_sum type: INTEGER mode: NULLABLE + description: null - name: scalar_parent_urlbar_impression_autofill_other_sum type: INTEGER mode: NULLABLE + description: null - name: scalar_parent_urlbar_impression_autofill_preloaded_sum type: INTEGER mode: NULLABLE + description: null - name: scalar_parent_urlbar_impression_autofill_url_sum type: INTEGER mode: NULLABLE + description: null - name: scalar_parent_telemetry_event_counts_sum type: RECORD mode: REPEATED @@ -670,9 +883,12 @@ - name: key type: STRING mode: NULLABLE + description: null - name: value type: INTEGER mode: NULLABLE + description: null + description: null - name: scalar_content_telemetry_event_counts_sum type: RECORD mode: REPEATED @@ -680,9 +896,12 @@ - name: key type: STRING mode: NULLABLE + description: null - name: value type: INTEGER mode: NULLABLE + description: null + description: null - name: scalar_parent_urlbar_searchmode_bookmarkmenu_sum type: RECORD mode: REPEATED @@ -690,9 +909,12 @@ - name: key type: STRING mode: NULLABLE + description: null - name: value type: INTEGER mode: NULLABLE + description: null + description: null - name: scalar_parent_urlbar_searchmode_handoff_sum type: RECORD mode: REPEATED @@ -700,9 +922,12 @@ - name: key type: STRING mode: NULLABLE + description: null - name: value type: INTEGER mode: NULLABLE + description: null + description: null - name: scalar_parent_urlbar_searchmode_keywordoffer_sum type: RECORD mode: REPEATED @@ -710,9 +935,12 @@ - name: key type: STRING mode: NULLABLE + description: null - name: value type: INTEGER mode: NULLABLE + description: null + description: null - name: scalar_parent_urlbar_searchmode_oneoff_sum type: RECORD mode: REPEATED @@ -720,9 +948,12 @@ - name: key type: STRING mode: NULLABLE + description: null - name: value type: INTEGER mode: NULLABLE + description: null + description: null - name: scalar_parent_urlbar_searchmode_other_sum type: RECORD mode: REPEATED @@ -730,9 +961,12 @@ - name: key type: STRING mode: NULLABLE + description: null - name: value type: INTEGER mode: NULLABLE + description: null + description: null - name: scalar_parent_urlbar_searchmode_shortcut_sum type: RECORD mode: REPEATED @@ -740,9 +974,12 @@ - name: key type: STRING mode: NULLABLE + description: null - name: value type: INTEGER mode: NULLABLE + description: null + description: null - name: scalar_parent_urlbar_searchmode_tabmenu_sum type: RECORD mode: REPEATED @@ -750,9 +987,12 @@ - name: key type: STRING mode: NULLABLE + description: null - name: value type: INTEGER mode: NULLABLE + description: null + description: null - name: scalar_parent_urlbar_searchmode_tabtosearch_sum type: RECORD mode: REPEATED @@ -760,9 +1000,12 @@ - name: key type: STRING mode: NULLABLE + description: null - name: value type: INTEGER mode: NULLABLE + description: null + description: null - name: scalar_parent_urlbar_searchmode_tabtosearch_onboard_sum type: RECORD mode: REPEATED @@ -770,9 +1013,12 @@ - name: key type: STRING mode: NULLABLE + description: null - name: value type: INTEGER mode: NULLABLE + description: null + description: null - name: scalar_parent_urlbar_searchmode_topsites_newtab_sum type: RECORD mode: REPEATED @@ -780,9 +1026,12 @@ - name: key type: STRING mode: NULLABLE + description: null - name: value type: INTEGER mode: NULLABLE + description: null + description: null - name: scalar_parent_urlbar_searchmode_topsites_urlbar_sum type: RECORD mode: REPEATED @@ -790,9 +1039,12 @@ - name: key type: STRING mode: NULLABLE + description: null - name: value type: INTEGER mode: NULLABLE + description: null + description: null - name: scalar_parent_urlbar_searchmode_touchbar_sum type: RECORD mode: REPEATED @@ -800,9 +1052,12 @@ - name: key type: STRING mode: NULLABLE + description: null - name: value type: INTEGER mode: NULLABLE + description: null + description: null - name: scalar_parent_urlbar_searchmode_typed_sum type: RECORD mode: REPEATED @@ -810,9 +1065,12 @@ - name: key type: STRING mode: NULLABLE + description: null - name: value type: INTEGER mode: NULLABLE + description: null + description: null - name: scalar_parent_os_environment_is_taskbar_pinned type: BOOLEAN mode: NULLABLE diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry/clients_last_seen/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/clients_last_seen/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry/clients_last_seen/schema.yaml 2024-07-30 19:51:00.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/clients_last_seen/schema.yaml 2024-07-30 20:00:44.000000000 +0000 @@ -77,33 +77,44 @@ - name: submission_date type: DATE mode: NULLABLE + description: Submission Date - name: first_seen_date type: DATE mode: NULLABLE + description: First Seen Date - name: second_seen_date type: DATE mode: NULLABLE + description: Second Seen Date - name: days_seen_bits type: INTEGER mode: NULLABLE + description: Bit pattern to identify when the client_id has sent a main ping in + the previous 28 days. - name: days_visited_1_uri_bits type: INTEGER mode: NULLABLE + description: Days Visited 1 URI Bits - name: days_visited_5_uri_bits type: INTEGER mode: NULLABLE + description: Days Visited 5 URI Bits - name: days_visited_10_uri_bits type: INTEGER mode: NULLABLE + description: Days Visited 10 URI Bits - name: days_had_8_active_ticks_bits type: INTEGER mode: NULLABLE + description: Days Had 8 Active Ticks Bits - name: days_opened_dev_tools_bits type: INTEGER mode: NULLABLE + description: Days Opened Dev Tools Bits - name: days_interacted_bits type: INTEGER mode: NULLABLE + description: Days Interacted Bits - name: days_visited_1_uri_normal_mode_bits type: INTEGER mode: NULLABLE @@ -132,18 +143,23 @@ - name: client_id type: STRING mode: NULLABLE + description: Client ID - name: aborts_content_sum type: INTEGER mode: NULLABLE + description: Aborts Content Sum - name: aborts_gmplugin_sum type: INTEGER mode: NULLABLE + description: null - name: aborts_plugin_sum type: INTEGER mode: NULLABLE + description: Aborts Plugin Sum - name: active_addons_count_mean type: FLOAT mode: NULLABLE + description: Active Addons Count Mean - name: active_addons type: RECORD mode: REPEATED @@ -151,69 +167,92 @@ - name: addon_id type: STRING mode: NULLABLE + description: Add-On Identifier - name: blocklisted type: BOOLEAN mode: NULLABLE + description: Blocklisted - name: name type: STRING mode: NULLABLE + description: Name - name: user_disabled type: BOOLEAN mode: NULLABLE + description: User Disabled - name: app_disabled type: BOOLEAN mode: NULLABLE + description: App Disabled - name: version type: STRING mode: NULLABLE + description: Version - name: scope type: INTEGER mode: NULLABLE + description: Scope - name: type type: STRING mode: NULLABLE + description: Type - name: foreign_install type: BOOLEAN mode: NULLABLE + description: Foreign Install - name: has_binary_components type: BOOLEAN mode: NULLABLE + description: Has Binary Components - name: install_day type: INTEGER mode: NULLABLE + description: Install Day - name: update_day type: INTEGER mode: NULLABLE + description: Update Day - name: signed_state type: INTEGER mode: NULLABLE + description: Signed State - name: is_system type: BOOLEAN mode: NULLABLE + description: Is System - name: is_web_extension type: BOOLEAN mode: NULLABLE + description: Is Web Extension - name: multiprocess_compatible type: BOOLEAN mode: NULLABLE + description: Multiprocess Compatible + description: Active Addons - name: active_hours_sum type: FLOAT mode: NULLABLE + description: Active Hours Sum - name: addon_compatibility_check_enabled type: BOOLEAN mode: NULLABLE + description: Addon Compatibility Check Enabled - name: app_build_id type: STRING mode: NULLABLE + description: App Build ID - name: app_display_version type: STRING mode: NULLABLE + description: App Display Version - name: app_name type: STRING mode: NULLABLE + description: App Name - name: app_version type: STRING mode: NULLABLE + description: App Version - name: attribution type: RECORD mode: NULLABLE @@ -221,147 +260,196 @@ - name: source type: STRING mode: NULLABLE + description: Source - name: medium type: STRING mode: NULLABLE + description: Medium - name: campaign type: STRING mode: NULLABLE + description: Campaign - name: content type: STRING mode: NULLABLE + description: Content - name: experiment type: STRING mode: NULLABLE + description: Experiment - name: variation type: STRING mode: NULLABLE + description: Variation - name: dltoken type: STRING mode: NULLABLE + description: Download Token - name: dlsource type: STRING mode: NULLABLE + description: Download Source - name: ua type: STRING mode: NULLABLE + description: null + description: Attribution - name: blocklist_enabled type: BOOLEAN mode: NULLABLE + description: Blocklist Enabled - name: channel type: STRING mode: NULLABLE + description: Channel - name: client_clock_skew_mean type: FLOAT mode: NULLABLE + description: Client Clock Skew Mean - name: client_submission_latency_mean type: FLOAT mode: NULLABLE + description: Client Submission Latency Mean - name: cpu_cores type: INTEGER mode: NULLABLE + description: CPU Cores - name: cpu_count type: INTEGER mode: NULLABLE + description: CPU Count - name: cpu_family type: INTEGER mode: NULLABLE + description: CPU Family - name: cpu_l2_cache_kb type: INTEGER mode: NULLABLE + description: CPU L2 Cache KB - name: cpu_l3_cache_kb type: INTEGER mode: NULLABLE + description: CPU L3 Cache KB - name: cpu_model type: INTEGER mode: NULLABLE + description: CPU Model - name: cpu_speed_mhz type: INTEGER mode: NULLABLE + description: CPU Speed MHz - name: cpu_stepping type: INTEGER mode: NULLABLE + description: CPU Stepping - name: cpu_vendor type: STRING mode: NULLABLE + description: CPU Vendor - name: crashes_detected_content_sum type: INTEGER mode: NULLABLE + description: Crashes Detected Content Sum - name: crashes_detected_gmplugin_sum type: INTEGER mode: NULLABLE + description: null - name: crashes_detected_plugin_sum type: INTEGER mode: NULLABLE + description: Crashes Detected Plugin Sum - name: crash_submit_attempt_content_sum type: INTEGER mode: NULLABLE + description: Crash Submit Attempt Content Sum - name: crash_submit_attempt_main_sum type: INTEGER mode: NULLABLE + description: Crash Submit Attempt Main Sum - name: crash_submit_attempt_plugin_sum type: INTEGER mode: NULLABLE + description: Crash Submit Attempt Plugin Sum - name: crash_submit_success_content_sum type: INTEGER mode: NULLABLE + description: Crash Submit Success Content Sum - name: crash_submit_success_main_sum type: INTEGER mode: NULLABLE + description: Crash Submit Success Main Sum - name: crash_submit_success_plugin_sum type: INTEGER mode: NULLABLE + description: Crash Submit Success Plugin Sum - name: default_search_engine type: STRING mode: NULLABLE + description: Default Search Engine - name: default_search_engine_data_load_path type: STRING mode: NULLABLE + description: Default Search Engine Data Load Path - name: default_search_engine_data_name type: STRING mode: NULLABLE + description: Default Search Engine Data Name - name: default_search_engine_data_origin type: STRING mode: NULLABLE + description: Default Search Engine Data origin - name: default_search_engine_data_submission_url type: STRING mode: NULLABLE + description: Default Search Engine Data Submission URL - name: devtools_toolbox_opened_count_sum type: INTEGER mode: NULLABLE + description: Dev Tools Toolbox Opened Count Sum - name: distribution_id type: STRING mode: NULLABLE + description: Distribution ID - name: e10s_enabled type: BOOLEAN mode: NULLABLE + description: E10s Enabled - name: env_build_arch type: STRING mode: NULLABLE + description: Environment Build Arch - name: env_build_id type: STRING mode: NULLABLE + description: Environment Build ID - name: env_build_version type: STRING mode: NULLABLE + description: Environment Build Version - name: environment_settings_intl_accept_languages type: STRING mode: REPEATED + description: Environment Settings Intl Accept Languages - name: environment_settings_intl_app_locales type: STRING mode: REPEATED + description: Environment Settings Intl App Locales - name: environment_settings_intl_available_locales type: STRING mode: REPEATED + description: Environment Settings Intl Available Locales - name: environment_settings_intl_requested_locales type: STRING mode: REPEATED + description: Environment Settings Intl Requested Localed - name: environment_settings_intl_system_locales type: STRING mode: REPEATED + description: Environment Settings Intl System Locales - name: environment_settings_intl_regional_prefs_locales type: STRING mode: REPEATED + description: Environment Settings Intl Regional Prefs Locales - name: experiments type: RECORD mode: REPEATED @@ -369,264 +457,351 @@ - name: key type: STRING mode: NULLABLE + description: Key - name: value type: STRING mode: NULLABLE + description: Value + description: Experiments - name: first_paint_mean type: FLOAT mode: NULLABLE + description: First Paint Mean - name: flash_version type: STRING mode: NULLABLE + description: Flash Version - name: country type: STRING mode: NULLABLE + description: Country - name: city type: STRING mode: NULLABLE + description: City - name: geo_subdivision1 type: STRING mode: NULLABLE + description: Geo Subdivision 1 - name: geo_subdivision2 type: STRING mode: NULLABLE + description: Geo Subdivision 2 - name: isp_name type: STRING mode: NULLABLE + description: ISP Name - name: isp_organization type: STRING mode: NULLABLE + description: ISP Organization - name: gfx_features_advanced_layers_status type: STRING mode: NULLABLE + description: GFX Features Advanced Layers Status - name: gfx_features_d2d_status type: STRING mode: NULLABLE + description: GFX Features D2D Status - name: gfx_features_d3d11_status type: STRING mode: NULLABLE + description: GFX Features D3D11 Status - name: gfx_features_gpu_process_status type: STRING mode: NULLABLE + description: GFX Features GPU Process Status - name: histogram_parent_devtools_aboutdebugging_opened_count_sum type: INTEGER mode: NULLABLE + description: Histogram Parent Dev Tools About Debugging Opened Count Sum - name: histogram_parent_devtools_animationinspector_opened_count_sum type: INTEGER mode: NULLABLE + description: Histogram Parent Dev Tools Animation Inspector Opened Count Sum - name: histogram_parent_devtools_browserconsole_opened_count_sum type: INTEGER mode: NULLABLE + description: Histogram Parent Dev Tools Browser Console Opened Count Sum - name: histogram_parent_devtools_canvasdebugger_opened_count_sum type: INTEGER mode: NULLABLE + description: Histogram Parent Dev Tools Canvas Debugger Opened Count Sum - name: histogram_parent_devtools_computedview_opened_count_sum type: INTEGER mode: NULLABLE + description: Histogram Parent Dev Tools Computed View Opened Count Sum - name: histogram_parent_devtools_custom_opened_count_sum type: INTEGER mode: NULLABLE + description: Histogram Parent Dev Tools Custom Opened Count Sum - name: histogram_parent_devtools_dom_opened_count_sum type: INTEGER mode: NULLABLE + description: Histogram Parent Dev Tools DOM Opened Count Sum - name: histogram_parent_devtools_eyedropper_opened_count_sum type: INTEGER mode: NULLABLE + description: Histogram Parent Dev Tools Eyedropper Opened Count Sum - name: histogram_parent_devtools_fontinspector_opened_count_sum type: INTEGER mode: NULLABLE + description: Histogram Parent Dev Tools Font Inspector Opened Count Sum - name: histogram_parent_devtools_inspector_opened_count_sum type: INTEGER mode: NULLABLE + description: Histogram Parent Dev Tools Inspector Opened Count Sum - name: histogram_parent_devtools_jsbrowserdebugger_opened_count_sum type: INTEGER mode: NULLABLE + description: Histogram Parent Dev Tools JS Browser Debugger Opened Count Sum - name: histogram_parent_de ```

⚠️ Only part of the diff is displayed.

Link to full diff