Separate DAU metrics into `search_dau_aggregates` table

To accommodate shredder mitigation changes, it has been decided to separate the DAU calculations into a separate aggregates table (which can be used downstream i.e., the search_revenue_levers_daily table)

This PR includes changes relates to mobile Upcoming PRs

add desktop
modify search_revenue_levers_daily to pull from search_dau_aggregates view

Checklist for reviewer:

[ ] Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title).
[ ] If the PR comes from a fork, trigger integration CI tests by running the Push to upstream workflow and provide the <username>:<branch> of the fork as parameter. The parameter will also show up in the logs of the manual-trigger-required-for-fork CI task together with more detailed instructions.
[ ] If adding a new field to a query, ensure that the schema and dependent downstream schemas have been updated.
[ ] When adding a new derived dataset, ensure that data is not available already (fully or partially) and recommend extending an existing dataset in favor of creating new ones. Data can be available in the bigquery-etl repository, looker-hub or in looker-spoke-default.

For modifications to schemas in restricted namespaces (see CODEOWNERS):

[ ] Follow the change control procedure

┆Issue is synchronized with this Jira Task

Integration report for "Merge branch 'main' into mobile_search_aggregates_dau"

`sql.diff`

Click to expand!

```diff diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_search_dashboard.py /tmp/workspace/generated-sql/dags/bqetl_search_dashboard.py --- /tmp/workspace/main-generated-sql/dags/bqetl_search_dashboard.py 2024-08-01 15:07:51.000000000 +0000 +++ /tmp/workspace/generated-sql/dags/bqetl_search_dashboard.py 2024-08-01 15:08:30.000000000 +0000 @@ -323,6 +323,17 @@ depends_on_past=False, ) + search_derived__search_dau_aggregates__v1 = bigquery_etl_query( + task_id="search_derived__search_dau_aggregates__v1", + destination_table="search_dau_aggregates_v1", + dataset_id="search_derived", + project_id="moz-fx-data-shared-prod", + owner="mozilla/revenue_forecasting_data_reviewers", + email=["akomar@mozilla.com", "telemetry-alerts@mozilla.com"], + date_partition_parameter="submission_date", + depends_on_past=False, + ) + search_derived__search_revenue_levers_daily__v1 = bigquery_etl_query( task_id="search_derived__search_revenue_levers_daily__v1", destination_table="search_revenue_levers_daily_v1", @@ -346,6 +357,70 @@ wait_for_search_derived__mobile_search_clients_daily__v1 ) + search_derived__search_dau_aggregates__v1.set_upstream( + wait_for_checks__fail_org_mozilla_fenix_derived__baseline_clients_last_seen__v1 + ) + + search_derived__search_dau_aggregates__v1.set_upstream( + wait_for_checks__fail_org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1 + ) + + search_derived__search_dau_aggregates__v1.set_upstream( + wait_for_checks__fail_org_mozilla_fennec_aurora_derived__baseline_clients_last_seen__v1 + ) + + search_derived__search_dau_aggregates__v1.set_upstream( + wait_for_checks__fail_org_mozilla_firefox_beta_derived__baseline_clients_last_seen__v1 + ) + + search_derived__search_dau_aggregates__v1.set_upstream( + wait_for_checks__fail_org_mozilla_firefox_derived__baseline_clients_last_seen__v1 + ) + + search_derived__search_dau_aggregates__v1.set_upstream( + wait_for_checks__fail_org_mozilla_focus_beta_derived__baseline_clients_last_seen__v1 + ) + + search_derived__search_dau_aggregates__v1.set_upstream( + wait_for_checks__fail_org_mozilla_focus_derived__baseline_clients_last_seen__v1 + ) + + search_derived__search_dau_aggregates__v1.set_upstream( + wait_for_checks__fail_org_mozilla_focus_nightly_derived__baseline_clients_last_seen__v1 + ) + + search_derived__search_dau_aggregates__v1.set_upstream( + wait_for_checks__fail_org_mozilla_ios_fennec_derived__baseline_clients_last_seen__v1 + ) + + search_derived__search_dau_aggregates__v1.set_upstream( + wait_for_checks__fail_org_mozilla_ios_firefox_derived__baseline_clients_last_seen__v1 + ) + + search_derived__search_dau_aggregates__v1.set_upstream( + wait_for_checks__fail_org_mozilla_ios_firefoxbeta_derived__baseline_clients_last_seen__v1 + ) + + search_derived__search_dau_aggregates__v1.set_upstream( + wait_for_checks__fail_org_mozilla_ios_focus_derived__baseline_clients_last_seen__v1 + ) + + search_derived__search_dau_aggregates__v1.set_upstream( + wait_for_checks__fail_org_mozilla_ios_klar_derived__baseline_clients_last_seen__v1 + ) + + search_derived__search_dau_aggregates__v1.set_upstream( + wait_for_checks__fail_org_mozilla_klar_derived__baseline_clients_last_seen__v1 + ) + + search_derived__search_dau_aggregates__v1.set_upstream( + wait_for_copy_deduplicate_all + ) + + search_derived__search_dau_aggregates__v1.set_upstream( + wait_for_search_derived__mobile_search_clients_daily__v1 + ) + search_derived__search_revenue_levers_daily__v1.set_upstream( wait_for_checks__fail_org_mozilla_fenix_derived__baseline_clients_last_seen__v1 ) Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search: search_dau_aggregates Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived: search_dau_aggregates_v1 diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search/search_dau_aggregates/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search/search_dau_aggregates/metadata.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search/search_dau_aggregates/metadata.yaml 1970-01-01 00:00:00.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search/search_dau_aggregates/metadata.yaml 2024-08-01 15:03:21.000000000 +0000 @@ -0,0 +1,13 @@ +friendly_name: Search Dau Aggregates +description: |- + Please provide a description for the query +owners: [] +labels: {} +bigquery: null +workgroup_access: +- role: roles/bigquery.dataViewer + members: + - workgroup:mozilla-confidential +references: + view.sql: + - moz-fx-data-shared-prod.search_derived.search_dau_aggregates_v1 diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search/search_dau_aggregates/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search/search_dau_aggregates/view.sql --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search/search_dau_aggregates/view.sql 1970-01-01 00:00:00.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search/search_dau_aggregates/view.sql 2024-08-01 15:01:35.000000000 +0000 @@ -0,0 +1,7 @@ +CREATE OR REPLACE VIEW + `moz-fx-data-shared-prod.search.search_dau_aggregates` +AS +SELECT + * +FROM + `moz-fx-data-shared-prod.search_derived.search_dau_aggregates_v1` diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_dau_aggregates_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_dau_aggregates_v1/metadata.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_dau_aggregates_v1/metadata.yaml 1970-01-01 00:00:00.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_dau_aggregates_v1/metadata.yaml 2024-08-01 15:03:24.000000000 +0000 @@ -0,0 +1,28 @@ +friendly_name: Search Revenue DAU aggregates +description: |- + Daily metrics including eligible DOU, + DAU_engaged_w_sap, DAU_engaged_w_default_engine + The table is labeled as "change_controlled", which implies + that changes require the approval of at least one owner. +owners: +- mozilla/revenue_forecasting_data_reviewers +labels: + incremental: true + schedule: daily + change_controlled: true + dag: bqetl_search_dashboard +scheduling: + dag_name: bqetl_search_dashboard +bigquery: + time_partitioning: + type: day + field: submission_date + require_partition_filter: true + expiration_days: null + range_partitioning: null + clustering: null +workgroup_access: +- role: roles/bigquery.dataViewer + members: + - workgroup:mozilla-confidential +references: {} diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_dau_aggregates_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_dau_aggregates_v1/query.sql --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_dau_aggregates_v1/query.sql 1970-01-01 00:00:00.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_dau_aggregates_v1/query.sql 2024-08-01 15:01:35.000000000 +0000 @@ -0,0 +1,293 @@ +-- Mobile DAU data -- merging baseline clients to AUA clients +## baseline ping -- mobile default search engine by client id +WITH mobile_baseline_engine AS ( + SELECT DISTINCT + DATE(submission_timestamp) AS submission_date, + client_info.client_id, + metrics.string.search_default_engine_code AS default_search_engine, + FROM + `moz-fx-data-shared-prod.fenix.baseline` + WHERE + DATE(submission_timestamp) = @submission_date + UNION ALL + SELECT DISTINCT + DATE(submission_timestamp) AS submission_date, + client_info.client_id, + metrics.string.search_default_engine AS default_search_engine + FROM + `moz-fx-data-shared-prod.firefox_ios.baseline` + WHERE + DATE(submission_timestamp) = @submission_date + UNION ALL + SELECT DISTINCT + DATE(submission_timestamp) AS submission_date, + client_info.client_id, + metrics.string.browser_default_search_engine AS default_search_engine + FROM + `moz-fx-data-shared-prod.focus_android.baseline` + WHERE + DATE(submission_timestamp) = @submission_date + UNION ALL + SELECT DISTINCT + DATE(submission_timestamp) AS submission_date, + client_info.client_id, + metrics.string.search_default_engine AS default_search_engine + FROM + `moz-fx-data-shared-prod.focus_ios.baseline` + WHERE + DATE(submission_timestamp) = @submission_date +), +## baseline ping search counts -- mobile search counts by client id +mobile_baseline_search AS ( + SELECT + DATE(submission_timestamp) AS submission_date, + client_info.client_id, + metrics.string.search_default_engine_code AS default_search_engine, + `moz-fx-data-shared-prod.udf.normalize_search_engine`(key_value.key) AS normalized_engine, + key_value.value AS search_count + FROM + `moz-fx-data-shared-prod.fenix.baseline`, + UNNEST(metrics.labeled_counter.metrics_search_count) AS key_value + WHERE + DATE(submission_timestamp) = @submission_date + AND key_value.value <= 10000 + UNION ALL + SELECT + DATE(submission_timestamp) AS submission_date, + client_info.client_id, + metrics.string.search_default_engine AS default_search_engine, + `moz-fx-data-shared-prod.udf.normalize_search_engine`(key_value.key) AS normalized_engine, + key_value.value AS search_count + FROM + `moz-fx-data-shared-prod.firefox_ios.baseline`, + UNNEST(metrics.labeled_counter.search_counts) AS key_value + WHERE + DATE(submission_timestamp) = @submission_date + AND key_value.value <= 10000 + UNION ALL + SELECT + DATE(submission_timestamp) AS submission_date, + client_info.client_id, + metrics.string.browser_default_search_engine AS default_search_engine, + `moz-fx-data-shared-prod.udf.normalize_search_engine`(key_value.key) AS normalized_engine, + key_value.value AS search_count + FROM + `moz-fx-data-shared-prod.focus_android.baseline`, + UNNEST(metrics.labeled_counter.browser_search_search_count) AS key_value + WHERE + DATE(submission_timestamp) = @submission_date + AND key_value.value <= 10000 + UNION ALL + SELECT + DATE(submission_timestamp) AS submission_date, + client_info.client_id, + metrics.string.search_default_engine AS default_search_engine, + `moz-fx-data-shared-prod.udf.normalize_search_engine`(key_value.key) AS normalized_engine, + key_value.value AS search_count + FROM + `moz-fx-data-shared-prod.focus_ios.baseline`, + UNNEST(metrics.labeled_counter.browser_search_search_count) AS key_value + WHERE + DATE(submission_timestamp) = @submission_date + AND key_value.value <= 10000 +), +## baseline-powered clients who qualify for KPI (activity filters applied) +mobile_dau_data AS ( + SELECT DISTINCT + submission_date, + "mobile" AS device, + country, + client_id + FROM + `mozdata.telemetry.mobile_active_users` + WHERE + submission_date = @submission_date + AND is_dau + # not including Fenix MozillaOnline, BrowserStack, Klar + AND app_name IN ("Focus iOS", "Firefox iOS", "Fenix", "Focus Android") +), +final_mobile_dau_counts AS ( + SELECT + submission_date, + country, + COUNT(DISTINCT client_id) AS eligible_dau, + COUNT( + DISTINCT IF( + ( + (submission_date < "2023-12-01" AND country NOT IN ('RU', 'UA', 'TR', 'BY', 'KZ', 'CN')) + OR (submission_date >= "2023-12-01" AND country NOT IN ('RU', 'UA', 'BY', 'CN')) + ), + client_id, + NULL + ) + ) AS google_eligible_dau, + COUNT( + DISTINCT IF( + default_search_engine LIKE '%google%' + AND ( + (submission_date < "2023-12-01" AND country NOT IN ('RU', 'UA', 'TR', 'BY', 'KZ', 'CN')) + OR (submission_date >= "2023-12-01" AND country NOT IN ('RU', 'UA', 'BY', 'CN')) + ), + client_id, + NULL + ) + ) AS google_dau_w_engine_as_default, + COUNT( + DISTINCT IF( + search_count > 0 + AND normalized_engine = 'Google' + AND ( + (submission_date < "2023-12-01" AND country NOT IN ('RU', 'UA', 'TR', 'BY', 'KZ', 'CN')) + OR (submission_date >= "2023-12-01" AND country NOT IN ('RU', 'UA', 'BY', 'CN')) + ), + client_id, + NULL + ) + ) AS google_dau_engaged_w_sap, + COUNT( + DISTINCT IF(default_search_engine LIKE '%bing%', client_id, NULL) + ) AS bing_dau_w_engine_as_default, + COUNT( + DISTINCT IF(search_count > 0 AND normalized_engine = 'Bing', client_id, NULL) + ) AS bing_dau_engaged_w_sap, + COUNT( + DISTINCT IF( + default_search_engine LIKE('%ddg%') + OR default_search_engine LIKE('%duckduckgo%'), + client_id, + NULL + ) + ) AS ddg_dau_w_engine_as_default, + COUNT( + DISTINCT IF(normalized_engine = "DuckDuckGo" AND search_count > 0, client_id, NULL) + ) AS ddg_dau_engaged_w_sap + FROM + mobile_dau_data + LEFT JOIN + mobile_baseline_engine + USING (submission_date, client_id) + LEFT JOIN + mobile_baseline_search + USING (submission_date, client_id, default_search_engine) + GROUP BY + submission_date, + country +), +-- Google Mobile (search only - as mobile search metrics is based on metrics +-- ping, while DAU should be based on main ping on Mobile, see also +-- https://mozilla-hub.atlassian.net/browse/RS-575) +mobile_data_google AS ( + SELECT + submission_date, + country, + google_eligible_dau, + google_dau_w_engine_as_default, + google_dau_engaged_w_sap + FROM + `moz-fx-data-shared-prod.search.mobile_search_clients_engines_sources_daily` + INNER JOIN + final_mobile_dau_counts + USING (submission_date, country) + WHERE + submission_date = @submission_date + AND ( + (submission_date < "2023-12-01" AND country NOT IN ('RU', 'UA', 'TR', 'BY', 'KZ', 'CN')) + OR (submission_date >= "2023-12-01" AND country NOT IN ('RU', 'UA', 'BY', 'CN')) + ) + AND ( + app_name IN ('Fenix', 'Firefox Preview', 'Focus', 'Focus Android Glean', 'Focus iOS Glean') + OR (app_name = 'Fennec' AND os = 'iOS') + ) + GROUP BY + submission_date, + country, + google_eligible_dau, + google_dau_w_engine_as_default, + google_dau_engaged_w_sap + ORDER BY + submission_date, + country +), +-- Bing & DDG Mobile (search only - as mobile search metrics is based on +-- metrics ping, while DAU should be based on main ping on Mobile, see also +-- https://mozilla-hub.atlassian.net/browse/RS-575) +mobile_data_bing_ddg AS ( + SELECT + submission_date, + country, + eligible_dau, + bing_dau_w_engine_as_default, + bing_dau_engaged_w_sap, + ddg_dau_w_engine_as_default, + ddg_dau_engaged_w_sap, + FROM + `moz-fx-data-shared-prod.search.mobile_search_clients_engines_sources_daily` + INNER JOIN + final_mobile_dau_counts + USING (submission_date, country) + WHERE + submission_date = @submission_date + AND ( + app_name IN ('Fenix', 'Firefox Preview', 'Focus', 'Focus Android Glean', 'Focus iOS Glean') + OR (app_name = 'Fennec' AND os = 'iOS') + ) + GROUP BY + submission_date, + country, + eligible_dau, + bing_dau_w_engine_as_default, + bing_dau_engaged_w_sap, + ddg_dau_w_engine_as_default, + ddg_dau_engaged_w_sap + ORDER BY + submission_date, + country +) +SELECT + submission_date, + 'Google' AS partner, + 'mobile' AS device, + 'n/a' AS channel, + country, + google_eligible_dau AS dau, + google_dau_engaged_w_sap AS dau_engaged_w_sap, + # custom engine bug merged in v121 + # null engine bug merged in v126 + # remove default engine data prior to June 2024 + IF( + submission_date >= "2024-06-01", + google_dau_w_engine_as_default, + NULL + ) AS dau_w_engine_as_default +FROM + mobile_data_google +UNION ALL +SELECT + submission_date, + 'Bing' AS partner, + 'mobile' AS device, + NULL AS channel, + country, + eligible_dau, + bing_dau_engaged_w_sap, + # custom engine bug merged in v121 + # null engine bug merged in v126 + # remove default engine data prior to June 2024 + IF(submission_date >= "2024-06-01", bing_dau_w_engine_as_default, NULL) AS dau_w_engine_as_default +FROM + mobile_data_bing_ddg +UNION ALL +SELECT + submission_date, + 'DuckDuckGo' AS partner, + 'mobile' AS device, + NULL AS channel, + country, + eligible_dau, + ddg_dau_engaged_w_sap, + # custom engine bug merged in v121 + # null engine bug merged in v126 + # remove default engine data prior to June 2024 + IF(submission_date >= "2024-06-01", ddg_dau_w_engine_as_default, NULL) AS dau_w_engine_as_default +FROM + mobile_data_bing_ddg diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_dau_aggregates_v1/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_dau_aggregates_v1/schema.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_dau_aggregates_v1/schema.yaml 1970-01-01 00:00:00.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_dau_aggregates_v1/schema.yaml 2024-08-01 15:01:35.000000000 +0000 @@ -0,0 +1,25 @@ +fields: + - mode: NULLABLE + name: submission_date + type: DATE + - mode: NULLABLE + name: partner + type: STRING + - mode: NULLABLE + name: device + type: STRING + - mode: NULLABLE + name: channel + type: STRING + - mode: NULLABLE + name: country + type: STRING + - mode: NULLABLE + name: dau + type: INTEGER + - mode: NULLABLE + name: dau_engaged_w_sap + type: INTEGER + - mode: NULLABLE + name: dau_w_engine_as_default + type: INTEGER ```

Link to full diff

Integration report for "Combine desktop + mobile DAU queries"

`sql.diff`

Click to expand!