search

mozilla / bigquery-etl

Bigquery ETL
https://mozilla.github.io/bigquery-etl
Mozilla Public License 2.0
251 stars 99 forks source link

Views deploy failure: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) #2300

Closed whd closed 3 years ago

whd commented 3 years ago 
+ cd /app
+ cp -R sql /tmp/sql
+ export MSG_PROBE_CACHE_DIR=/tmp/msg_probe_cache
+ mkdir -p /tmp/msg_probe_cache
+ ./script/generate_all_views --target-project moz-fx-data-shared-prod --sql-dir /tmp/sql
+ python -m bigquery_etl.view.generate_stable_views --target-project moz-fx-data-shared-prod --sql-dir /tmp/sql --parallelism=20
telemetry_derived/foo/query.sql                             OK
+ ./script/bqetl glean_usage generate --project-id moz-fx-data-shared-prod --output-dir /tmp/sql
INFO:root:For lockwise-ios, found Glean dependencies: ['glean-core', 'glean-android']
INFO:root:For firefox-android-release, found Glean dependencies: ['glean-core', 'nimbus', 'logins-store', 'engine-gecko-beta', 'lib-crash', 'android-places', 'glean-android', 'support-migration', 'sync']
INFO:root:For firefox-for-fire-tv, found Glean dependencies: ['glean-core', 'glean-android']
INFO:root:For firefox-ios-release, found Glean dependencies: ['glean-core', 'nimbus', 'glean-android']
INFO:root:For reference-browser, found Glean dependencies: ['glean-core', 'lib-crash', 'glean-android']
INFO:root:For lockwise-android, found Glean dependencies: ['glean-core', 'glean-android', 'sync']
INFO:root:For mozregression, found Glean dependencies: ['glean-core', 'glean-android']
INFO:root:For firefox-reality, found Glean dependencies: ['glean-core', 'glean-android', 'sync']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_ios/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_ios/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_ios/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_ios/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_ios/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_ios/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_ios/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_ios/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_ios/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_ios/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_ios/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_ios/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_ios/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_ios/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_ios/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_ios/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_fire_tv/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/usage/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/usage/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_fire_tv/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_fire_tv/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_fire_tv/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_fire_tv/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_fire_tv/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_fire_tv/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_fire_tv/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/session_end/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/session_end/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/addresses_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/addresses_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/bookmarks_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/bookmarks_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/creditcards_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/creditcards_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/history_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/history_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/logins_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/logins_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/tabs_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/tabs_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/addresses_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/addresses_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/bookmarks_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/bookmarks_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/creditcards_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/creditcards_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/history_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/history_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/logins_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/logins_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/tabs_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/tabs_sync/metadata.yaml
INFO:root:For firefox-for-echo-show, found Glean dependencies: ['glean-core', 'glean-android']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_echo_show/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_echo_show/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_echo_show/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_echo_show/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_echo_show/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_echo_show/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_echo_show/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_echo_show/metrics/metadata.yaml
INFO:root:For mach, found Glean dependencies: ['glean-core', 'glean-android']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/usage/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/usage/metadata.yaml
INFO:root:For firefox-focus-ios, found Glean dependencies: ['glean-core']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_ios/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_ios/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_ios/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_ios/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_ios/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_ios/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_ios/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_ios/metrics/metadata.yaml
INFO:root:For firefox-reality-pc, found Glean dependencies: ['glean-core']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/launch/view.sql
INFO:root:For firefox-klar-ios, found Glean dependencies: ['glean-core']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_ios/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_ios/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/launch/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_ios/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_ios/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_ios/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_ios/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_ios/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_ios/metrics/metadata.yaml
INFO:root:For firefox-focus-android, found Glean dependencies: ['glean-core', 'lib-crash', 'glean-android']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/activation/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/activation/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/activation/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/activation/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/first_session/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/first_session/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/installation/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/installation/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/startup_timeline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/startup_timeline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/migration/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/migration/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/addresses_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/addresses_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/bookmarks_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/bookmarks_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/creditcards_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/creditcards_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/history_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/history_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/logins_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/logins_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/tabs_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/tabs_sync/metadata.yaml
INFO:root:For bergamot, found Glean dependencies: ['glean-js']
INFO:root:For firefox-klar-android, found Glean dependencies: ['glean-core', 'lib-crash', 'glean-android']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/activation/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/activation/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/metrics/metadata.yaml
INFO:root:For mozilla-vpn, found Glean dependencies: ['glean-js']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/bergamot/custom/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/bergamot/custom/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/bergamot/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/bergamot/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozilla_vpn/main/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozilla_vpn/main/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozilla_vpn/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozilla_vpn/deletion_request/metadata.yaml
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/app/bigquery_etl/cli/__init__.py", line 56, in cli
    group(prog_name=prog_name)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/app/bigquery_etl/cli/glean_usage.py", line 112, in generate
    pool.map(
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/app/bigquery_etl/glean_usage/glean_app_ping_views.py", line 68, in generate_per_app
    for ping_name in p.get_pings():
  File "/usr/local/lib/python3.8/site-packages/mozilla_schema_generator/glean_ping.py", line 146, in get_pings
    return self._get_ping_data().keys()
  File "/usr/local/lib/python3.8/site-packages/mozilla_schema_generator/glean_ping.py", line 141, in _get_ping_data
    dependency_pings = self._get_json(self.ping_url_template.format(dependency))
  File "/usr/local/lib/python3.8/site-packages/mozilla_schema_generator/generic_ping.py", line 220, in _get_json
    return json.loads(GenericPing._get_json_str(url))
  File "/usr/local/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
telemetry_derived/foo/query.sql                             OK
jklukas commented 3 years ago

cc @scholtzan and @wlach

This looks like it's occurring in glean_app_ping_views and may be related to https://github.com/mozilla/bigquery-etl/pull/2291

wlach commented 3 years ago

I'll take a look.

wlach commented 3 years ago

It's really hard to know what's going on here without better logs. I have a patch upstream in MSG which should help here.

wlach commented 3 years ago

@whd Might be worth trying another views deploy (can wait until next week if you're busy)

whd commented 3 years ago

I re-ran views deploys today and they succeeded :shrug:

whd commented 3 years ago

So maybe this is a transient issue, as it happened again on the most recent deploy to shared-prod.

[Pipeline] sh
+ cd /app
+ cp -R sql /tmp/sql
+ export MSG_PROBE_CACHE_DIR=/tmp/msg_probe_cache
+ mkdir -p /tmp/msg_probe_cache
+ ./script/generate_all_views --target-project moz-fx-data-shared-prod --sql-dir /tmp/sql
+ python -m bigquery_etl.view.generate_stable_views --target-project moz-fx-data-shared-prod --sql-dir /tmp/sql --parallelism=20
telemetry_derived/foo/query.sql                             OK
+ ./script/bqetl glean_usage generate --project-id moz-fx-data-shared-prod --output-dir /tmp/sql
INFO:root:For reference-browser, found Glean dependencies: ['glean-core', 'lib-crash', 'glean-android']
INFO:root:For firefox-for-fire-tv, found Glean dependencies: ['glean-core', 'glean-android']
INFO:root:For firefox-ios-release, found Glean dependencies: ['glean-core', 'nimbus', 'glean-android']
INFO:root:For lockwise-ios, found Glean dependencies: ['glean-core', 'glean-android']
INFO:root:For lockwise-android, found Glean dependencies: ['glean-core', 'glean-android', 'sync']
INFO:root:For firefox-android-release, found Glean dependencies: ['glean-core', 'nimbus', 'logins-store', 'engine-gecko-beta', 'lib-crash', 'android-places', 'glean-android', 'support-migration', 'sync']
INFO:root:For firefox-reality, found Glean dependencies: ['glean-core', 'glean-android', 'sync']
INFO:root:For mozregression, found Glean dependencies: ['glean-core', 'glean-android']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_fire_tv/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_fire_tv/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_fire_tv/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_fire_tv/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_fire_tv/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_fire_tv/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_fire_tv/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_fire_tv/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_ios/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_ios/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_ios/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_ios/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_ios/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/usage/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/usage/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_ios/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_ios/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_ios/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/reference_browser/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/reference_browser/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/reference_browser/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/reference_browser/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/reference_browser/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/reference_browser/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/reference_browser/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/reference_browser/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_ios/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_ios/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_ios/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_ios/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_ios/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_ios/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_ios/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_ios/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/addresses_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/addresses_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/bookmarks_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/bookmarks_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/creditcards_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/creditcards_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/history_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/history_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/logins_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/logins_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/tabs_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/tabs_sync/metadata.yaml
INFO:root:For firefox-for-echo-show, found Glean dependencies: ['glean-core', 'glean-android']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_echo_show/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_echo_show/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_echo_show/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_echo_show/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_echo_show/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_echo_show/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_echo_show/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_echo_show/metrics/metadata.yaml
INFO:root:For firefox-focus-ios, found Glean dependencies: ['glean-core']
INFO:root:For firefox-reality-pc, found Glean dependencies: ['glean-core']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/launch/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/launch/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_ios/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/metrics/metadata.yaml
INFO:root:For firefox-klar-ios, found Glean dependencies: ['glean-core']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_ios/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_ios/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_ios/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_ios/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_ios/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_ios/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_ios/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_ios/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_ios/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_ios/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_ios/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_ios/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_ios/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_ios/metrics/metadata.yaml
INFO:root:For mach, found Glean dependencies: ['glean-core', 'glean-android']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/usage/view.sql
INFO:root:For firefox-focus-android, found Glean dependencies: ['glean-core', 'lib-crash', 'glean-android']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/activation/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/activation/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/usage/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_ios/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/metrics/metadata.yaml
INFO:root:For bergamot, found Glean dependencies: ['glean-js']
INFO:root:For firefox-klar-android, found Glean dependencies: ['glean-core', 'lib-crash', 'glean-android']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/activation/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/activation/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/bergamot/custom/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/bergamot/custom/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/bergamot/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/bergamot/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/activation/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/activation/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/first_session/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/first_session/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/installation/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/installation/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/startup_timeline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/startup_timeline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/migration/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/migration/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/addresses_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/addresses_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/bookmarks_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/bookmarks_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/creditcards_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/creditcards_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/history_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/history_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/logins_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/logins_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/tabs_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/tabs_sync/metadata.yaml
INFO:root:For mozilla-vpn, found Glean dependencies: ['glean-js']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozilla_vpn/main/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozilla_vpn/main/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozilla_vpn/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozilla_vpn/deletion_request/metadata.yaml
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/app/bigquery_etl/cli/__init__.py", line 56, in cli
    group(prog_name=prog_name)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/app/bigquery_etl/cli/glean_usage.py", line 112, in generate
    pool.map(
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/app/bigquery_etl/glean_usage/glean_app_ping_views.py", line 68, in generate_per_app
    for ping_name in p.get_pings():
  File "/usr/local/lib/python3.8/site-packages/mozilla_schema_generator/glean_ping.py", line 146, in get_pings
    return self._get_ping_data().keys()
  File "/usr/local/lib/python3.8/site-packages/mozilla_schema_generator/glean_ping.py", line 141, in _get_ping_data
    dependency_pings = self._get_json(self.ping_url_template.format(dependency))
  File "/usr/local/lib/python3.8/site-packages/mozilla_schema_generator/generic_ping.py", line 219, in _get_json
    return json.loads(GenericPing._get_json_str(url))
  File "/usr/local/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
telemetry_derived/foo/query.sql
wlach commented 3 years ago

https://github.com/mozilla/mozilla-schema-generator/pull/204 never got merged so we're still not getting good error logs. I'll make sure that happens and do up a new release of MSG so we can track this down.

whd commented 3 years ago

Another thing to note is that the last two failures only affected shared-prod views deploys and not mozdata deploys for some reason. It looks like the next deploy will have the new logging.

whd commented 3 years ago

Aaand the latest view deploy succeeded again in both shared-prod and mozdata. I'll update here if the error happens again.

whd commented 3 years ago

Ok, this happened in mozdata but not shared-prod, and I see a (probably transient) failure to read from probeinfo in the new logging:

+ cd /app
+ cp -R sql /tmp/sql
+ export MSG_PROBE_CACHE_DIR=/tmp/msg_probe_cache
+ mkdir -p /tmp/msg_probe_cache
+ ./script/generate_all_views --target-project moz-fx-data-shared-prod --sql-dir /tmp/sql
+ python -m bigquery_etl.view.generate_stable_views --target-project moz-fx-data-shared-prod --sql-dir /tmp/sql --parallelism=20
telemetry_derived/foo/query.sql                             OK
+ ./script/bqetl glean_usage generate --project-id moz-fx-data-shared-prod --output-dir /tmp/sql
INFO:root:For firefox-reality, found Glean dependencies: ['glean-core', 'glean-android', 'sync']
INFO:root:For lockwise-ios, found Glean dependencies: ['glean-core', 'glean-android']
INFO:root:For firefox-android-release, found Glean dependencies: ['glean-core', 'nimbus', 'logins-store', 'engine-gecko-beta', 'lib-crash', 'android-places', 'glean-android', 'support-migration', 'sync']
INFO:root:For firefox-ios-release, found Glean dependencies: ['glean-core', 'nimbus', 'glean-android']
INFO:root:For mozregression, found Glean dependencies: ['glean-core', 'glean-android']
INFO:root:For lockwise-android, found Glean dependencies: ['glean-core', 'glean-android', 'sync']
INFO:root:For firefox-for-fire-tv, found Glean dependencies: ['glean-core', 'glean-android']
INFO:root:For reference-browser, found Glean dependencies: ['glean-core', 'lib-crash', 'glean-android']
ERROR:root:Unable to process JSON for url: https://probeinfo.telemetry.mozilla.org/glean/glean-android/pings?t=2021-09-01T15:08:50.765620
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_ios/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_ios/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_ios/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_ios/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_ios/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_ios/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_ios/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_ios/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/session_end/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/session_end/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_ios/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/addresses_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/addresses_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/bookmarks_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/bookmarks_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/creditcards_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/creditcards_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_ios/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_ios/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_ios/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_ios/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/reference_browser/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/reference_browser/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/reference_browser/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/reference_browser/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/reference_browser/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/reference_browser/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/reference_browser/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/reference_browser/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_ios/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_ios/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_ios/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/history_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/history_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/logins_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/logins_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/tabs_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality/tabs_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/usage/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/usage/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozregression/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/addresses_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/addresses_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/bookmarks_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/bookmarks_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/creditcards_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/creditcards_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/history_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/history_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/logins_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/logins_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/tabs_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/lockwise_android/tabs_sync/metadata.yaml
INFO:root:For firefox-for-echo-show, found Glean dependencies: ['glean-core', 'glean-android']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_echo_show/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_echo_show/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_echo_show/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_echo_show/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_echo_show/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_echo_show/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_echo_show/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_echo_show/metrics/metadata.yaml
INFO:root:For firefox-reality-pc, found Glean dependencies: ['glean-core']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/launch/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/launch/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/firefox_reality_pc/metrics/metadata.yaml
INFO:root:For firefox-focus-ios, found Glean dependencies: ['glean-core']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_ios/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_ios/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_ios/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_ios/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_ios/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_ios/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_ios/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_ios/metrics/metadata.yaml
INFO:root:For firefox-klar-ios, found Glean dependencies: ['glean-core']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_ios/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_ios/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_ios/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_ios/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_ios/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_ios/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_ios/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_ios/metrics/metadata.yaml
INFO:root:For mach, found Glean dependencies: ['glean-core', 'glean-android']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/usage/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/usage/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mach/metrics/metadata.yaml
INFO:root:For firefox-focus-android, found Glean dependencies: ['glean-core', 'lib-crash', 'glean-android']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/activation/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/activation/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/focus_android/metrics/metadata.yaml
INFO:root:For firefox-klar-android, found Glean dependencies: ['glean-core', 'lib-crash', 'glean-android']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/activation/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/activation/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/klar_android/metrics/metadata.yaml
INFO:root:For mozilla-vpn, found Glean dependencies: ['glean-js']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/activation/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/activation/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/first_session/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/first_session/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/installation/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/installation/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/startup_timeline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/startup_timeline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/baseline/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/baseline/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/events/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/events/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/metrics/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/metrics/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/migration/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/migration/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/addresses_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/addresses_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/bookmarks_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/bookmarks_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/creditcards_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/creditcards_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/history_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/history_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/logins_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/logins_sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/sync/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/tabs_sync/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/fenix/tabs_sync/metadata.yaml
INFO:root:For bergamot, found Glean dependencies: ['glean-js']
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozilla_vpn/main/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozilla_vpn/main/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozilla_vpn/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/mozilla_vpn/deletion_request/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/bergamot/custom/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/bergamot/custom/metadata.yaml
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/bergamot/deletion_request/view.sql
INFO:root:Writing /tmp/sql/moz-fx-data-shared-prod/bergamot/deletion_request/metadata.yaml
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/app/bigquery_etl/cli/__init__.py", line 60, in cli
    group(prog_name=prog_name)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/app/bigquery_etl/cli/glean_usage.py", line 112, in generate
    pool.map(
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/app/bigquery_etl/glean_usage/glean_app_ping_views.py", line 68, in generate_per_app
    for ping_name in p.get_pings():
  File "/usr/local/lib/python3.8/site-packages/mozilla_schema_generator/glean_ping.py", line 146, in get_pings
    return self._get_ping_data().keys()
  File "/usr/local/lib/python3.8/site-packages/mozilla_schema_generator/glean_ping.py", line 141, in _get_ping_data
    dependency_pings = self._get_json(self.ping_url_template.format(dependency))
  File "/usr/local/lib/python3.8/site-packages/mozilla_schema_generator/generic_ping.py", line 224, in _get_json
    return json.loads(GenericPing._get_json_str(url))
  File "/usr/local/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
telemetry_derived/foo/query.sql
whd commented 3 years ago

It turns out we've already encountered this issue before in https://github.com/mozilla/probe-scraper/issues/323 and I completely forgot about it. :jason identified a workaround that we might want to implement as this appears to happen relatively frequently of late.

whd commented 3 years ago

This hasn't happened in a while (but also schemas deploys were blocked). Closing in favor of https://github.com/mozilla/probe-scraper/issues/323, if it happens again and we forget about it we'll at least be able to find it in this repo as well via this issue.