Closed perrymcmanis144 closed 5 months ago
Another failure in another repo: https://github.com/mozilla/experimenter/actions/runs/8283756036/job/22668600610?pr=10409
Stack trace from cloud function:
Traceback (most recent call last):
File "/layers/google.python.pip/pip/lib/python3.10/site-packages/flask/app.py", line 2190, in wsgi_app
response = self.full_dispatch_request()
File "/layers/google.python.pip/pip/lib/python3.10/site-packages/flask/app.py", line 1486, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/layers/google.python.pip/pip/lib/python3.10/site-packages/flask/app.py", line 1484, in full_dispatch_request
rv = self.dispatch_request()
File "/layers/google.python.pip/pip/lib/python3.10/site-packages/flask/app.py", line 1469, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "/layers/google.python.pip/pip/lib/python3.10/site-packages/functions_framework/__init__.py", line 99, in view_func
return function(request._get_current_object())
File "/workspace/probe_scraper/glean_push.py", line 47, in main
updated_paths = runner.main(
File "/workspace/probe_scraper/runner.py", line 708, in main
upload_paths += load_glean_metrics(
File "/workspace/probe_scraper/runner.py", line 609, in load_glean_metrics
raise ValueError("Found duplicate Glean metrics, check email for details")
ValueError: Found duplicate Glean metrics, check email for details
This is what I found in the logs:
Glean has detected duplicated metric identifiers coming from the product 'mozillavpn-backend-cirrus'.
- 'cirrus_events.enrollment' defined more than once in mozillavpn-backend-cirrus, nimbus-cirrus
- 'cirrus_events.enrollment_status' defined more than once in mozillavpn-backend-cirrus, nimbus-cirrus
- 'cirrus_events.instance_name' defined more than once in mozillavpn-backend-cirrus, nimbus-cirrus
I think an email should have been sent too emails don't get sent for the cloud function
I think this may be related: https://github.com/mozilla/probe-scraper/pull/708
And yes, emails were sent out. I got one along with several nimbus folks. I've actually gotten several emails now, stating the same thing.
I think the issue is that metrics and pings for mozillavpn-backend-cirrus
is still "cached" in the gcs bucket even though #708 removed them. I can create a backup and then try deleting it and it should allow things to run again. Certainly a bug somewhere here but I'll just get things working again
This should now be fixed after whd deleted the metrics file. I verified that runs are succeeding now. Some notes:
I opened a PR against glean and noticed that I had a probe-scraper 500 error. I tried repushing a few minutes later and it still persisted. It appears that some other folks might be experiencing it as well: https://mozilla.slack.com/archives/CEE12R4E8/p1710431615148949?thread_ts=1707921362.947109&cid=CEE12R4E8
Log from the PR: https://github.com/mozilla/glean/actions/runs/8283575535/job/22667074476