Closed BenWu closed 5 months ago
A failed run in fxa https://github.com/mozilla/fxa/actions/runs/8333020299/job/22803514704
Turns out cloud logging is missing a lot of entries and it's actually about 86 failures in the past week. I started a grafana dashboard to track this and I'm working on setting up alerts for it https://earthangel-b40313e5.influxcloud.net/d/eXUKp2JSk/probe-scraper
The glean-push cloud function is occasionally running out of memory with a message like
Memory limit of 1024 MiB exceeded with 1024 MiB used. Consider increasing the memory limit, see https://cloud.google.com/functions/docs/configuring/memory
, causing some intermittent failures.This is fairly low volume (10 times in the past week as of 2024-03-18) so not urgent but it's something to track since failed runs can cause issues for repos with infrequent commits. This is happening with multiple repos so it's not limited to a specific app.
This link should show logs for this error from the last week: https://cloudlogging.app.goo.gl/ooZVen6szh4D9xvF6