search

mozilla / probe-scraper

Scrape and publish Telemetry probe data from Firefox
https://mozilla.github.io/probe-scraper/
Mozilla Public License 2.0
21 stars 53 forks source link

glean-push function occasionally running out of memory #713

Closed BenWu closed 5 months ago

BenWu commented 5 months ago

The glean-push cloud function is occasionally running out of memory with a message like Memory limit of 1024 MiB exceeded with 1024 MiB used. Consider increasing the memory limit, see https://cloud.google.com/functions/docs/configuring/memory, causing some intermittent failures.

This is fairly low volume (10 times in the past week as of 2024-03-18) so not urgent but it's something to track since failed runs can cause issues for repos with infrequent commits. This is happening with multiple repos so it's not limited to a specific app.

This link should show logs for this error from the last week: https://cloudlogging.app.goo.gl/ooZVen6szh4D9xvF6

BenWu commented 5 months ago

A failed run in fxa https://github.com/mozilla/fxa/actions/runs/8333020299/job/22803514704

BenWu commented 5 months ago

Turns out cloud logging is missing a lot of entries and it's actually about 86 failures in the past week. I started a grafana dashboard to track this and I'm working on setting up alerts for it https://earthangel-b40313e5.influxcloud.net/d/eXUKp2JSk/probe-scraper