pangeo-forge / pangeo-forge-orchestrator

Database API and GitHub App backend for Pangeo Forge Cloud.
https://api.pangeo-forge.org/docs
Apache License 2.0
4 stars 1 forks source link

Recipe parsing overruns memory quotas #147

Closed cisaacstern closed 10 months ago

cisaacstern commented 2 years ago

Looks like the subprocess calls to pangeo-forge-runner tend to be taking more memory than the workers have allocated:

2022-09-29T01:38:44.101371+00:00 heroku[web.1]: Process running mem=782M(144.7%)
2022-09-29T01:38:44.104294+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
...
2022-09-29T01:40:32.268356+00:00 heroku[web.1]: Process running mem=674M(106.7%)
2022-09-29T01:40:32.294304+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)

At this point this seems to be mostly in the "warning" not "fatal error" territory, but it's worth being aware of, and we should increase the quota if it causes problems.

cisaacstern commented 2 years ago

Another view on this, from the Heroku dashboard:

Screen Shot 2022-09-28 at 6 55 01 PM