Heroku dynos are super memory constrained, so they have begun to fail while importing scraped data. By contrast, GH Actions runners for public repos have 16 GB of RAM, and run quite a bit faster (about 2 hours for the nightly import) without the risk of exceeding memory quota.
Heroku dynos are super memory constrained, so they have begun to fail while importing scraped data. By contrast, GH Actions runners for public repos have 16 GB of RAM, and run quite a bit faster (about 2 hours for the nightly import) without the risk of exceeding memory quota.
I've created an action to run ETL against a database at
DATABASE_URL
(a repo secret for openness-project-nmid). Let's update the scrape action to trigger that ETL workflow, rather than creating a run on a one-off Heroku dyno.Resources: