datamade / nmid-scrapers

Scrapers for NMID
MIT License
0 stars 0 forks source link

Update scrape action to trigger ETL in main repo #26

Closed hancush closed 4 months ago

hancush commented 4 months ago

Heroku dynos are super memory constrained, so they have begun to fail while importing scraped data. By contrast, GH Actions runners for public repos have 16 GB of RAM, and run quite a bit faster (about 2 hours for the nightly import) without the risk of exceeding memory quota.

I've created an action to run ETL against a database at DATABASE_URL (a repo secret for openness-project-nmid). Let's update the scrape action to trigger that ETL workflow, rather than creating a run on a one-off Heroku dyno.

Resources: