search

ubiquity / scraper-kernel

A Puppeteer-based scraping platform with modular, page-level scraping logic.
0 stars 3 forks source link

Enrichment #5

Closed 0x4007 closed 1 year ago

0x4007 commented 1 year ago

Users in the database should be enriched using the GitHub User API and the Twitter API.

Due to rate limits, these should run in small batches (e.g. 1 per second?) via GitHub Actions cronjobs until rate limit is hit.

0x4007 commented 1 year ago

This issue should be ported to a new repo for the page controllers once the page controllers are separated from the scraper kernel.