github-scraper to download all issues/PRs and keep them in sync

[milahu]

aka "live mirror" of github issues and PRs

better than the online version, because we can use custom search

should be a daemon = run continuously, watch github events, fetch incremental updates

should use the official github API (dont scrape html)

different than #1 (webhooks, push events) because use polling (pull events) for clients with no access to the nixos org (to configure webhooks)

https://docs.github.com/en/rest/activity/events

question: how fine-grained are pull events?

keywords

• github sync issues
• github issues downloader
• github scraper
• github watcher
• github archiver
• github offline mirror
• github proxy
• github bot
• github client

https://docs.github.com/en/rest/guides/scripting-with-the-rest-api-and-javascript

incremental web scraping

https://stackoverflow.com/questions/2576785/web-crawler-update-strategy

keywords

• event driven web scraping
• web scraping live data
• real time web scraping

[milahu]

similar tools

https://github.com/Xiphe/github-sync-labels-milestones

https://github.com/jessfraz/gitable

https://github.com/evanp/github-todotxt

https://github.com/Souvikns/Notion-Board/blob/main/src/adapters/github_adapter.ts

https://github.com/vtambourine/github-events-watcher

https://github.com/microsoft/gh-sync