search

nrjones8 / robots-dot-txt-archive-bot

A project to collect, archive, and publish robots.txt files from across the internet - with a focus on government websites
https://robots-dot-txt-db.com/
6 stars 0 forks source link

streamline the data update process #12

Open nrjones8 opened 4 years ago

nrjones8 commented 4 years ago

maybe just add it to the gh action that runs once a week? ideally it's

  1. collect data
  2. update the combined CSV
  3. commit that all to the repo
  4. publish the new CSV to datasette