tosdr / edit.tosdr.org

👍👎 A new web app to rate services
https://edit.tosdr.org
GNU Affero General Public License v3.0
213 stars 37 forks source link

Bring back crawl button #959

Closed michielbdejong closed 3 years ago

michielbdejong commented 3 years ago

Testing this locally now.

michielbdejong commented 3 years ago

It all seems to sort-of work. It's scary, but let's move fast and break stuff, then improve iteratively.

With this, basically, the approver-list is oldest-first, so we'll see some very old points appear in our home views. As we review them, let's recrawl the docs, and the idea is that many of the old 'pending' will become 'pending-not-found', and at the same time, some 'approved-not-found' will appear. As we fix those (click 'Update Excerpt', we gradually fix our outdated points.

michielbdejong commented 3 years ago

The cron job for check_quotes and deploy_site are running hourly for now, so we get good fine-grained checkpoints if things go wrong. As we improve, I think we can reduce it to once every 8 hours.

michielbdejong commented 3 years ago

Ideally we would have a button for 'check quotes in this doc' but for now the only known way to do that is using the node src/eto-admin.js 789 check_quotes 90 command from https://github.com/tosdr/tosback-crawler This means only people who have PSQL access can do this. Others will have to wait for :16 minutes past the hour, when the script runs to check quotes for all docs.