search

ckan / ckanext-deadoralive

A CKAN extension for the Dead or Alive link checker service
GNU Affero General Public License v3.0
6 stars 8 forks source link

Turn the link checker script into a web service #15

Closed seanh closed 10 years ago

seanh commented 10 years ago

Current plan is to us Flask and APScheduler.

If it were an actual web app instead of just an API script the link checker could provide its own API for the site to use to request ad-hoc link checks in response to user interaction. Something like:

/check_these_links accepts a list of objects: `[ {"id" : "...", "url": "..."}, ... ]. The service will check the links and post the results back to the site as usual.

Authentication

The ad-hoc mode requires auth. I like @wardi's suggestion that the web service is configured with a list of site's that it works for, each time the service receives a request for ad-hoc link checks it contacts the site that the request claims to be from and asks if whether it made this request, before proceeding.

This means the link checker service doesn't need to handle authentication, it can just fall back on the site's to do it.

seanh commented 10 years ago 
Link checker                                                     Client site
                 <----------------- Please check these links: -
                                         [<id_1>, <id_2> ...]
                 - 200 OK ------------------------------------>

                 - What's the URL for resource <id_1>? ------->
                 <---- The URL for resource <id_1> is <url_1> -  (Gets URL from db)
(Checks <url_1>) - Resource <id_1>'s URL is broken ----------->  (Saves result)

                 - What's the URL for resource <id_2>? ------->
                 <---- The URL for resource <id_2> is <url_2> -  (Gets URL from db)
(Checks <url_2>) - Resource <id_2>'s URL is broken ----------->  (Saves result)
                                ...
seanh commented 10 years ago

Use cases:

  1. Buttons the CKAN web UI: "Check this resource / dataset / organization / site for broken links now"
  2. Check a resource's link straight away when the user creates a new resource or changes a resource's URL

Note that once the link checker is a web service it will still be doing regular hourly link checks 90% of the time, the two ad-hoc use-cases above are the 10% nice-to-have cases.

seanh commented 10 years ago

Moved to https://github.com/ckan/deadoralive/issues/1