ElixirTeSS / TeSS_scrapers

TeSS HTML page scrapers in Ruby looking for training resources and events metadata.
Other
9 stars 9 forks source link

New Scraper Framework #68

Open njall opened 5 years ago

njall commented 5 years ago

Recreate the Ruby scraper library in NodeJS.

The motivation is to broaden the ability of 3rd parties to contribute scrapers by using a more ubiquitous language.

To recreate the scraper framework you'll need to make a new API client that interfaces with the write API. The Ruby version is kept as a separate library here: https://github.com/ElixirTeSS/TeSS_api_client

Then you'll have to produce a common utility library (see Scraper modules in https://github.com/ElixirTeSS/TeSS_scrapers/blob/master/lib/tess/scrapers/scraper.rb). In Ruby, the scrapers inherit from this module which interfaces with the TeSS API and collects statistics about success rates.