search

CleanCodeCompany / adopt-ireland-scraper

0 stars 0 forks source link

Design common infrastructure for scrapers #1

Open CFitzsimons opened 6 years ago

CFitzsimons commented 6 years ago

Ideally each scraper should output the same set of data. Something in the form of:

{
  entryDate: "date entered into recuse site", 
  breed: "dog breed...",
  age: "age...",
  sex: "sex..."
}

We'll need to verify that all the incoming data is the correct type (mostly, if not all strings) and that they are not null or empty.

CFitzsimons commented 6 years ago

@JackTilley92 Just a suggestion but libraries like ajv allow us to use json schema to validate data. We could write a small json schema and use that to validate data we get back from the scraper.

ajv: https://github.com/epoberezkin/ajv json schema: http://json-schema.org/

JSON Schema is probably a bit overkill for what we're doing here but it is extremely useful in a variety of languages.