search

DemocracyClub / boundary-bot

🤖 Scraper for Boundary Commission reviews
1 stars 1 forks source link

add crawler #7

Closed chris48s closed 6 years ago

chris48s commented 6 years ago

Refs #6

Started noodling on this with scrapy. I've knocked up a spider class which can spit out a lit of objects like 

[
  {
    "local_auth": "bolsover",
    "latest_stage": "The Bolsover (Electoral Changes) Order 2017"
  },
  {
    "local_auth": "basingstoke-and-deane",
    "latest_stage": "Consultation on warding arrangements"
  },
  {
    "local_auth": "babergh",
    "latest_stage": "Consultation on draft recommendations"
  }
]

This allows us to work out what stage each review is at while it is in progress. Next job is to actually use the spider inside the main scraper so we can save the results to the database, work out when the description changes and raise Slack/GH notifications based on those events..

coveralls commented 6 years ago

Coverage Status

Changes Unknown when pulling 63e3a028b6e137f7cd4aa1be6f397fa2f5a7f59e on spider into on master.

coveralls commented 6 years ago

Coverage Status

Changes Unknown when pulling 63e3a028b6e137f7cd4aa1be6f397fa2f5a7f59e on spider into on master.

coveralls commented 6 years ago

Coverage Status

Changes Unknown when pulling 61249b44f9dd8c8d4c63ed46a3062f7b444180a0 on spider into on master.

coveralls commented 6 years ago

Coverage Status

Changes Unknown when pulling a955c0bc1189623ec8b6509bda0c08dd50061c58 on spider into on master.

coveralls commented 6 years ago

Coverage Status

Changes Unknown when pulling d8b0fbff0b8fa42d13b3504e6c613ba88037c802 on spider into on master.

chris48s commented 6 years ago

Deployment:

Next things to have a look at..