search

datamade / court-scrapers

MIT License
2 stars 0 forks source link

Optimize court call scrape #53

Closed antidipyramid closed 6 months ago

antidipyramid commented 6 months ago

Overview

This PR optimizes the court call scrape by:

  1. reducing unnecessary requests for associated case calendar values
  2. using scrapy's priority queue to avoid "out of order" requests (e.g. we'll complete all the requests for the cases on page 1 of results before moving on to page 2)

Both of these improvements make court call scrapes faster and reduces rejected requests.

Connects #52