Open pjsier opened 5 years ago
I'll take a crack at this one today, the PDFs look to be easily parsed out by pdfminer.six
So I'll need some clarification on how we want to go about with this one, @ben-nathanson @bonfirefan.
Do we want to scrape every individual meeting? Because with what looks like at least 10 meetings per day, that could clog up the gears of City-scrapers by quite a bit. Or do we want to simply list the first one of them, which is 9:00AM?
It's good that you're thinking about scale. In this example we will want to scrape every individual meeting.
I would also say that we're building on top of existing tools that were designed to handle significant workloads, so it should be a long time before we encounter issues around scaling.
I think my concern is moreso that it might take up a lot of space on the calendar itself. Are we fine with having 10+ similar meetings listed on the same day in the calendar?
@cheog was looking into this with certain python modules, and I'm looking into it with different python modules--in my case, PDFMiner.
Spider Name:
pitt_zoning
Website:
http://pittsburghpa.gov/dcp/zba-schedule
Scraping notes:
Meeting info is all in PDF