search

yimbymelbourne / council-meeting-agenda-scraper

A method of getting and scraping council agendas to streamline housing abundance advocacy.
16 stars 18 forks source link

[New Scraper] VIC: Monash #40

Open jasonatclarkey opened 7 months ago

jasonatclarkey commented 7 months ago

Putting my hand up for this for now. First time writing python.

Council name: Monash Council

Link to agenda webpage: https://www.monash.vic.gov.au/About-Us/Council/Council-Meetings/Agendas-Minutes

Webpage quirks This is not a simple one. There is a CDN in front of this that may are-you-a-human-check you. Then, the page requires an call off to another endpoint, that looks like it requires some session/cookie data. It's probably some sort of DMS/LMS/CRN. EVEN THEN you don't get one PDF, you get a list.. ex:

7.1.2. 319-321 Springvale Road, Glen Waverley Development
7.1.3. TPA/40955D - 170-174 Highbury Rd Mount Waverley
7.1.4. 21 Banksia Street Clayton Construction - Extension of Time
7.2.1. Gender Equality Act 2020 Reporting Progress
7.2.2. 2023/24 Monash Quick Response Grants Program Recipients
7.2.3. Draft Monash Cricket Participation Plan
7.3.1. 2024092 - Cisco Internet Protocol Telephony
8.1. NOM - Councillor Discretionary Fund
10.1. Proposed Sale of Central Car Park, Glen Waverley

Each with their own PDF.

This will require refactoring to support I assume. Looks like this: https://github.com/yimbymelbourne/council-meeting-agenda-scraper/issues/22 Probably need to return a list[] of some sort.

So we'll hit their public page. Get a cookie Post to another endpoint Parse the JSON Build a list of PDFs

Additional context Add any other context or screenshots about the scraper request here.

jasonatclarkey commented 7 months ago

I have something working, but it's not pretty and the list[] is probably not what #22 would end up consuming.

2024-02-08 18:02:44,776 [INFO] MonashScraper:
           Name: ['7.1.1. Town Planning Schedule Report', '7.1.2. 319-321 Springvale Road, Glen Waverley Development', '7.1.3. TPA/40955D - 170-174 Highbury Rd Mount Waverley', '7.1.4. 21 Banksia Street Clayton Construction - Extension of Time', '7.2.1. Gender Equality Act 2020 Reporting Progress', '7.2.2. 2023/24 Monash Quick Response Grants Program Recipients', '7.2.3. Draft Monash Cricket Participation Plan', '7.3.1. 2024092 - Cisco Internet Protocol Telephony', '8.1. NOM - Councillor Discretionary Fund', '10.1. Proposed Sale of Central Car Park, Glen Waverley', '30 January 2024 - Council Meeting Decisions']
           Date: 30 January 2024
           Time: None
        BaseURI: https://www.monash.vic.gov.au
    DownloadURI: ['https://www.monash.vic.gov.au/files/assets/public/v/1/about-us/council/agendas/2024/30-january/7.1.1-town_planning_schedule_report.pdf', 'https://www.monash.vic.gov.au/files/assets/public/v/1/about-us/council/agendas/2024/30-january/7.1.2-319-321_springvale_road_glen-waverley-use-and-development-of-4-storey-medical-centre-carpark.pdf', 'https://www.monash.vic.gov.au/files/assets/public/v/1/about-us/council/agendas/2024/30-january/7.1.3-tpa40955d-170-174_highbury_.pdf', 'https://www.monash.vic.gov.au/files/assets/public/v/1/about-us/council/agendas/2024/30-january/7.1.4-21_banksia_street_clayton-extension-of-time.pdf', 'https://www.monash.vic.gov.au/files/assets/public/v/2/about-us/council/agendas/2024/30-january/gender_equality_act_2020_report_updated.pdf', 'https://www.monash.vic.gov.au/files/assets/public/v/1/about-us/council/agendas/2024/30-january/7.2.2-202324_monash_quick_response_grants.pdf', 'https://www.monash.vic.gov.au/files/assets/public/v/2/about-us/council/agendas/2024/30-january/7.2.3-draft_monash_cricket_participation.pdf', 'https://www.monash.vic.gov.au/files/assets/public/v/1/about-us/council/agendas/2024/30-january/7.3.1-2024092-cisco_internet_protocol.pdf', 'https://www.monash.vic.gov.au/files/assets/public/v/1/about-us/council/agendas/2024/30-january/8.1-councils_discretionary_expend.pdf', 'https://www.monash.vic.gov.au/files/assets/public/v/1/about-us/council/agendas/2024/30-january/10.1-proposed_sale_of_central_car_p.pdf', 'https://www.monash.vic.gov.au/files/assets/public/v/1/about-us/council/agendas/2024/30-january/30-january-2024-council-meeting-decisions.pdf']
2024-02-08 18:02:44,776 [INFO] MonashScraper: monash scraper finished successfully