When meetings start, their comment pages are moved to the "past" scope and removed from the main eComment JavaScript file. This PR also checks the JavaScript file for the "past" scope so we don't lose comment links for ongoing/past meetings.
Testing
Pull down this branch, then cd into your local copy of the DataMade fork of scrapers-us-municipal and install this package.
Run pupa update lametro events --window=14 --rpm=0.
Confirm you see the following log entry once:
04/10/2020 11:25:42 INFO scrapelib: GET - https://metro.granicusideas.com/meetings.js
04/10/2020 11:25:42 INFO scrapelib: GET - https://metro.granicusideas.com/meetings.js?scope=past
Confirm you see at least two log messages that look like Adding eComment link URL from URL.
Inspect the events in the scraped data directory and confirm the comment links were added to extras appropriately: grep -r "ecomment" /tmp/cache/_data/lametro/event_*. (Assumes you're using the pupa_settings.py file in the DataMade fork – update the path to the data directory if it's different.)
Description
When meetings start, their comment pages are moved to the "past" scope and removed from the main eComment JavaScript file. This PR also checks the JavaScript file for the "past" scope so we don't lose comment links for ongoing/past meetings.
Testing
cd
into your local copy of the DataMade fork ofscrapers-us-municipal
and install this package.pupa update lametro events --window=14 --rpm=0
.Adding eComment link URL from URL
.extras
appropriately:grep -r "ecomment" /tmp/cache/_data/lametro/event_*
. (Assumes you're using thepupa_settings.py
file in the DataMade fork – update the path to the data directory if it's different.)