search

pgh-public-meetings / city-scrapers-pitt

Pittsburgh City Scrapers: sourcing public meetings in Pittsburgh
https://pgh-public-meetings.github.io/events/
MIT License
19 stars 66 forks source link

0030 spider pgh mayor office comm aff #43

Closed mishugana closed 4 years ago

mishugana commented 5 years ago

Please note lines 27 and 28 of the scraper: username = USERNAME_GOES_HERE password = PASSWORD_GOES_HERE There will need to be your nextdoor credentials.

bonfirefan commented 5 years ago

Hey @mishugana ! If you can sync your fork to let the build pass, we can merge this in. Otherwise, check the box on the right of this pull request to allow edits from maintainers.

Thanks :)

bonfirefan commented 4 years ago

A reminder to sync this with the upstream fork so that Travis tests will past! Also, to enable edits from maintainers!

Enable edits from maintainers!:

In the the right sidebar of your pull request, select Allow edits from maintainers. You can change these permissions at any time by unselecting this option.

Syncing your fork:

git remote add upstream https://github.com/bonfirefan/city-scrapers-pitt.git
git fetch upstream
git merge upstream/master
mishugana commented 4 years ago

i have synced with upstream and fixed the token to work with nextdoors changes. im on a pretty bumpy road in rural Laos editing on my phone via ssh, so im not going to push the changes until i can run some in depth tests, but here are updated offending lines

xpath_string = "//script[contains(text(),'CSRF')]" pattern = re.compile(r"var CSRFTOKEN = \'(.*?)\';", re.MULTILINE | re.DOTALL) token = response.xpath(xpath_string).re(pattern)[0]

will also allow edits when i get to a laptop

mishugana commented 4 years ago

ive pushed the new changes, its still failing the Travis tests, probably due to the username password issue. if you have a good idea of how to work around it let me know.

mishugana commented 4 years ago

https://softwareengineering.stackexchange.com/questions/370943/store-password-in-open-source-project

this solution should work, however it may still cause issues for beginners who pull the project without putting in the creds for this scraper (not sure if that is true, but if so, we can make this fail in a more graceful manner)

please make sure that you also follow https://docs.travis-ci.com/user/environment-variables/ for the env variables, marking them as secure, so they dont get leaked in logs. but also use a throwaway password (and if something leaks, at least its just nextdoor)