search

ProgressiveCoders / projects

164 stars 5 forks source link

Looking for a little volunteer coding help to fight the Muslim ban #88

Open sylvia0x opened 7 years ago

sylvia0x commented 7 years ago

IRAP

I'm the Emergency Legal Response Manager with the International Refugee Assistance Project (IRAP), one of the groups fighting the Muslim ban. Learn more about us here: http://www.refugeerights.org.

Who we're looking for

Someone with experience with Python and web engineering.

Relevant Links

https://github.com/ecprice/newsdiffs
I want to adapt the Newsdiffs project to track the changing statements/guidance the Administration is putting out regarding the "Muslim ban". Specific sites to crawl would include https://travel.state.gov/content/travel/en/news/important-announcement.html, https://www.dhs.gov/news/2017/03/06/qa-protecting-nation-foreign-terrorist-entry-united-states and https://help.cbp.gov/app/answers/detail/a_id/67/noIntercept/1/session/L3RpbWUvMTQ4OTk2NDI3OS9zaWQvTFVRLWRaZG4%3D.

sdotson commented 7 years ago

How frequently do you want to crawl those pages? And what format would you like to see the changes (if there are any detected)?

sylvia0x commented 7 years ago

Hi @sdotson. No strong preferences about either of your questions. For frequency, I'd say whatever is practical -- maybe daily or twice daily? For format, something similar to what Newsdiffs.org uses would be great.

sdotson commented 7 years ago

Would something like this suit your needs? https://www.followthatpage.com/

Or perhaps one of these? http://www.smashingapps.com/2015/08/14/8-free-tools-to-monitor-any-website-content-changes.html

mcverter commented 7 years ago

I have written a few crawlers in python and php. I could help out.

sdotson commented 7 years ago

@mcverter I'm not that familiar with python but I think it should be straight forward to fork the newsdiff project and edit it to crawl the three pages.

sylvia0x commented 7 years ago

@mcverter That would be amazing! Please let me know what I can do to help! @sdotson Agree, I think it shouldn't be hard. I also tried using followthatpage.com but it's a little clunky.

mcverter commented 7 years ago

@sylvia0x Are you on progcode.org? Let's chat there.

mcverter commented 7 years ago

For now, archive.org maintains snapshots: https://web-beta.archive.org/web/20170314174200/https://www.dhs.gov/news/2017/03/06/qa-protecting-nation-foreign-terrorist-entry-united-states https://web-beta.archive.org/web/*/https://travel.state.gov/content/travel/en/news/important-announcement.html http://web.archive.org/web/20170323214055/https://help.cbp.gov/app/answers/detail/a_id/67/noIntercept/1/session/L3RpbWUvMTQ4OTk2NDI3OS9zaWQvTFVRLWRaZG4

gmisura commented 7 years ago

Is this still active/needed?