search

AnthonyBloomer / rcp

Python client for RealClearPolitics.
28 stars 9 forks source link

certain pages fail - could use for political forum #6

Closed j-j-v closed 5 years ago

j-j-v commented 5 years ago

thanks for writing this - works very well. i noticed a few of the "General election" pages fail to return a result in get_poll_data()'s function, e.g

get_poll_data(https://www.realclearpolitics.com/epolls/latest_polls/state/)

any hope/help on this? thanks, Joe

AnthonyBloomer commented 5 years ago

Hi Joe,

Thanks for reporting this issue. I've updated the get_polls function with an optional url parameter.

Previously, the function would just search the latest polls URL: /epolls/latest_polls/.

With this update you can pass in the URL: 

get_polls(url='https://www.realclearpolitics.com/epolls/latest_polls/state/`)

This will return a data structure like this:

[{'poll': 'Atlanta Journal-Constitution',
  'title': 'Georgia: Trump vs. Biden',
  'url': 'https://www.realclearpolitics.com/epolls/2020/president/ga/georgia_trump_vs_biden-6974.html'},
 {'poll': 'Atlanta Journal-Constitution',
  'title': 'Georgia: Trump vs. Warren',
  'url': 'https://www.realclearpolitics.com/epolls/2020/president/ga/georgia_trump_vs_warren-6975.html'},
 {'poll': 'Atlanta Journal-Constitution',
  'title': 'Georgia: Trump vs. Sanders',
  'url': 'https://www.realclearpolitics.com/epolls/2020/president/ga/georgia_trump_vs_sanders-6979.html'},
 {'poll': 'Atlanta Journal-Constitution',
  'title': 'Georgia: Trump vs. Buttigieg',
  'url': 'https://www.realclearpolitics.com/epolls/2020/president/ga/georgia_trump_vs_buttigieg-6976.html'},
 {'poll': 'Atlanta Journal-Constitution',
  'title': 'Georgia: Trump vs. Harris',
  'url': 'https://www.realclearpolitics.com/epolls/2020/president/ga/georgia_trump_vs_harris-6978.html'},
...

You could then iterate over this result and pass the url key into the get_poll_data function to get the polling data for that poll, for example:

for poll in polls:
    td = get_poll_data(poll['url'])
    pprint(td)

I hope that helps!

j-j-v commented 5 years ago

that's what i call service!! trying it now Anthony =)

AnthonyBloomer commented 5 years ago

Nice, 1.2.1 is now available on PyPi. Enjoy! Let me know if you have any issues!

https://pypi.org/project/realclearpolitics/