search

twneale / tadpoles

Scrape images of my kids from tadpoles.com and archive them in s3
BSD 3-Clause "New" or "Revised" License
18 stars 20 forks source link

Getting a "No months left to scrape" error. #4

Open kevinmbowen opened 6 years ago

kevinmbowen commented 6 years ago

Is anyone seeing this error?
04:13:55 WARNING app::iter_monthyear: No months left to scrape. Stopping.

See output below when running: ./scripts/update.sh

04:13:39 INFO app::do_login: Sleeping 2 seconds. 04:13:39 DEBUG app::sleep: Sleeping 2.16 04:13:41 INFO app::do_login: Clicking 'sumbit' button. 04:13:41 DEBUG app::sleep: Sleeping 2.56 04:13:44 INFO app::do_login: New url 04:13:44 INFO app::do_login: https://www.tadpoles.com/home_or_work 04:13:44 INFO app::dump_cookies: Dumping cookies. 04:13:44 INFO app::load_cookies: Loading cookies. 04:13:44 INFO app::add_cookies_to_browser: Adding the cookies to the browser. 04:13:44 INFO app::navigate_url: Navigating to 'https://www.tadpoles.com/parents' 04:13:44 INFO app::requestify_cookies: Transforming the cookies for requests lib. 04:13:44 INFO app::navigate_url: Navigating to 'https://www.tadpoles.com/parents' 04:13:55 WARNING app::iter_monthyear: No months left to scrape. Stopping. 04:13:55 INFO app::exit: Shutting down browser 04:13:55 INFO app::exit: Shutting down xfvb display

BenV commented 5 years ago

You need to change the xpath:

month_xpath_tmpl = '//*[@id="app"]/div[3]/div[1]/ul/li[%d]/div/div/div/div/span[%d]'

I'm forked off a different repo but here is the pull: https://github.com/AgentTyr/tadpoles/pull/2