search

grantwilliams / wg-gesucht-crawler-cli

Python web crawler / scraper for WG-Gesucht. Crawls the WG-Gesucht site for new apartment listings and send a message to the poster, based off your saved filters and saved text
MIT License
77 stars 28 forks source link

Crawler Error #14

Open sechsneun opened 4 years ago

sechsneun commented 4 years ago

Hi everyone, I'd love to use this tool to help me with my search for any type of housing, however, I'm getting a crawler error.

File "/Users/danijel/anaconda3/bin/wg-gesucht-crawler-cli", line 10, in <module> sys.exit(cli()) File "/Users/danijel/anaconda3/lib/python3.7/site-packages/click/core.py", line 722, in __call__ return self.main(*args, **kwargs) File "/Users/danijel/anaconda3/lib/python3.7/site-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/Users/danijel/anaconda3/lib/python3.7/site-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/danijel/anaconda3/lib/python3.7/site-packages/click/core.py", line 535, in invoke return callback(*args, **kwargs) File "/Users/danijel/anaconda3/lib/python3.7/site-packages/wg_gesucht/cli.py", line 109, in cli wg_gesucht.search() File "/Users/danijel/anaconda3/lib/python3.7/site-packages/wg_gesucht/crawler.py", line 465, in search ad_list = self.fetch_ads(filters_to_check) File "/Users/danijel/anaconda3/lib/python3.7/site-packages/wg_gesucht/crawler.py", line 289, in fetch_ads url_list.extend(self.process_filter_results(search_results)) File "/Users/danijel/anaconda3/lib/python3.7/site-packages/wg_gesucht/crawler.py", line 240, in process_filter_results post_date_link = result.find("td", {"class": "ang_spalte_datum"}).find("a")

So apparently they changed their source code, as I cant find the class ang_spalte_datum there any more. I just don't understand the logic well enough to find the new class they replaced it with.

Any help is appreciated, I'm really betting on this bot to find a place to live

grantwilliams commented 4 years ago

Hey @sechsneun Answered this in the other issue, https://github.com/grantwilliams/wg-gesucht-crawler-cli/issues/10#issuecomment-568722522 Let me know if that helps?