Error when adding the 2Captcha service

[rusagent]

I pulled freshly and added the 2Captcha Service but not i get this:

 File "flathunt.py", line 89, in <module>
    main()
  File "flathunt.py", line 86, in main
    launch_flat_hunt(config)
  File "flathunt.py", line 46, in launch_flat_hunt
    hunter.hunt_flats()
  File "/home/user/janhuntboi/flathunter/hunter.py", line 42, in hunt_flats
    for expose in processor_chain.process(self.crawl_for_exposes(max_pages)):
  File "/home/user/janhuntboi/flathunter/hunter.py", line 22, in crawl_for_exposes
    for searcher in self.config.searchers()
  File "/home/user/janhuntboi/flathunter/hunter.py", line 23, in <listcomp>
    for url in self.config.get('urls', list())])
  File "/home/user/janhuntboi/flathunter/abstract_crawler.py", line 136, in crawl
    return self.get_results(url, max_pages)
  File "/home/user/janhuntboi/flathunter/crawl_immobilienscout.py", line 63, in get_results
    return self.get_entries_from_javascript()
  File "/home/user/janhuntboi/flathunter/crawl_immobilienscout.py", line 105, in get_entries_from_javascript
    return [ self.extract_entry_from_javascript(entry) for entry in entry_list ]
  File "/home/user/janhuntboi/flathunter/crawl_immobilienscout.py", line 105, in <listcomp>
    return [ self.extract_entry_from_javascript(entry) for entry in entry_list ]
  File "/home/user/janhuntboi/flathunter/crawl_immobilienscout.py", line 111, in extract_entry_from_javascript
    'image': entry["resultlist.realEstate"]["galleryAttachments"]["attachment"][0]["@xlink.href"] if "galleryAttachments" in entry["resultlist.realEstate"] else "https://www.static-immobilienscout24.de/statpic/placeholder_house/496c95154de31a357afa978cdb7f15f0_placeholder_medium.png",

[mordax7]

https://github.com/flathunters/flathunter/pull/81 just got merged. Can you please try fetching the last code stand and try again?

[Elo1338]

Just tried it. Still the same error. I just installed it and it worked, when I used only one immowelt link. After pasting a few other links it stopped working with the same error message with an additional

KeyError: 0

at the end.

Looks like only Immobilienscout is the problem. I deleted those links and it worked again.

[codders]

Hi @Elo1338 ,

Can you share which URLs you are using in your config? The error above is specifically related to Immobilienscout - Immowelt shouldn't be running the Immobilienscout crawler.

[Elo1338]

Hey @codders , Immowelt links worked fine, the Immobilienscout links are the problem. Sorry for the confusion.

[rusagent]

Any news?

[codders]

@rusagent Did you see what @mordax777 posted? There was some new code yesterday that might help. Does the latest work for you?

[rusagent]

Alright that works so far. Just started the bot though.

Would it be possible to use this addon somehow ? https://chrome.google.com/webstore/detail/buster-captcha-solver-for/mpbjkejclgfgadiemmefgebjfooflfhl?hl=en

[rusagent]

Well i still get an error. But this happens even if i comment out the captcha config...

Traceback (most recent call last):
  File "flathunt.py", line 89, in <module>
    main()
  File "flathunt.py", line 86, in main
    launch_flat_hunt(config)
  File "flathunt.py", line 46, in launch_flat_hunt
    hunter.hunt_flats()
  File "/home/user/janhuntboi/flathunter/hunter.py", line 42, in hunt_flats
    for expose in processor_chain.process(self.crawl_for_exposes(max_pages)):
  File "/home/user/janhuntboi/flathunter/hunter.py", line 22, in crawl_for_exposes
    for searcher in self.config.searchers()
  File "/home/user/janhuntboi/flathunter/hunter.py", line 23, in <listcomp>
    for url in self.config.get('urls', list())])
  File "/home/user/janhuntboi/flathunter/abstract_crawler.py", line 136, in crawl
    return self.get_results(url, max_pages)
  File "/home/user/janhuntboi/flathunter/crawl_immobilienscout.py", line 63, in get_results
    return self.get_entries_from_javascript()
  File "/home/user/janhuntboi/flathunter/crawl_immobilienscout.py", line 105, in get_entries_from_javascript
    return [ self.extract_entry_from_javascript(entry) for entry in entry_list ]
  File "/home/user/janhuntboi/flathunter/crawl_immobilienscout.py", line 105, in <listcomp>
    return [ self.extract_entry_from_javascript(entry) for entry in entry_list ]
  File "/home/user/janhuntboi/flathunter/crawl_immobilienscout.py", line 111, in extract_entry_from_javascript
    'image': entry["resultlist.realEstate"]["galleryAttachments"]["attachment"][0]["@xlink.href"] if "galleryAttachments" in entry["resultlist.realEstate"] else "https://www.static-immobilienscout24.de/statpic/placeholder_house/496c95154de31a357afa978cdb7f15f0_placeholder_medium.png",
KeyError: 0

[codders]

@rusagent What URL are you using on ImmoScout? I can try and reproduce the issue here.

[rusagent]

urls:

• https://www.immobilienscout24.de/Suche/radius/wohnung-mieten?centerofsearchaddress=Berlin;;;1276003001;Berlin;&numberofrooms=1.0-&price=-500.0&livingspace=-70.0&geocoordinates=52.51051;13.43068;5.0&enteredFrom=result_list
• https://www.immowelt.de/liste/berlin/wohnungen/mieten?prima=500&sort=relevanz
• https://www.ebay-kleinanzeigen.de/s-wohnung-mieten/berlin/anzeige:angebote/preis::400/wohnung-mieten/k0c203l3331+wohnung_mieten.qm_d:,60+wohnung_mieten.zimmer_d:,2

[codders]

I'm taking a look at this this morning. The problem you are seeing is because the crawler is expecting a browse page and not a search page. But I can try and make the crawler more general. Not sure if I get that done today - maybe over the weekend.

[Elo1338]

Sorry for the dumb question but what exactly is the difference between a browse page and a search page and how do I access a browse page in Immobilienscout?

[rusagent]

Good question - i have no answer :(

[codders]

@rusagent Okay. I think I fixed that in #84 . Can you pull the latest code and retry?

[rusagent]

Seems like it is working (Y) Thanks man!

[codders]

Cool. Good to know.