KeyError: livingSpace, when trying to crawl Immobilienscout24

[DerLeole]

As the title says, I am crawling immoscout exclusively on a new device (it didnt work at all on my Debian 10 VPS, see my previous issue). Captcha solver works, but as soon as the bot gets to the "extract_entry_from_javascript" portion it crashes with the error mentioned in the title.

[alexanderroidl]

Hi Leolele99, could you please provide the full error log, as well as the contents of your config file and info about the OS you are using?

[DerLeole]

Sure.

The OS is PopOS 22.04, which should be overall very similar to Ubuntu 22.04.

Here is the full Error Log:

Traceback (most recent call last):
  File "/opt/flathunter/flathunt.py", line 105, in <module>
    main()
  File "/opt/flathunter/flathunt.py", line 101, in main
    launch_flat_hunt(config, heartbeat)
  File "/opt/flathunter/flathunt.py", line 31, in launch_flat_hunt
    hunter.hunt_flats()
  File "/opt/flathunter/flathunter/hunter.py", line 54, in hunt_flats
    for expose in processor_chain.process(self.crawl_for_exposes(max_pages)):
  File "/opt/flathunter/flathunter/hunter.py", line 33, in crawl_for_exposes
    return chain(*[try_crawl(searcher,url, max_pages)
  File "/opt/flathunter/flathunter/hunter.py", line 33, in <listcomp>
    return chain(*[try_crawl(searcher,url, max_pages)
  File "/opt/flathunter/flathunter/hunter.py", line 25, in try_crawl
    return searcher.crawl(url, max_pages)
  File "/opt/flathunter/flathunter/abstract_crawler.py", line 158, in crawl
    return self.get_results(url, max_pages)
  File "/opt/flathunter/flathunter/crawl_immobilienscout.py", line 59, in get_results
    return self.get_entries_from_javascript()
  File "/opt/flathunter/flathunter/crawl_immobilienscout.py", line 94, in get_entries_from_javascript
    return self.get_entries_from_json(result_json)
  File "/opt/flathunter/flathunter/crawl_immobilienscout.py", line 99, in get_entries_from_json
    return [
  File "/opt/flathunter/flathunter/crawl_immobilienscout.py", line 100, in <listcomp>
    self.extract_entry_from_javascript(entry.value) for entry in jsonpath_expr.find(json)
  File "/opt/flathunter/flathunter/crawl_immobilienscout.py", line 118, in extract_entry_from_javascript
    'size': str(entry["livingSpace"]),
KeyError: 'livingSpace'

If the respective line is somehow fixed with a debug string instead of looking up the entry, the same error occurs with the "numberOfRooms" key.

My config file is the following (some info replaced fro privacy reasons with REDACTED):

---
# Enable verbose mode (print DEBUG log messages)
verbose: True

# Should the bot endlessly looop through the URLs?
# Between each loop it waits for <sleeping_time> seconds.
# Note that Ebay will (temporarily) block your IP if you
# poll too often - don't lower this below 600 seconds if you
# are crawling Ebay.
loop:
    active: yes
    sleeping_time: 600

# Location of the Database to store already seen offerings
# Defaults to the current directory
#database_location: /path/to/database

# List the URLs containing your filter properties below.
# Currently supported services: www.immobilienscout24.de,
# www.immowelt.de, www.wg-gesucht.de, and www.ebay-kleinanzeigen.de.
# List the URLs in the following format:
# urls:
#   - https://www.immobilienscout24.de/Suche/...
#   - https://www.wg-gesucht.de/...
urls:
  - https://www.immobilienscout24.de/Suche/de/bayern/muenchen/wohnung-mieten?numberofrooms=1.0-3.0&price=-750.0&livingspace=20.0-100.0&minimuminternetspeed=100000&exclusioncriteria=swapflat&pricetype=calculatedtotalrent&sorting=2
  - https://www.immobilienscout24.de/Suche/de/bayern/muenchen/wg-zimmer?startrentaldate=2022-10-01&price=-700.0&livingspace=17.0-&sorting=2

# Define filters to exclude flats that don't meet your critera.
# Supported filters include 'max_rooms', 'min_rooms', 'max_size', 'min_size',
#   'max_price', 'min_price', and 'excluded_titles'.
#
# 'excluded_titles' takes a list of regex patterns that match against
# the title of the flat. Any matching titles will be excluded.
# More to Python regex here: https://docs.python.org/3/library/re.html
#
# Example:
# filters:
#   excluded_titles:
#     - "wg"
#     - "zwischenmiete"
#   min_price: 700
#   max_price: 1000
#   min_size: 50
#   max_size: 80
#   max_price_per_square: 1000
filters:
  excluded_titles:
    - "tauschwohnung"
    - "tausch"
    - "zwischenmiete"
  min_price: 300
  max_price: 750
  min_size: 15

# There are often city districts in the address which
# Google Maps does not like. Use this blacklist to remove
# districts from the search.
blacklist:
  - Innenstadt

# If an expose includes an address, the bot is capable of
# displaying the distance and time to travel (duration) to
# some configured other addresses, for specific kinds of
# travel.
#  
# Available kinds of travel ('gm_id') can be found in the
# Google Maps API documentation, but basically there are:
#   - "bicycling"
#   - "transit" (public transport)
#   - "driving"
#   - "walking"
# 
# The example configuration below includes a place for
# "John", located at the main train station of munich.
# Two kinds of travel (bicycle and transit) are requested,
# each with a different label. Furthermore a place for
# "Jane" is included, located at the given destination and
# with the same kinds of travel.
durations:
    - name: Hauptbahnhof
      destination: Hauptbahnhof, München
      modes: 
          - gm_id: transit
            title: "Öff."
          - gm_id: bicycling
            title: "Rad"
    - name: Uni
      destination: Boltzmannstraße 1, Garching
      modes: 
          - gm_id: transit
            title: "Öff."
          - gm_id: bicycling
            title: "Rad"

# Multiline message (yes, the | is supposed to be there), 
# to format the message received from the Telegram bot. 
# 
# Available placeholders:
#   - {title}: The title of the expose
#   - {rooms}: Number of rooms
#   - {price}: Price for the flat
#   - {durations}: Durations calculated by GMaps, see above
#   - {url}: URL to the expose
message: |
    {title}
    Zimmer: {rooms}
    Größe: {size}
    Preis: {price}
    Ort: {address}

    {url}

    Anfahrt:
    {durations}

# Calculating durations requires access to the Google Maps API. 
# Below you can configure the URL to access the API, with placeholders.
# The URL should most probably just kept like that. 
# To use the Google Maps API, an API key is required. You can obtain one
# without costs from the Google App Console (just google for it).
# Additionally, to enable the API calls in the code, set the 'enable' key to True
google_maps_api:
    key: REDACTED
    url: REDACTED
    enable: True

# If you are planning to scrape immoscout24.de, the bot will need 
# to circumvent the sites captcha protection by using a captcha 
# solving service. Register at either imagetypers or 2captcha 
# (the former is prefered), desposit some funds, uncomment the 
# corresponding lines below and replace your API key/token.
# you will also have to install a Chrome Web Driver and write below 
# the executable path, the driver_arguments can be left as is.
# captcha:
#       imagetypers:
#             token: alskdjaskldjfklj
#       2captcha:
#             api_key: alskdjaskldjfklj
#       driver_path: YOUR_CHROME_DRIVER_PATH
#       driver_arguments:
#         - "--headless"

captcha:
        imagetyperz:
              token: REDACTED
        driver_arguments:
          - "--headless"
          - "--disable-dev-shm-usage"
          - "--no-sandbox"
          - "--disable-extensions"
          - "disable-infobars"
          - "--disable-gpu"

# You can select whether to be notified by telegram or via a mattermost
# webhook. For all notifiers selected here a configuration must be provided
# below.
# notifiers:
#   - telegram
#   - mattermost
notifiers:
    - telegram

# Sending messages using Telegram requires a Telegram Bot configured. 
# Telegram.org offers a good documentation about how to create a bot.
# Once you read it, will make sense. Still: bot_token should hold the
# access token of your bot and receiver_ids should list the client ids
# of receivers. Note that those receivers are required to already have
# started a conversation with your bot. 
#
# telegram:
#   bot_token: 160165XXXXXXX....
#   receiver_ids:
#       - 12345....
#       - 67890....

telegram:
  bot_token: REDACTED
  receiver_ids:
      - REDACTED

# Sending messages via mattermost requires a webhook url provided by a
# mattermost server. You can find a description how to set up a webhook with
# the official mattermost documentation:
# https://docs.mattermost.com/developer/webhooks-incoming.html
# mattermost:
#   webhook_url: https://mattermost.example.com/signup_user_complete/?id=abcdef12356

# If you are running the web interface, you can configure Login with Telegram support
# Follow the instructions here to register your domain with the Telegram bot:
# https://core.telegram.org/widgets/login
#
# website:
#    bot_name: bot_name_xxx
#    domain: flathunter.example.com
#    session_key: SomeSecretValue
#    listen:
#      host: 127.0.0.1
#      port: 8080

# If you are deploying to google cloud,
# uncomment this and set it to your project id. More info in the readme.
# google_cloud_project_id: my-flathunters-project-id

# For websites like idealista.it, there are anti-crawler measures that can be
# circumvented using proxies.
use_proxy_list: False

[codders]

This seems to be fixed for now.