Additional environment variables or mount config.yaml in docker-compose

[mxfilerelatedcache]

I just set up the bot using docker-compose on my server and it works like a charm! Great work. However, when setting up the environment variables in the docker-compose.yaml, I realised some variables that would occur in the config.yaml are missing and not yet implemented as environment variables. For the durations using Google Maps API that would be okay, but I feel that adding the IS24 Cookie seems like an important step.

The readme mentions that both can be combined (environment variables & config.yaml), however is not quite clear as to how exactly this would work. Currently, I have the config.yaml in my working directory (where it should be), yet it does not seem to be used by the script. Am I doing something wrong while mounting the volume in the docker-compose.yaml? (I doubt that because the processed_ids.db seems to work fine) or am I missing something else?

Here is my docker-compose.yaml:

version: '3'

services:
  app:
    build: .
    shm_size: 512M # Workaround to fix docker crashes because of to small /dev/shm
    platform: linux/amd64 # For M1 to install google-chrome-stable; However still crashes on M1 when launching chrome (selenium.common.exceptions.WebDriverException: Message: unknown error: unable to discover open window in chrome (Session info: headless chrome=109.0.5414.119))
    command: python flathunt.py
    restart: always
    environment:
      - FLATHUNTER_TARGET_URLS=https://www.immobilienscout24.de/Suche/...
      - FLATHUNTER_LOOP_PERIOD_SECONDS=600
      - FLATHUNTER_NOTIFIERS=telegram
      - FLATHUNTER_TELEGRAM_BOT_TOKEN=XXX
      - FLATHUNTER_TELEGRAM_RECEIVER_IDS=XXX
      - FLATHUNTER_IMAGETYPERZ_TOKEN=XXX
      - FLATHUNTER_HEADLESS_BROWSER=yes
      - FLATHUNTER_FILTER_EXCLUDED_TITLES=XXX
      - FLATHUNTER_FILTER_MAX_PRICE=XXX
      - FLATHUNTER_FILTER_MIN_ROOMS=XXX
    volumes:
      - ./:/usr/src/app # not ideal, but the only way to incorporate the database (processed_ids.db), which is currently created at runtime

and this is my config.yaml:

durations:
  - name: Location 1
    destination: Location, City
    modes:
      - gm_id: transit
        title: "Öff."
      - gm_id: bicycling
        title: "Rad"
message: |
  {title}
  Zimmer: {rooms}
  Größe: {size}
  Preis: {price}
  Ort: {address}
  Wegzeit: {durations}

  {url}

google_maps_api:
  key: XXX
  enable: True

immoscout_cookie: XXX

[pomeloy]

From a quick glance, having the IS24-cookie or other crawler-specific settings read from an ENV variable is currently not possible without some code overhaul or refactoring.

Your configuration seems fine. Can you try changing the message format in the config.yaml to check whether the file gets read?

[mxfilerelatedcache]

Hey @pomeloy, I did some trial & error and the solution seems to be the following: Once you define the urls inside the config.yaml instead of thedocker-compose.yaml, the script seems to look for the information in the config.yaml, and then also used the cookie from there.

As the issue technically still persists, I would leave it open for now. However, with my fix, it is solvable.

[codders]

I've added the option to override the IS24 cookie with an environment variable in #587 - you can now set FLATHUNTER_IS24_COOKIE. In general it should be fine to combine the config file with the environment variables. If the environment variables are present, they will override the settings in the file. And if no config file is present, everything should be read from the environment.

I've added some tests for this. If you find that the functionality isn't working as expected with the latest code on main, please let me know. If not, please close the ticket as resolved.

Thanks for the report!

[mxfilerelatedcache]

Hey, thanks for your effort!

I just added FLATHUNTER_IS24_COOKIE as an environment variable and tested whether it works using the urls from the config.yaml, thus a combination of both config.yaml and environment variables. Everything now works as expected.

I close the issue now.