search

flathunters / flathunter

A bot to help people with their rental real-estate search. 🏠🤖
GNU Affero General Public License v3.0
831 stars 179 forks source link

Key error 'website' #125

Closed tandomayo closed 2 years ago

tandomayo commented 3 years ago

Very new to coding so I need a bit of help (using pycharm).

Tried running the code according to the instructions. I get an error saying: "C:\Users......... Traceback (most recent call last): File "C:/Users.......", line 30, in listen = config['website']['listen'] File "C:\Users.......", line 43, in getitem return self.config[value] KeyError: 'website'

Process finished with exit code 1"

Trying to understand if I am putting in the website incorrectly?

"urls:

paulutsch commented 2 years ago

If you ran 'main.py', try running 'flathunt.py' instead.

tandomayo commented 2 years ago

I ran "flathunt.py" instead......I think its working.

Here is my config.yaml file just to be sure:

# Enable verbose mode (print DEBUG log messages)
# verbose: true

# Should the bot endlessly looop through the URLs?
# Between each loop it waits for <sleeping_time> seconds.
# Note that Ebay will (temporarily) block your IP if you
# poll too often - don't lower this below 600 seconds if you
# are crawling Ebay.
loop:
    active: yes
    sleeping_time: 600

# Location of the Database to store already seen offerings
# Defaults to the current directory
#database_location: /path/to/database

# List the URLs containing your filter properties below.
# Currently supported services: www.immobilienscout24.de,
# www.immowelt.de, www.wg-gesucht.de, and www.ebay-kleinanzeigen.de.
# List the URLs in the following format:
# urls:
#   - https://www.immobilienscout24.de/Suche/...
#   - https://www.wg-gesucht.de/...
urls:
   - https://www.ebay-kleinanzeigen.de/stadt/koeln/

# Define filters to exclude flats that don't meet your critera.
# Supported filters include 'max_rooms', 'min_rooms', 'max_size', 'min_size',
#   'max_price', 'min_price', and 'excluded_titles'.
#
# 'excluded_titles' takes a list of regex patterns that match against
# the title of the flat. Any matching titles will be excluded.
# More to Python regex here: https://docs.python.org/3/library/re.html
#
# Example:
# filters:
#   excluded_titles:
#     - "wg"
#     - "zwischenmiete"
#   min_price: 700
#   max_price: 1000
#   min_size: 50
#   max_size: 80
#   max_price_per_square: 1000
#filters:

# There are often city districts in the address which
# Google Maps does not like. Use this blacklist to remove
# districts from the search.
#blacklist:
#  - Innenstadt

# If an expose includes an address, the bot is capable of
# displaying the distance and time to travel (duration) to
# some configured other addresses, for specific kinds of
# travel.
#  
# Available kinds of travel ('gm_id') can be found in the
# Google Maps API documentation, but basically there are:
#   - "bicycling"
#   - "transit" (public transport)
#   - "driving"
#   - "walking"
# 
# The example configuration below includes a place for
# "John", located at the main train station of munich.
# Two kinds of travel (bicycle and transit) are requested,
# each with a different label. Furthermore a place for
# "Jane" is included, located at the given destination and
# with the same kinds of travel.
#durations:
#    - name: John
#      destination: Hauptbahnhof, München
#      modes:
#          - gm_id: transit
#            title: "Öff."
#          - gm_id: bicycling
#            title: "Rad"
#    - name: Jane
#      destination: Karlsplatz, München
#      modes:
#          - gm_id: transit
#            title: "Öff."
#          - gm_id: driving
#            title: "Auto"

# Multiline message (yes, the | is supposed to be there), 
# to format the message received from the Telegram bot. 
# 
# Available placeholders:
#   - {title}: The title of the expose
#   - {rooms}: Number of rooms
#   - {price}: Price for the flat
#   - {durations}: Durations calculated by GMaps, see above
#   - {url}: URL to the expose
#message: |
#    {title}
#    Zimmer: {rooms}
#    Größe: {size}
#    Preis: {price}
#    Ort: {address}

#    {url}

# Calculating durations requires access to the Google Maps API. 
# Below you can configure the URL to access the API, with placeholders.
# The URL should most probably just kept like that. 
# To use the Google Maps API, an API key is required. You can obtain one
# without costs from the Google App Console (just google for it).
# Additionally, to enable the API calls in the code, set the 'enable' key to True
#google_maps_api:
#    key: YOUR_API_KEY
#    url: https://maps.googleapis.com/maps/api/distancematrix/json?origins={origin}&destinations={dest}&mode={mode}&sensor=true&key={key}&arrival_time={arrival}
#    enable: False

# Register at 2captcha and enter your API key below. you will also
# have to install a Chrome Web Driver and write below the path to
# the executable.
#captcha:
#  api_key: YOUR_API_KEY
#  driver_path: YOUR_CHROME_DRIVER_PATH
#  driver_arguments:
#    - "--headless"

# Sending messages using Telegram requires a Telegram Bot configured. 
# Telegram.org offers a good documentation about how to create a bot.
# Once you read it, will make sense. Still: bot_token should hold the
# access token of your bot and receiver_ids should list the client ids
# of receivers. Note that those receivers are required to already have
# started a conversation with your bot. 
#
# telegram:
#   bot_token: 160165XXXXXXX....
#   receiver_ids:
#       - 12345....
#       - 67890....
telegram:
    bot_token: //
    receiver_ids: //

# If you are running the web interface, you can configure Login with Telegram support
# Follow the instructions here to register your domain with the Telegram bot:
# https://core.telegram.org/widgets/login
#
# website:
#    bot_name: bot_name_xxx
#    domain: flathunter.example.com
#    session_key: SomeSecretValue
#    listen:
#      host: 127.0.0.1
#      port: 8080

# If you are deploying to google cloud,
# uncomment this and set it to your project id. More info in the readme.
# google_cloud_project_id: my-flathunters-project-id

# For websites like idealista.it, there are anti-crawler measures that can be
# circumvented using proxies.
# use_proxy_list: True
fkalbe commented 2 years ago

You'll need to provide a Kleinanzeigen URL that shows you the listings that match your criteria. Right now, your URL (https://www.ebay-kleinanzeigen.de/stadt/koeln/) only points to the landing page for Cologne. Could you try something like https://www.ebay-kleinanzeigen.de/s-wohnung-mieten/koeln/c203l945 and report whether you still run into issues?

tandomayo commented 2 years ago

So I changed it and now it doesn't work. @fkalbe :(

C:\Users\T\PycharmProjects\flathunter\venv\Scripts\python.exe C:/Users/T/PycharmProjects/flathunter/flathunt.py
[2021/10/19 18:45:43|config.py         |INFO    ]: Using config C:\Users\T\PycharmProjects\flathunter/config.yaml
Traceback (most recent call last):
  File "C:/Users/T/PycharmProjects/flathunter/flathunt.py", line 89, in <module>
    main()
  File "C:/Users/T/PycharmProjects/flathunter/flathunt.py", line 86, in main
    launch_flat_hunt(config)
  File "C:/Users/T/PycharmProjects/flathunter/flathunt.py", line 46, in launch_flat_hunt
    hunter.hunt_flats()
  File "C:\Users\T\PycharmProjects\flathunter\flathunter\hunter.py", line 42, in hunt_flats
    for expose in processor_chain.process(self.crawl_for_exposes(max_pages)):
  File "C:\Users\T\PycharmProjects\flathunter\flathunter\sender_telegram.py", line 32, in process_expose
    self.send_msg(message)
  File "C:\Users\T\PycharmProjects\flathunter\flathunter\sender_telegram.py", line 39, in send_msg
    for chat_id in self.receiver_ids:
TypeError: 'int' object is not iterable

Process finished with exit code 1
fkalbe commented 2 years ago

It looks like there might a problem with your config file. Have you made sure that the Telegram Receiver-IDs have been entered in the correct format? You could also post your config file here for us to take a look on. Just make sure that you censor any secrets/passwords by replacing a few characters with asterix.

tandomayo commented 2 years ago

@fkalbe - you were right! I changed the format of the receiver id and now the code runs but a new problem arises - I do not get any messages on my telegram bot.

Here is the config file

# Enable verbose mode (print DEBUG log messages)
# verbose: true

# Should the bot endlessly looop through the URLs?
# Between each loop it waits for <sleeping_time> seconds.
# Note that Ebay will (temporarily) block your IP if you
# poll too often - don't lower this below 600 seconds if you
# are crawling Ebay.
loop:
    active: yes
    sleeping_time: 700

# Location of the Database to store already seen offerings
# Defaults to the current directory
#database_location: /path/to/database

# List the URLs containing your filter properties below.
# Currently supported services: www.immobilienscout24.de,
# www.immowelt.de, www.wg-gesucht.de, and www.ebay-kleinanzeigen.de.
# List the URLs in the following format:
# urls:
#   - https://www.immobilienscout24.de/Suche/...
#   - https://www.wg-gesucht.de/...
urls:
   - https://www.ebay-kleinanzeigen.de/s-wohnung-mieten/koeln/c203l945

# Define filters to exclude flats that don't meet your critera.
# Supported filters include 'max_rooms', 'min_rooms', 'max_size', 'min_size',
#   'max_price', 'min_price', and 'excluded_titles'.
#
# 'excluded_titles' takes a list of regex patterns that match against
# the title of the flat. Any matching titles will be excluded.
# More to Python regex here: https://docs.python.org/3/library/re.html
#
# Example:
# filters:
#   excluded_titles:
#     - "wg"
#     - "zwischenmiete"
#   min_price: 700
#   max_price: 1000
#   min_size: 50
#   max_size: 80
#   max_price_per_square: 1000
#filters:

# There are often city districts in the address which
# Google Maps does not like. Use this blacklist to remove
# districts from the search.
#blacklist:
#  - Innenstadt

# If an expose includes an address, the bot is capable of
# displaying the distance and time to travel (duration) to
# some configured other addresses, for specific kinds of
# travel.
#  
# Available kinds of travel ('gm_id') can be found in the
# Google Maps API documentation, but basically there are:
#   - "bicycling"
#   - "transit" (public transport)
#   - "driving"
#   - "walking"
# 
# The example configuration below includes a place for
# "John", located at the main train station of munich.
# Two kinds of travel (bicycle and transit) are requested,
# each with a different label. Furthermore a place for
# "Jane" is included, located at the given destination and
# with the same kinds of travel.
#durations:
#    - name: John
#      destination: Hauptbahnhof, München
#      modes:
#          - gm_id: transit
#            title: "Öff."
#          - gm_id: bicycling
#            title: "Rad"
#    - name: Jane
#      destination: Karlsplatz, München
#      modes:
#          - gm_id: transit
#            title: "Öff."
#          - gm_id: driving
#            title: "Auto"

# Multiline message (yes, the | is supposed to be there), 
# to format the message received from the Telegram bot. 
# 
# Available placeholders:
#   - {title}: The title of the expose
#   - {rooms}: Number of rooms
#   - {price}: Price for the flat
#   - {durations}: Durations calculated by GMaps, see above
#   - {url}: URL to the expose
#message: |
#    {title}
#    Zimmer: {rooms}
#    Größe: {size}
#    Preis: {price}
#    Ort: {address}

#    {url}

# Calculating durations requires access to the Google Maps API. 
# Below you can configure the URL to access the API, with placeholders.
# The URL should most probably just kept like that. 
# To use the Google Maps API, an API key is required. You can obtain one
# without costs from the Google App Console (just google for it).
# Additionally, to enable the API calls in the code, set the 'enable' key to True
#google_maps_api:
#    key: YOUR_API_KEY
#    url: https://maps.googleapis.com/maps/api/distancematrix/json?origins={origin}&destinations={dest}&mode={mode}&sensor=true&key={key}&arrival_time={arrival}
#    enable: False

# Register at 2captcha and enter your API key below. you will also
# have to install a Chrome Web Driver and write below the path to
# the executable.
#captcha:
#  api_key: YOUR_API_KEY
#  driver_path: YOUR_CHROME_DRIVER_PATH
#  driver_arguments:
#    - "--headless"

# Sending messages using Telegram requires a Telegram Bot configured. 
# Telegram.org offers a good documentation about how to create a bot.
# Once you read it, will make sense. Still: bot_token should hold the
# access token of your bot and receiver_ids should list the client ids
# of receivers. Note that those receivers are required to already have
# started a conversation with your bot. 
#
# telegram:
#   bot_token: 160165XXXXXXX....
#   receiver_ids:
#       - 12345....
#       - 67890....
telegram:
    bot_token: 21xxxxxxxxx
    receiver_ids:
       - 2xxxxxxx

# If you are running the web interface, you can configure Login with Telegram support
# Follow the instructions here to register your domain with the Telegram bot:
# https://core.telegram.org/widgets/login
#
# website:
#    bot_name: bot_name_xxx
#    domain: flathunter.example.com
#    session_key: SomeSecretValue
#    listen:
#      host: 127.0.0.1
#      port: 8080

# If you are deploying to google cloud,
# uncomment this and set it to your project id. More info in the readme.
# google_cloud_project_id: my-flathunters-project-id

# For websites like idealista.it, there are anti-crawler measures that can be
# circumvented using proxies.
# use_proxy_list: True
fkalbe commented 2 years ago

Have you added your bot to your telegram chats? If I remember correctly, you also need to message your bot first with /start, otherwise it won't be able to send you anything. If this doesn't help, you could try to delete the processed_ids.db file in the flathunter root directory to reset any already processed listings. Afterwards, restart the bot and check if this helped.

tandomayo commented 2 years ago

I had added my bot and started it. I tried what you suggested with the processed_ids but it still does not work. I know the code compiles as it shows statements like the following:

[2021/11/07 22:25:32|hunter.py |INFO ]: New offer: Suche Wg In Köln

But I just dont receive them on Telegram.

fkalbe commented 2 years ago

I suspect this might be an issue with Telegram, rather the bot not sending messages. To rule this out, you could use the Telegram API to send a test message using the following URL:

https://api.telegram.org/bot<YOUR_BOT_TOKEN>/sendMessage?chat_id=<YOUR_RECEIVER_ID>&parse_mode=Markdown&text=test

Simply replace <YOUR_BOT_TOKEN> and <YOUR_RECEIVER_ID> with your own information. Afterwards, copy the URL into the address bar of your browser and hit enter. Your browser should indicate whether the message was send successfully and you should receive the test message on your phone. If this doesn't work, then something is wrong with your the setup of your Telegram bot.

tandomayo commented 2 years ago

I tried what you said and the bot works. I got the test message.

I also tried looking at my indentation and with pycharm it automatically takes care of it.

I am really lost as to why I am not getting these messages and kinda tired....been working on this code for 3 hours now and still no luck 😞

fkalbe commented 2 years ago

Sorry to hear that 😕 My next idea would be to run the crawler in verbose mode, which gives us additional debug information, including whether any messages were sent. In your config file, just uncomment the second line (by removing the #) so that it says:

[...]
verbose: true
[...]

Feel free to post the logs after running the bot again (but be careful because your API key might be in there).

tandomayo commented 2 years ago

API key? Is that what I am missing?

Have a look at the logs:

[2021/11/14 18:06:10|abstract_crawler.py|DEBUG   ]: Got search URL https://www.ebay-kleinanzeigen.de/s-wohnung-mieten/koeln/c203l945
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 50667 Köln Altstadt
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 75 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: Gesuch
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 50933 Köln Braunsfeld
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 4 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 108 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 51103 Kalk
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 2 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 45 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 50996 Rodenkirchen
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 1 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 37,50 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 50674 Köln Altstadt
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 3 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 85 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 51103 Höhenberg
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 2 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 54 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 51149 Porz
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 1 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 45 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 50733 Nippes
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 3 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 70 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 51145 Porz
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 1 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 67 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 51103 Höhenberg
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 1 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 23 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 50737 Weidenpesch
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 2 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 51 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 50823 Ehrenfeld
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 3 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 86 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 50674 Köln Altstadt
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 2 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 60 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 50939 Köln Klettenberg
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 1 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 42 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 50767 Heimersdorf
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 3 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 105 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 50733 Nippes
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 2 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 70 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 50767 Heimersdorf
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 3 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 80 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 50735 Nippes
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 3 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 90 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 50937 Köln Sülz
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 1 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 50 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 50823 Ehrenfeld
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 2,5 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 63 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 50670 Köln Altstadt
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 4 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 90 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 50999 Sürth
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 2 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 54 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 50969 Zollstock
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 4 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 95,33 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 51069 Köln Dellbrück
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 2 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 63 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 50931 Lindenthal
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 2 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 45 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 50677 Köln Altstadt
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 2 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 50 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 50937 Köln Sülz
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 3 Zimmer
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: 95 m²
[2021/11/14 18:06:10|crawl_ebaykleinanzeigen.py|DEBUG   ]: extracted: 27
[2021/11/14 18:06:10|abstract_crawler.py|DEBUG   ]: Number of found entries: 27
[2021/11/14 18:06:10|idmaintainer.py   |DEBUG   ]: is_processed(1926412974)
[2021/11/14 18:06:10|idmaintainer.py   |DEBUG   ]: is_processed(1905268842)
[2021/11/14 18:06:11|idmaintainer.py   |DEBUG   ]: is_processed(1932683476)
[2021/11/14 18:06:11|idmaintainer.py   |DEBUG   ]: is_processed(1932672619)
[2021/11/14 18:06:11|idmaintainer.py   |DEBUG   ]: is_processed(1932643498)
[2021/11/14 18:06:11|idmaintainer.py   |DEBUG   ]: is_processed(1932643046)
[2021/11/14 18:06:11|idmaintainer.py   |DEBUG   ]: is_processed(1932563690)
[2021/11/14 18:06:11|idmaintainer.py   |DEBUG   ]: is_processed(1932518515)
[2021/11/14 18:06:11|idmaintainer.py   |DEBUG   ]: is_processed(1932400525)
[2021/11/14 18:06:12|idmaintainer.py   |DEBUG   ]: is_processed(1932380006)
[2021/11/14 18:06:12|idmaintainer.py   |DEBUG   ]: is_processed(1932301683)
[2021/11/14 18:06:12|idmaintainer.py   |DEBUG   ]: is_processed(1932271813)
[2021/11/14 18:06:12|idmaintainer.py   |DEBUG   ]: is_processed(1932216464)
[2021/11/14 18:06:12|idmaintainer.py   |DEBUG   ]: is_processed(1932166520)
[2021/11/14 18:06:12|idmaintainer.py   |DEBUG   ]: is_processed(1852770243)
[2021/11/14 18:06:13|idmaintainer.py   |DEBUG   ]: is_processed(1928137559)
[2021/11/14 18:06:13|idmaintainer.py   |DEBUG   ]: is_processed(1852733347)
[2021/11/14 18:06:13|idmaintainer.py   |DEBUG   ]: is_processed(1932101501)
[2021/11/14 18:06:13|idmaintainer.py   |DEBUG   ]: is_processed(1918686524)
[2021/11/14 18:06:13|idmaintainer.py   |DEBUG   ]: is_processed(1931997783)
[2021/11/14 18:06:14|idmaintainer.py   |DEBUG   ]: is_processed(1931960166)
[2021/11/14 18:06:14|idmaintainer.py   |DEBUG   ]: is_processed(1931916867)
[2021/11/14 18:06:14|idmaintainer.py   |DEBUG   ]: is_processed(1931840734)
[2021/11/14 18:06:14|idmaintainer.py   |DEBUG   ]: is_processed(1931837597)
[2021/11/14 18:06:14|idmaintainer.py   |DEBUG   ]: is_processed(1931792927)
[2021/11/14 18:06:14|idmaintainer.py   |DEBUG   ]: is_processed(1931718758)
[2021/11/14 18:06:14|idmaintainer.py   |DEBUG   ]: is_processed(1931665814)

Let me know if you can help.

fkalbe commented 2 years ago

By API-Key I meant your Telegram bot token. However, I don't think that's the issue right now. Instead, the crawler is not even trying to send any messages, because it has seen (and processed) all these listings before. Please try once again to delete your processed_ids.db file, restart the crawler and post the debug logs.

tandomayo commented 2 years ago

Hey @fkalbe - I understood the config file was wrong and was able to get the code working.

Thank you so much for the support and help! I will keep you posted if anything goes wrong.