search

flathunters / flathunter

A bot to help people with their rental real-estate search. 🏠🤖
GNU Affero General Public License v3.0
831 stars 179 forks source link

Can´t get the Script runing | Config.yml Error #163

Closed TheRealGenetic closed 2 years ago

TheRealGenetic commented 2 years ago

Hey i have some problems with the config.yml can somebody tell me what is wrong ? Output /usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.9) or chardet (3.0.4) doesn't match a supported version! warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported " [2022/04/08 17:12:36|config.py |INFO ]: Using config /home/noa/py/flathunter-main/config.yaml Traceback (most recent call last): File "flathunt.py", line 111, in main() File "flathunt.py", line 79, in main config = Config(config_handle.name) File "/home/noa/py/flathunter-main/flathunter/config.py", line 28, in init self.config = yaml.safe_load(file) File "/usr/local/lib/python3.8/dist-packages/yaml/init.py", line 125, in safe_load return load(stream, SafeLoader) File "/usr/local/lib/python3.8/dist-packages/yaml/init.py", line 81, in load return loader.get_single_data() File "/usr/local/lib/python3.8/dist-packages/yaml/constructor.py", line 49, in get_single_data node = self.get_single_node() File "/usr/local/lib/python3.8/dist-packages/yaml/composer.py", line 36, in get_single_node document = self.compose_document() File "/usr/local/lib/python3.8/dist-packages/yaml/composer.py", line 55, in compose_document node = self.compose_node(None, None) File "/usr/local/lib/python3.8/dist-packages/yaml/composer.py", line 84, in compose_node node = self.compose_mapping_node(anchor) File "/usr/local/lib/python3.8/dist-packages/yaml/composer.py", line 127, in compose_mapping_node while not self.check_event(MappingEndEvent): File "/usr/local/lib/python3.8/dist-packages/yaml/parser.py", line 98, in check_event self.current_event = self.state() File "/usr/local/lib/python3.8/dist-packages/yaml/parser.py", line 438, in parse_block_mapping_key raise ParserError("while parsing a block mapping", self.marks[-1], yaml.parser.ParserError: while parsing a block mapping in "/home/noa/py/flathunter-main/config.yaml", line 10, column 1 expected , but found '' in "/home/noa/py/flathunter-main/config.yaml", line 22, column 2

Config.yml(With Removed api keys)

Enable verbose mode (print DEBUG log messages)

verbose: true

Should the bot endlessly looop through the URLs?

Between each loop it waits for seconds.

Note that Ebay will (temporarily) block your IP if you

poll too often - don't lower this below 600 seconds if you

are crawling Ebay.

loop: active: yes sleeping_time: 500

Location of the Database to store already seen offerings

Defaults to the current directory

database_location: /path/to/database

List the URLs containing your filter properties below.

Currently supported services: www.immobilienscout24.de,

www.immowelt.de, www.wg-gesucht.de, and www.ebay-kleinanzeigen.de.

List the URLs in the following format:

urls:

Define filters to exclude flats that don't meet your critera.

Supported filters include 'max_rooms', 'min_rooms', 'max_size', 'min_size',

'max_price', 'min_price', and 'excluded_titles'.

#

'excluded_titles' takes a list of regex patterns that match against

the title of the flat. Any matching titles will be excluded.

More to Python regex here: https://docs.python.org/3/library/re.html

#

Example:

filters: excluded_titles:

There are often city districts in the address which

Google Maps does not like. Use this blacklist to remove

districts from the search.

blacklist:

If an expose includes an address, the bot is capable of

displaying the distance and time to travel (duration) to

some configured other addresses, for specific kinds of

travel.

Available kinds of travel ('gm_id') can be found in the

Google Maps API documentation, but basically there are:

- "bicycling"

- "transit" (public transport)

- "driving"

- "walking"

The example configuration below includes a place for

"John", located at the main train station of munich.

Two kinds of travel (bicycle and transit) are requested,

each with a different label. Furthermore a place for

"Jane" is included, located at the given destination and

with the same kinds of travel.

durations:

Multiline message (yes, the | is supposed to be there),

to format the message received from the Telegram bot.

Available placeholders:

- {title}: The title of the expose

- {rooms}: Number of rooms

- {price}: Price for the flat

- {durations}: Durations calculated by GMaps, see above

- {url}: URL to the expose

message: | {title} Zimmer: {rooms} Größe: {size} Preis: {price} Ort: {address} Fahrzeit: {durations}

{url}

Calculating durations requires access to the Google Maps API.

Below you can configure the URL to access the API, with placeholders.

The URL should most probably just kept like that.

To use the Google Maps API, an API key is required. You can obtain one

without costs from the Google App Console (just google for it).

Additionally, to enable the API calls in the code, set the 'enable' key to True

google_maps_api: key: AIzaSyCTN4I..... url: https://maps.googleapis.com/maps/api/distancematrix/json?origins={origin}&destinations={dest}&mode={mode}&sensor=true&key={key}&arrival_time={arrival} enable: true

Register at 2captcha and enter your API key below. you will also

have to install a Chrome Web Driver and write below the path to

the executable.

captcha: api_key: YOUR_API_KEY driver_path: YOUR_CHROME_DRIVER_PATH driver_arguments:

You can select whether to be notified by telegram or via a mattermost

webhook. For all notifiers selected here a configuration must be provided

below.

notifiers:

- telegram

- mattermost

notifiers:

Sending messages using Telegram requires a Telegram Bot configured.

Telegram.org offers a good documentation about how to create a bot.

Once you read it, will make sense. Still: bot_token should hold the

access token of your bot and receiver_ids should list the client ids

of receivers. Note that those receivers are required to already have

started a conversation with your bot.

# telegram: bot_token: 5230284557:AAGf7js....... receiver_ids:

Sending messages via mattermost requires a webhook url provided by a

mattermost server. You can find a description how to set up a webhook with

the official mattermost documentation:

https://docs.mattermost.com/developer/webhooks-incoming.html

mattermost:

webhook_url: https://mattermost.example.com/signup_user_complete/?id=abcdef12356

If you are running the web interface, you can configure Login with Telegram support

Follow the instructions here to register your domain with the Telegram bot:

https://core.telegram.org/widgets/login

#

website:

bot_name: bot_name_xxx

domain: flathunter.example.com

session_key: SomeSecretValue

listen:

host: 127.0.0.1

port: 8080

If you are deploying to google cloud,

uncomment this and set it to your project id. More info in the readme.

google_cloud_project_id: my-flathunters-project-id

For websites like idealista.it, there are anti-crawler measures that can be

circumvented using proxies.

use_proxy_list: True

codders commented 2 years ago

Looks from your error message like a problem with the config file:

in "/home/noa/py/flathunter-main/config.yaml", line 10, column 1 expected , but found '' in "/home/noa/py/flathunter-main/config.yaml", line 22, column 2

I would check what you have between line 10 column 1 and line 22 column 2, and see if any special characters (", : ) seem out of place.

TheRealGenetic @.***> schrieb am Fr., 8. Apr. 2022, 19:15:

Hey i have some problems with the config.yml can somebody tell me what is wrong ? Output /usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.9) or chardet (3.0.4) doesn't match a supported version! warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported " [2022/04/08 17:12:36|config.py |INFO ]: Using config /home/noa/py/flathunter-main/config.yaml Traceback (most recent call last): File "flathunt.py", line 111, in main() File "flathunt.py", line 79, in main config = Config(config_handle.name) File "/home/noa/py/flathunter-main/flathunter/config.py", line 28, in init self.config = yaml.safe_load(file) File "/usr/local/lib/python3.8/dist-packages/yaml/init.py", line 125, in safe_load return load(stream, SafeLoader) File "/usr/local/lib/python3.8/dist-packages/yaml/init.py", line 81, in load return loader.get_single_data() File "/usr/local/lib/python3.8/dist-packages/yaml/constructor.py", line 49, in get_single_data node = self.get_single_node() File "/usr/local/lib/python3.8/dist-packages/yaml/composer.py", line 36, in get_single_node document = self.compose_document() File "/usr/local/lib/python3.8/dist-packages/yaml/composer.py", line 55, in compose_document node = self.compose_node(None, None) File "/usr/local/lib/python3.8/dist-packages/yaml/composer.py", line 84, in compose_node node = self.compose_mapping_node(anchor) File "/usr/local/lib/python3.8/dist-packages/yaml/composer.py", line 127, in compose_mapping_node while not self.check_event(MappingEndEvent): File "/usr/local/lib/python3.8/dist-packages/yaml/parser.py", line 98, in check_event self.current_event = self.state() File "/usr/local/lib/python3.8/dist-packages/yaml/parser.py", line 438, in parse_block_mapping_key raise ParserError("while parsing a block mapping", self.marks[-1], yaml.parser.ParserError: while parsing a block mapping in "/home/noa/py/flathunter-main/config.yaml", line 10, column 1 expected , but found '' in "/home/noa/py/flathunter-main/config.yaml", line 22, column 2

Config.yml(With Removed api keys)

Enable verbose mode (print DEBUG log messages) verbose: true Should the bot endlessly looop through the URLs? Between each loop it waits for

seconds. Note that Ebay will (temporarily) block your IP if you poll too often - don't lower this below 600 seconds if you are crawling Ebay. loop: active: yes sleeping_time: 500 Location of the Database to store already seen offerings Defaults to the current directory #database_location: /path/to/database List the URLs containing your filter properties below. Currently supported services: www.immobilienscout24.de, www.immowelt.de, www.wg-gesucht.de, and www.ebay-kleinanzeigen.de. List the URLs in the following format: urls: - https://www.immobilienscout24.de/Suche/radius/wohnung-mieten?centerofsearchaddress=M%C3%BCnchen;;;1276002059;Bayern;&numberofrooms=1.0-&price=-670.0&pricetype=rentpermonth&geocoordinates=48.13912;11.58022;20.0&enteredFrom=one_step_search - https://www.wg-gesucht.de/1-zimmer-wohnungen-in-Munchen.90.1.1.0.html?offer_filter=1&city_id=90&noDeact=1&categories%5B%5D=1&rent_types%5B%5D=0&rMax=670 Define filters to exclude flats that don't meet your critera. Supported filters include 'max_rooms', 'min_rooms', 'max_size', 'min_size', 'max_price', 'min_price', and 'excluded_titles'. 'excluded_titles' takes a list of regex patterns that match against the title of the flat. Any matching titles will be excluded. More to Python regex here: https://docs.python.org/3/library/re.html Example: filters: excluded_titles: - "wg" - "zwischenmiete" - "tauschwohnung" There are often city districts in the address which Google Maps does not like. Use this blacklist to remove districts from the search. blacklist: - Innenstadt If an expose includes an address, the bot is capable of displaying the distance and time to travel (duration) to some configured other addresses, for specific kinds of travel. Available kinds of travel ('gm_id') can be found in the Google Maps API documentation, but basically there are: - "bicycling" - "transit" (public transport) - "driving" - "walking" The example configuration below includes a place for "John", located at the main train station of munich. Two kinds of travel (bicycle and transit) are requested, each with a different label. Furthermore a place for "Jane" is included, located at the given destination and with the same kinds of travel. durations: - name: Öffi destination: Westpark, München modes: - gm_id: transit title: "Öff." - gm_id: bicycling title: "Rad" - name: Roller destination: Westpark, München modes: - gm_id: transit title: "Öff." - gm_id: driving title: "bicycling" Multiline message (yes, the | is supposed to be there), to format the message received from the Telegram bot. Available placeholders: - {title}: The title of the expose - {rooms}: Number of rooms - {price}: Price for the flat - {durations}: Durations calculated by GMaps, see above - {url}: URL to the expose message: | {title} Zimmer: {rooms} Größe: {size} Preis: {price} Ort: {address} Fahrzeit: {durations} {url} Calculating durations requires access to the Google Maps API. Below you can configure the URL to access the API, with placeholders. The URL should most probably just kept like that. To use the Google Maps API, an API key is required. You can obtain one without costs from the Google App Console (just google for it). Additionally, to enable the API calls in the code, set the 'enable' key to True google_maps_api: key: AIzaSyCTN4I..... url: https://maps.googleapis.com/maps/api/distancematrix/json?origins={origin}&destinations={dest}&mode={mode}&sensor=true&key={key}&arrival_time={arrival} enable: true Register at 2captcha and enter your API key below. you will also have to install a Chrome Web Driver and write below the path to the executable. captcha: api_key: YOUR_API_KEY driver_path: YOUR_CHROME_DRIVER_PATH driver_arguments: - "--headless" You can select whether to be notified by telegram or via a mattermost webhook. For all notifiers selected here a configuration must be provided below. notifiers: - telegram - mattermost notifiers: - telegram Sending messages using Telegram requires a Telegram Bot configured. Telegram.org offers a good documentation about how to create a bot. Once you read it, will make sense. Still: bot_token should hold the access token of your bot and receiver_ids should list the client ids of receivers. Note that those receivers are required to already have started a conversation with your bot. telegram: bot_token: 5230284557:AAGf7js....... receiver_ids: - 1....... Sending messages via mattermost requires a webhook url provided by a mattermost server. You can find a description how to set up a webhook with the official mattermost documentation: https://docs.mattermost.com/developer/webhooks-incoming.html mattermost: webhook_url: https://mattermost.example.com/signup_user_complete/?id=abcdef12356 If you are running the web interface, you can configure Login with Telegram support Follow the instructions here to register your domain with the Telegram bot: https://core.telegram.org/widgets/login website: bot_name: bot_name_xxx domain: flathunter.example.com session_key: SomeSecretValue listen: host: 127.0.0.1 port: 8080 If you are deploying to google cloud, uncomment this and set it to your project id. More info in the readme. google_cloud_project_id: my-flathunters-project-id For websites like idealista.it, there are anti-crawler measures that can be circumvented using proxies. use_proxy_list: True — Reply to this email directly, view it on GitHub , or unsubscribe . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>
TheRealGenetic commented 2 years ago

@codders i don´t see any errors here is my config file https://pastebin.com/ENieAnqE also i see the error

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.9) or chardet (3.0.4) doesn't match a supported version! warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

any ideei

codders commented 2 years ago

I've pushed an update that should fix the dependency problems. I don't know if that will resolve your issue though. Let me know.

mordax7 commented 2 years ago

@TheRealGenetic Do you have white spaces in front(for example before the url)?

alexanderroidl commented 2 years ago

As there has not been a response in two months we will regard this issue as fixed. Feel free to open another one in case though! @TheRealGenetic