search

orangecoding / fredy

:heart: Fredy - [F]ind [R]eal [E]states [D]amn Eas[y] - Fredy will constantly search for new listings on sites like Immoscout or Immowelt and send new results to you, so that you can focus on more important things in life ;)
http://www.orange-coding.net
MIT License
212 stars 54 forks source link

400 Bad Request returned by telegram api, "can't parse entities", logging could be improved as well #24

Closed carstenhag closed 3 years ago

carstenhag commented 3 years ago

I had set up a telegram adapter. The chat ID was taken from @RawDataBot. Further, I have validated the token by running

curl -X GET https://api.telegram.org/botREDACTED:REDACTED successfully.

The fredy dashboard says it has found 20 listings, but there was an error while sending the message to telegram. It would help if there were more debug statements, as currently it's unclear what the api call payload was, only the response is logged.

shell@ubuntu-2gb-fsn1-6:~/fredy$ yarn run start
yarn run v1.22.5
$ node index.js
Started Fredy successfully. Ui can be accessed via http://localhost:9998
Started API service on port 9998
ErrorClass [HTTPError]: Response code 400 (Bad Request)
    at IncomingMessage.<anonymous> (/home/shell/fredy/node_modules/yarl/main.js:179:22)
    at IncomingMessage.emit (node:events:381:22)
    at endReadableNT (node:internal/streams/readable:1307:12)
    at processTicksAndRejections (node:internal/process/task_queues:81:21) {
  host: 'api.telegram.org',
  hostname: 'api.telegram.org',
  method: 'POST',
  path: '/botREDACTED:REDACTED/sendMessage',
  response: {
    statusCode: 400,
    statusMessage: 'Bad Request',
    body: `{"ok":false,"error_code":400,"description":"Bad Request: can't parse entities: Can't find end of the entity starting at byte offset 58"}`
  }
}
orangecoding commented 3 years ago

I'll check with telegram soonish

orangecoding commented 3 years ago

I cannot reproduce this issue, for me, telegram works. if you select telegram as adapter, type in the needed data and click on "try", what does it say?

carstenhag commented 3 years ago

Oh, didn't realize there's a "try" button. Yes, that works. It posted the message. I'll try to check whether it still returns that entities error when sending a real message

orangecoding commented 3 years ago

Could be it was a follow-up issue from the scraping-ant problem. If you pull the latest fredy version, it should be a bit more resistent against failures.

If it works, please close the issues as solved. Thanks

carstenhag commented 3 years ago

Pulled 30min ago, built it and ran it. Still getting the same issue. I guess that it has to do something with the listings found, I am using this as search URL: https://www.immobilienscout24.de/Suche/radius/wohnung-mieten?centerofsearchaddress=Heidelberg;;;;;Bahnstadt&numberofrooms=3.0-&pricetype=rentpermonth&geocoordinates=49.40472;8.66334;4.0&sorting=2&enteredFrom=result_list

orangecoding commented 3 years ago

Works like a charm for meimage

carstenhag commented 3 years ago

Hmm ok, weird, I implemented some debug logging and will wait until it pulls listings the next time

orangecoding commented 3 years ago

I will re-write the telegram adapter tomorrow (or on the weekend), it was on my todo list anyway as the lib im using is not maintained anymore and the code to send stuff to the botuser is pretty simple. This might (or might not) solve your issue...

carstenhag commented 3 years ago 
shell@ubuntu-2gb-fsn1-6:~/fredy$ vim debug.log 

yarn run v1.22.5
$ node index.js
Started Fredy successfully. Ui can be accessed via http://localhost:9998
Started API service on port 9998
Job: TestJob | Service _TestCall_ found _1_ new listings:

Job: m3zP_rhDbkZLgx-_3_2sS | Service _immoscout_ found _20_ new listings:

*WG-geeignete 4 ZKB in Heidelberg-Rohrbach*
Rohrbach, Heidelberg | 720 € | 75 m²
[LINK](https://www.immobilienscout24.de/expose/128069163)

,*Exklusive, gepflegte 4-Zimmer-Wohnung mit Bal...*
Bergstraße 110, Handschuhsheim, Heidelberg | 1.680 € | 135 m²
[LINK](https://www.immobilienscout24.de/expose/128060811)

,*Tolle Altstadtwohnung (leider ohne Möbel)*
Altstadt, Heidelberg | 1.120 € | 95 m²
[LINK](https://www.immobilienscout24.de/expose/71210337)

,*Helle + gepflegte 4-Zimmer-Wohnung mit Balkon*
Breslauerstraße 44, Kirchheim, Heidelberg | 880 € | 82 m²
[LINK](https://www.immobilienscout24.de/expose/128054644)

,*Ansprechende 3-Zimmer-DG-Wohnung mit EBK in H...*
Rathausstr. 1, Rohrbach, Heidelberg | 940 € | 80 m²
[LINK](https://www.immobilienscout24.de/expose/128006059)

,*großzügige 3,5 ZKB-Wohnung mit großem überdac...*
Hintere Lisgewann 28, Eppelheim, Rhein-Neckar-Kreis | 980 € | 96 m²
[LINK](https://www.immobilienscout24.de/expose/75152040)

,*Helle, freundliche 3-Zimmer-Wohnung mit Balko...*
Rottmannstr. 1, Handschuhsheim, Heidelberg | 760 € | 61 m²
[LINK](https://www.immobilienscout24.de/expose/128026925)

,*HD- Brückenkopfstrasse Anno1898 : 3 ZKB im Da...*
Neuenheim, Heidelberg | 800 € | 63 m²
[LINK](https://www.immobilienscout24.de/expose/128018303)

,*HD-Weststadt: Schöne 3-Zimmerwohnung in der B...*
Blumenstraße 40, Weststadt, Heidelberg | 1.195 € | 77 m²
[LINK](https://www.immobilienscout24.de/expose/71301056)

,*Stilvolle 3-Zimmer-Wohnung mit Balkon und EBK...*
Endemannstraße 13, Weststadt, Heidelberg | 1.200 € | 85 m²
[LINK](https://www.immobilienscout24.de/expose/125116362)

,*Möblierte Wohnung zur Zwischenmiete*
Richard Wagner Strasse 2, Handschuhsheim, Heidelberg | 1.200 € | 100 m²
[LINK](https://www.immobilienscout24.de/expose/124571144)

,*Im Herzen der Heidelberger Altstadt: 4-Zimmer...*
Altstadt, Heidelberg | 1.840 € | 130 m²
[LINK](https://www.immobilienscout24.de/expose/108059514)

,*Seniorengerechte Wohnung in Wieblingen*
Mannheimer Str. 267, Wieblingen, Heidelberg | 641,34 € | 93,9 m²
[LINK](https://www.immobilienscout24.de/expose/127954458)

,*Moderne Wohnung direkt am Puls von Neuenheim*
Brückenstrasse 17, Neuenheim, Heidelberg | 1.300 € | 80 m²
[LINK](https://www.immobilienscout24.de/expose/127917200)

,*Traumhafter, sanierter Altbau mit verglasten ...*
Rohrbach, Heidelberg | 1.150 € | 90 m²
[LINK](https://www.immobilienscout24.de/expose/127541883)

,*Großzügige 4 Zimmer Wohnung in Heidelberg - S...*
Slevogtstr. 6, Südstadt, Heidelberg | 1.990 € | 151 m²
[LINK](https://www.immobilienscout24.de/expose/127944294)

,*Jugendstil: Altes neu beleben! Sehr schöne, h...*
Weststadt, Heidelberg | 1.040 € | 86,5 m²
[LINK](https://www.immobilienscout24.de/expose/72921232)

,*Gepflegte 3-Zimmer-Maisonette-Wohnung im EG m...*
Wieblingen, Heidelberg | 720 € | 66 m²
[LINK](https://www.immobilienscout24.de/expose/127887890)

,*Exklusive, sanierte 3,5-Zimmer-Hochparterre-W...*
Weststadt, Heidelberg | 1.100 € | 89 m²
[LINK](https://www.immobilienscout24.de/expose/127887242)

,*Sonnige, ruhige Wohnung mit Garten*
Pfaffengrund, Heidelberg | 700 € | 60 m²
[LINK](https://www.immobilienscout24.de/expose/84013986)

{ parse_mode: 'Markdown' }

ErrorClass [HTTPError]: Response code 400 (Bad Request)
    at IncomingMessage.<anonymous> (/home/shell/fredy/node_modules/yarl/main.js:179:22)
    at IncomingMessage.emit (node:events:381:22)
    at endReadableNT (node:internal/streams/readable:1307:12)
    at processTicksAndRejections (node:internal/process/task_queues:81:21) {
  host: 'api.telegram.org',
  hostname: 'api.telegram.org',
  method: 'POST',
  path: '/bot1763015523:AAHCnWUwxsq1HyQ95KfRKVMI699_V8sqd9k/sendMessage',
  response: {
    statusCode: 400,
    statusMessage: 'Bad Request',
    body: `{"ok":false,"error_code":400,"description":"Bad Request: can't parse entities: Can't find end of the entity starting at byte offset 58"}`
  }
}

But this also doesn't help me with debugging, haha. Okay, sounds good :)

orangecoding commented 3 years ago

Essentially your output is exactly as mine...

orangecoding commented 3 years ago

@carstenhag please pull again and try it. It should (TM) work now. At least for me it worked no matter what text I let it send to me.

See

https://github.com/orangecoding/fredy/commit/9f0bcbd85fd39d5d5c211b5ce35671d2717ab657

By the way, pull requests are welcomed ;)

carstenhag commented 3 years ago

It does work now, yay! My sister is really happy now :)

About the PRs: I have a really hard time getting into untyped js code, so it's not trivial for me :D.