search

tdlib / telegram-bot-api

Telegram Bot API server
https://core.telegram.org/bots
Boost Software License 1.0
3.11k stars 593 forks source link

API SSL error with some ipv6 addresses #556

Open tootai opened 6 months ago

tootai commented 6 months ago

Hi list,

I use Telegram bot on a Debian12 ipv6 server running librenms and Transport being Telegram. I also wrote a script to send messages to a group. Since those last days I didn't receive any alert from Librenms and I discover that curl fail with error 28 SSL connection timeout when testing from Librenms. An error occurs too after 2 minutes when I run curl from command line like

curl https://api.telegram.org/bot/sendMessage?chat_id=&text=Is%20not,%20functional%20yet % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- 0:02:01 --:--:-- 0 curl: (35) Recv failure: Connection reset by peer

Running curl -4 ... and the job is done. api.telegram.org is resolved in ipv4 and ipv6, nc -v 443 succeeded as well as a mtr to the IP.

I discovered that with another ipv6 address, from the same server, it's OK ! What's going on here ? The failing ipv6 address being from the IP range 2001:67c:1298::/48.

Thanks for your support

levlam commented 6 months ago

For me api.telegram.org resolves to 2001:67c:4e8:f004::9, which isn't in the mentioned network.

tootai commented 6 months ago

Exactly, but why I didn't get full answer with IPv6 belonging to the above /48 range ?

levlam commented 6 months ago

Why should you receive an answer from the 2001:67c:1298::/48 IP address, if api.telegram.org resolves to another address?

tootai commented 6 months ago

OK, seems I didn't explain clearly: my IP is from 2001:67c:1298::/48 range. My other IP is from 2a01:4f8:210::/48 range. Connecting to api using the IP from this second range is OK but when I use the other IP from the first range I see some traffic with tcpdump, outgoing and incoming, which means both servers are connected and exchanging datas but nothing else, it finish with a timeout in terminal mode or error 28 SSL inside Librenms.

Hope you now understand the problematic ;)

levlam commented 6 months ago

This looks like an Internet connectivity issue, which happens much more often with IPv6 networks. You can try to use tcptraceroute to find the cause of the issue and report it to your Internet provider.

tootai commented 6 months ago

I don't think so: from the same IP range other computer I have the same problem. Also this server is ONLY ipv6 and running 24h/24h (backups, web server, emails aso) so I would not put 1 cent of the connectivity. Server is in a DC with other of our servers, no one is showing a problem. And remember, Librenms is running on it watching our servers in various countries, not any alert about such a failure.

tootai commented 6 months ago

mtr report 17ms avg time with 0% loss for 100 packets sent

levlam commented 6 months ago

mtr uses ICMPv6 packets, so its results are often irrelevant when investigating TCP connectivity issues.

tootai commented 6 months ago

Please, dont't try to explain me how mtr/icmp/... works. Generally we all use ICMP/ICMPv6 to check connectivity. If you feel out of idea, just tell it.

Remember with one IP range it's OK the other one not, both using the same Internet provider till a certain point.

levlam commented 6 months ago

If site works from one network and doesn't work from another, then this is very likely to be a network connectivity issue. The used protocol for data transfer is of utmost importance in the case of such network issues. For api.telegram.org there is no difference from which IPv6 address the request is received.

tootai commented 6 months ago

I opened a case on peering partner. I ran a tcptraceroute6 and got for port 80

root@zone-s:/usr/local/bin# tcptraceroute6 api.telegram.org traceroute to api.telegram.org (2001:67c:4e8:f004::9) from 2001:67c:1298::ff04, port 80, from port 17001, 30 hops max, 60 bytes packets 1 2a01:4f8:120:1104::70:1 (2a01:4f8:120:1104::70:1) 0.103 ms 0.146 ms 0.115 ms 2 fd99:a:b:98:67::1 (fd99:a:b:98:67::1) 3.552 ms 3.555 ms 3.306 ms 3 2a00:6340:1000:2024::2 (2a00:6340:1000:2024::2) 7.566 ms 6.779 ms 6.660 ms 4 2a00:6340:1000:2024::1 (2a00:6340:1000:2024::1) 6.857 ms 6.779 ms 6.559 ms 5 C35362-410.C35362-413.telegram.org (2001:7f8::f259:0:2) 17.739 ms 13.356 ms 15.435 ms 6 2001:67c:4e8:801a:1:9102:1:1 (2001:67c:4e8:801a:1:9102:1:1) 13.292 ms 15.342 ms 13.189 ms 7 2001:67c:4e8:801a:1:801e:1:2 (2001:67c:4e8:801a:1:801e:1:2) 15.329 ms 13.406 ms 13.152 ms 8 2001:67c:4e8:eeee:801e:4:0:2632 (2001:67c:4e8:eeee:801e:4:0:2632) 13.457 ms 13.291 ms 13.314 ms 9 2001:67c:4e8:f004::9 (2001:67c:4e8:f004::9) 12.911 ms [open] 12.881 ms 13.014 ms

and for port 443

root@zone-s:/usr/local/bin# tcptraceroute6 api.telegram.org 443 traceroute to api.telegram.org (2001:67c:4e8:f004::9) from 2001:67c:1298::ff04, port 443, from port 16883, 30 hops max, 60 bytes packets 1 2a01:4f8:120:1104::70:1 (2a01:4f8:120:1104::70:1) 0.119 ms 0.159 ms 0.213 ms 2 fd99:a:b:98:67::1 (fd99:a:b:98:67::1) 3.155 ms 3.157 ms 2.949 ms 3 2a00:6340:1000:2024::2 (2a00:6340:1000:2024::2) 6.859 ms 6.714 ms 6.757 ms 4 * * * 5 * * * 6 C35362-410.C35362-413.telegram.org (2001:7f8::f259:0:2) 13.879 ms 14.173 ms 13.704 ms 7 2001:67c:4e8:801a:1:9102:1:1 (2001:67c:4e8:801a:1:9102:1:1) 13.683 ms 13.744 ms 13.570 ms 8 2001:67c:4e8:801a:1:801e:1:2 (2001:67c:4e8:801a:1:801e:1:2) 13.531 ms 13.381 ms 13.570 ms 9 2001:67c:4e8:eeee:801e:4:0:2632 (2001:67c:4e8:eeee:801e:4:0:2632) 13.501 ms 13.552 ms 13.301 ms 10 2001:67c:4e8:f004::9 (2001:67c:4e8:f004::9) 12.885 ms [open] 12.742 ms 12.926 ms

In verbose mode `* Host api.telegram.org:443 was resolved.

Nothing more

So like icmpv6, tcp6 seems OK but still in error. Banging my head :)

levlam commented 6 months ago

Could you also try mtr --tcp -P 443?

levlam commented 6 months ago

Could you also try from some other IP address from the 2001:67c:1298::0/48 network other than 2001:67c:1298::ff04?

tootai commented 6 months ago

Same with all 2001:67c:1298::/64 range (I use this /64 and 2001:67c:1298:10::/64 from this /48 range)

root@zone-s:/tmp# mtr --tcp -P 443 --report-wide api.telegram.org Start: 2024-03-20T15:39:21+0100 [4/1258] HOST: zone-s Loss% Snt Last Avg Best Wrst StDev 1.|-- 2a01:4f8:120:1104::70:1 0.0% 10 0.4 0.3 0.2 0.5 0.1 2.|-- fd99:a:b:98:67::1 0.0% 10 3.2 3.4 3.2 4.8 0.5 3.|-- 2a00:6340:1000:2024::2 0.0% 10 6.8 7.0 6.7 7.7 0.3 4.|-- 2a00:6340:1000:2024::1 50.0% 10 7.2 7.5 7.2 8.3 0.5 5.|-- C35362-410.C35362-413.telegram.org 60.0% 10 14.5 17.6 13.7 27.9 6.9 6.|-- C35362-410.C35362-413.telegram.org 0.0% 10 15.2 14.7 13.6 19.2 1.7 2001:67c:4e8:801a:1:9102:1:1
2001:67c:4e8:7003:1:9102:1:1
2001:67c:4e8:801b:1:9102:1:1
7.|-- 2001:67c:4e8:801a:1:801e:1:2 0.0% 10 14.3 14.0 13.7 14.3 0.2 2001:67c:4e8:801a:1:9102:1:1
2001:67c:4e8:7003:1:9102:1:1
2001:67c:4e8:801b:1:801f:1:2
2001:67c:4e8:801a:1:801f:1:2
2001:67c:4e8:7003:1:801e:1:2
2001:67c:4e8:801b:1:9102:1:1
8.|-- 2001:67c:4e8:eeee:801f:4:0:2631 0.0% 10 13.8 13.8 13.5 14.1 0.2 2001:67c:4e8:eeee:801e:4:0:2632
2001:67c:4e8:7003:1:801e:1:2
2001:67c:4e8:eeee:801e:4:0:2634
2001:67c:4e8:eeee:801f:4:0:2634
2001:67c:4e8:801a:1:801f:1:2
2001:67c:4e8:eeee:801f:4:0:2633
2001:67c:4e8:801b:1:801e:1:2
2001:67c:4e8:801b:1:801f:1:2
9.|-- 2001:67c:4e8:eeee:801f:4:0:2633 0.0% 10 13.8 13.8 13.4 14.1 0.2 2001:67c:4e8:eeee:801f:4:0:2635
2001:67c:4e8:eeee:801e:4:0:2636
2001:67c:4e8:eeee:801e:4:0:2631
2001:67c:4e8:eeee:801f:4:0:2631
2001:67c:4e8:eeee:801e:4:0:2634 2001:67c:4e8:f004::9
2001:67c:4e8:eeee:801f:4:0:2634
10.|-- 2001:67c:4e8:f004::9 0.0% 8 13.7 13.6 13.3 14.2 0.3

I downloaded with curl from various repos including debian-netinstall.iso, everything like expected.

tootai commented 6 months ago

First packets output of tcpdump from failing IP:

11:33:41.430289` ens1 Out IP6 2001:67c:1298::ff04.60280 > 2001:67c:4e8:f004::9.https: Flags [S], seq 2294335465, win 64800, options [mss 1440,sackOK,TS val 2421837041 ecr 0,nop,wscale 7], length 0

11:33:41.443297 ens1 In IP6 2001:67c:4e8:f004::9.https > 2001:67c:1298::ff04.60280: Flags [S.], seq 3879809308, ack 2294335466, win 28560, options [mss 1440,sackOK,TS val 3435357583 ecr 2421837041,nop,wscale 10], length 0 11:33:41.443327 ens1 Out IP6 2001:67c:1298::ff04.60280 > 2001:67c:4e8:f004::9.https: Flags [.], ack 1, win 507, options [nop,nop,TS val 2421837054 ecr 3435357583], length 0 11:33:41.446243 ens1 Out IP6 2001:67c:1298::ff04.60280 > 2001:67c:4e8:f004::9.https: Flags [P.], seq 1:518, ack 1, win 507, options [nop,nop,TS val 2421837057 ecr 3435357583], length 517 11:33:41.459314 ens1 In IP6 2001:67c:4e8:f004::9.https > 2001:67c:1298::ff04.60280: Flags [.], ack 518, win 29, options [nop,nop,TS val 3435357587 ecr 2421837057], length 0 11:33:41.495913 ens1 In IP6 2001:67c:4e8:f004::9.https > 2001:67c:1298::ff04.60280: Flags [P.], seq 5525:5698, ack 518, win 29, options [nop,nop,TS val 3435357597 ecr 2421837057], length 173 11:33:41.495932 ens1 Out IP6 2001:67c:1298::ff04.60280 > 2001:67c:4e8:f004::9.https: Flags [.], ack 1, win 507, options [nop,nop,TS val 2421837107 ecr 3435357587,nop,nop,sack 1 {5525:5698}], length 0

As you can notice, the before last line is a seq:5525:5698, ack 518 followed by a new ack1. Now the output of a working IP:

11:43:41.211636 ens1 Out IP6 2a01:4f8:120:1104::70:12.49804 > 2001:67c:4e8:f004::9.443: Flags [S], seq 3848103319, win 64800, options [mss 1440,sackOK,TS val 2764072909 ecr 0,nop,wscale 7], length 0
11:43:41.222328 ens1 In IP6 2001:67c:4e8:f004::9.443 > 2a01:4f8:120:1104::70:12.49804: Flags [S.], seq 3343337882, ack 3848103320, win 28560, options [mss 1440,sackOK,TS val 4200978678 ecr 2764072909,nop,wscale 10], length 0
11:43:41.222377 ens1 Out IP6 2a01:4f8:120:1104::70:12.49804 > 2001:67c:4e8:f004::9.443: Flags [.], ack 1, win 507, options [nop,nop,TS val 2764072919 ecr 4200978678], length 0
11:43:41.225508 ens1 Out IP6 2a01:4f8:120:1104::70:12.49804 > 2001:67c:4e8:f004::9.443: Flags [P.], seq 1:518, ack 1, win 507, options [nop,nop,TS val 2764072922 ecr 4200978678], length 517
11:43:41.236092 ens1 In IP6 2001:67c:4e8:f004::9.443 > 2a01:4f8:120:1104::70:12.49804: Flags [.], ack 518, win 29, options [nop,nop,TS val 4200978681 ecr 2764072922], length 0
11:43:41.237072 ens1 In IP6 2001:67c:4e8:f004::9.443 > 2a01:4f8:120:1104::70:12.49804: Flags [.], seq 1:1429, ack 518, win 29, options [nop,nop,TS val 4200978682 ecr 2764072922], length 1428
11:43:41.237082 ens1 Out IP6 2a01:4f8:120:1104::70:12.49804 > 2001:67c:4e8:f004::9.443: Flags [.], ack 1429, win 501, options [nop,nop,TS val 2764072934 ecr 4200978682], length 0

Here the before last line is seq1:1429 followed by the ack 1429 line which does the sequences continuing as they should.

Question now is why in the first case I receive a false seq number with P flag ?

levlam commented 6 months ago

Thank you for the additional information.

Could you run curl --resolve api.telegram.org:443:[2001:67c:4e8:f004::8] https://api.telegram.org/bot123456789:AAAAAAAA/getme, collect a pcap file for the request using tcpdump, and send the file to https://t.me/tdlib_bot?

tootai commented 6 months ago

root@zone-s:~# curl --resolve api.telegram.org:443:[2001:67c:4e8:f004::8] https://api.telegram.org/bot123456789:AAAAAAAA/getme {"ok":false,"error_code":401,"description":"Unauthorized"}root@zone-s:~#

levlam commented 6 months ago

It's really strange that you are able to receive the answer from api.telegram.org through another IPv6 address, but not through the default one. This could mean that someone in the middle intentionally blocks HTTPS to the specififc IPv6 address. Also, it can be seen from tcpdump that TCP connection was established and is dropped some time after it was established, hence this isn't a network connectivity issue.

tootai commented 6 months ago

Do you really meant ipv6 2001:67c:4e8:f004::8 and not ::9 ?

Output in verbose mode (tried with both IPs)

tootai commented 6 months ago

I have to add that against ipv6 ending with ::8 I get the same result -above one- with both IPs. If I do the test against ::9, only the "good" ipv6 shows the same result, the failing one giving

curl -v --resolve api.telegram.org:443:[2001:67c:4e8:f004::9] https://api.telegram.org/bot123456789:AAAAAAAA/getme

That's all. To summarize:

curl --resolve api.telegram.org:443:[2001:67c:4e8:f004::8] https://api.telegram.org/bot123456789:AAAAAAAA/getme {"ok":false,"error_code":401,"description":"Unauthorized"}

Gives the same result with both IPs as

curl --resolve api.telegram.org:443:[2001:67c:4e8:f004::9] https://api.telegram.org/bot123456789:AAAAAAAA/getme {"ok":false,"error_code":401,"description":"Unauthorized"}

fail on IP src 2001:67c:1298::ff04

levlam commented 6 months ago

Do you really meant ipv6 2001:67c:4e8:f004::8 and not ::9 ?

Yes. We wanted to test with the same server with a different IPv6 address.

tootai commented 6 months ago

So as I get same result with my both ipv6 IPs against ::8 it means that something is wrong with ::9

tootai commented 6 months ago

Up ?

levlam commented 6 months ago

Did you receive response from the peering partner?

tootai commented 6 months ago

They say they are not involved, which seems to be right for me. Connecting to ::8 is OK to ::9 not, so they are in the pass on both IPs and don't fail with ::8

levlam commented 6 months ago

::8 has connected you with a server that is available with ::9, so there are no issues with TCP, IPv6, or Telegram servers. TCP with ::9 also works, so there is a specific issue with using HTTPS from your server to ::9.

Commifreak commented 5 months ago

I have the same issue for the telegram API when using IPv6! Sometimes its just working but most of time its not. v4 always fine.

curl -6 -vvv -S -s -q -X POST "https://api.telegram.org/bot${TOKEN}/sendMessage"
*   Trying 2001:67c:4e8:f004::9:443...
* Connected to api.telegram.org (2001:67c:4e8:f004::9) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: Connection reset by peer in connection to api.telegram.org:443
* Closing connection 0
* TLSv1.0 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, decode error (562):
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to api.telegram.org:443

Unbenannt

Any other IPv6 server with curl tests just fine and always works.

around84 commented 2 months ago

I fully confirm that there is still a bug:

curl -6 -vvv -S -s -q -X POST "https://api.telegram.org/bot${TOKEN}/sendMessage"
*   Trying [2001:67c:4e8:f004::9]:443...
* Connected to api.telegram.org (2001:67c:4e8:f004::9) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* Recv failure: Соединение разорвано другой стороной
* OpenSSL SSL_connect: Соединение разорвано другой стороной in connection to api.telegram.org:443
* Closing connection 0
curl: (35) Recv failure: Соединение разорвано другой стороной
levlam commented 2 months ago

@around84 This is still looks like an Internet connectivity issue, which happens much more often with IPv6 networks, or a block by government/Internet provider.

psykokwak-com commented 1 month ago

Got this issue too.

xniksysx commented 1 month ago

What if try to set MTU lower then 1500 on ipv6 interface?

xniksysx commented 1 month ago

image if i dont set in Mikrotik ipv6 ND MTU then i cant connect to any in *.telegram.org. Check on 2 ISP in Russia. ND MTU set as PPPoE MTU.

tootai commented 4 weeks ago

No changes on my side. Link to pcap capture https://tdrive.tootai.net/s/GNGzJWr7NQwdkMX