seega
commented
6 years ago @coolermister where server located? ips from https://core.telegram.org/getProxyConfig available?
Open coolermister opened 6 years ago
seega
commented
6 years ago @coolermister where server located? ips from https://core.telegram.org/getProxyConfig available?
p1ratrulezzz
commented
6 years ago Btw, I the same issue was for me. It starts, shows good logs but I can't connect at all. Doesn't work with docker either. And I noticed, that it doesnt work only on servers located in Russia (SimpleCloud). For Hetzner and even for OVH (which now blocks telegram servers btw) it was OK!. I even tried to use it within VPN - no profit at all.
And as for https://core.telegram.org/getProxyConfig - it was ok for all servers (even without vpn)
for author of this subject: use Hetzner cloud, I can prove that it works OK.
coolermister
commented
6 years ago @seega server is located in Belarus, so it's avaliable. I did everything as it is written in readme file, and all seems to be ok. @p1ratrulezzz all these is really strange) did you compare results of getProxyConfig? may be they answer different subnets for different countries?
p1ratrulezzz
commented
6 years ago Hm, no, didnt compare. But i tried to copy them from one server to another, anyway no result with it. Also possible that there is a dependency on virtualization and server processor. But in both cases it was KVM I believe. And port was open actually, but telegram could not connect to it.
coolermister
commented
6 years ago hm. Interesting theory. And made sense, because they use asm insertions. I use vmware esxi 6.5. Tomorrow i will try to compile it on native linux machine
p1ratrulezzz
commented
6 years ago There is a docker image also. Maybe also you could try this setup PC with vmware esxi --> router (vpn client-->some other country), I have a theory that this software ignores routing table completely but not sure
coolermister
commented
6 years ago i've tried docker image also. have the same issue as here https://github.com/TelegramMessenger/MTProxy/issues/50 . But, machine with docker, also works on esxi) Esxi server have 2x opteron 6172. I will answer here as I will test this on native linux, with intel) maybe they use some instructions, which amd isn't provide, or as you said, maybe it's virtualization.. but in sources, i found only sse instructions, which is standart. And again, they use asm includes, and i don't know x86_64 asm, only avr)
seega
commented
6 years ago Same problem at docker for windows (intel) with home router
againDDM
commented
6 years ago I ran into a similar problem.
coolermister
commented
6 years ago so, it's the same issue on native linux. i've add -v to command, and i saw same picture as @againDDM.
detailed_log.txt
Example error:
[13484][2018-06-02 17:01:09.006332 local] New connection 10.10.158.230:44914 -> 91.108.4.185:8888
[13484][2018-06-02 17:01:09.006352 local] Closing connection socket 51
[13485][2018-06-02 17:01:09.011161 local] socket 34: disconnected (epoll_ready=2005), cleaning
[13485][2018-06-02 17:01:09.011201 local] Disconnected from RPC Middle-End (fd=34)
[13485][2018-06-02 17:01:09.011218 local] Creating NEW connection to 149.154.162.36:80
[13485][2018-06-02 17:01:09.011353 local] New connection 10.10.158.230:37210 -> 149.154.162.36:80
[13485][2018-06-02 17:01:09.011369 local] Closing connection socket 34
[13484][2018-06-02 17:01:09.012370 local] socket 57: disconnected (epoll_ready=2005), cleaning
[13484][2018-06-02 17:01:09.012390 local] Disconnected from RPC Middle-End (fd=57)
[13484][2018-06-02 17:01:09.012410 local] Creating NEW connection to 149.154.165.109:8888
[13483][2018-06-02 17:01:09.012439 local] socket 14: disconnected (epoll_ready=2005), cleaning
[13483][2018-06-02 17:01:09.^C[pid 13486] [time 1527948069] SIGINT handled.
[13486][2018-06-02 17:01:09.033171 local] Closing connection socket 35
@againDDM can you reduce your huge log?) or may be upload it as file, or something So this proxy does Dos attack on router) it creates connection to telegram servers, the connection is dropped, but it isn't close connection successfully, so it remains in the routing table of router. And all start all over again
coolermister
commented
6 years ago So, i don't know that else i can find out and say about this issue.. Waiting for developers)
Mirraz
commented
6 years ago I have the same issue. In a very verbose log I have:
[425][2018-06-02 18:28:12.043363 local] received mtproto packet of 40 bytes
[425][2018-06-02 18:28:12.043369 local] nowhere to forward user query from connection 11, dropping
[425][2018-06-02 18:28:12.043371 local] free_msg_buffer(40)
[425][2018-06-02 18:28:12.043374 local] ext_rpcs_execute: cannot forward mtproto packet
itsMoji
commented
6 years ago Because of Telegram servers listening on 8888 port ( Check https://core.telegram.org/getProxyConfig )
Make sure port 8888 is open in your firewall (CSF, ip tables, etc...) so your server can connect to Telegram servers
l0rda
commented
6 years ago Same for me. Docker image not working on VPS(Centos 7 x86_64) at vscale.io, but same config is working on dedicated server(Centos 7 x86_64) at hetzner.
It is successfully started, backend servers list is loaded, but client cannot connect, port is open.
coolermister
commented
6 years ago @itsMoji yes, the port is open, and it can connect. And in my country Telegram servers isn't blocked. Log with -vv option: verbose log.txt
portah
commented
6 years ago @coolermister I believe I had the same problem '--nat-info ' param to MTProxy server fixed the issue.
coolermister
commented
6 years ago @portah Well, it partially helped. Now it have different error. When i trying to connect whith in local network.
[2522][2018-06-04 01:12:45.313851 local] Connected to RPC Middle-End (fd=26)
[2525][2018-06-04 01:12:45.315138 local] Connected to RPC Middle-End (fd=26)
[2523][2018-06-04 01:12:45.315510 local] Connected to RPC Middle-End (fd=25)
[2524][2018-06-04 01:12:45.318356 local] Connected to RPC Middle-End (fd=34)
[2524][2018-06-04 01:13:04.738829 local] New connection 10.10.158.43:50945 -> 10.10.158.233:8888
[2523][2018-06-04 01:13:04.738829 local] New connection 10.10.158.43:50946 -> 10.10.158.233:8888
[2523][2018-06-04 01:13:04.739255 local] trying to determine connection type
[2524][2018-06-04 01:13:04.742620 local] New connection 10.10.158.43:50947 -> 10.10.158.233:8888
[2524][2018-06-04 01:13:04.742743 local] trying to determine connection type
[pid 2522] [time 1528063984] received signal 17
[pid 2522] [time 1528063984] received signal 17
[2522][2018-06-04 01:13:05.004845 local] Child 2523 terminated, aborting
[pid 2525] [time 1528063985] SIGTERM handled.
[2525][2018-06-04 01:13:05.005244 local] Terminated by SIGTERM.
[pid 2522] [time 1528063985] received signal 17The same is when i connect using public ip, but in New connection i see router address
coolermister
commented
6 years ago And same error as here https://github.com/TelegramMessenger/MTProxy/issues/50, but ports are open
torkve
commented
6 years ago Same for me. I have server in Hetzner, ports are open, but I cannot connect to proxy. I tried to start docker image and self-compiled build in host system, but neither works.
It just enters some busyloop endlessly reconnecting telegram servers and doesn't work.
Specifying --address or --nat-info changed nothing.
p1ratrulezzz
commented
6 years ago @torkve, as for me on hetzner it works fine ( i use cloud server in finland). Also on a dedicated server with i7-980 cpu it works great. Check my fork for this repo it has patch for binding ip address ( clone sources and make the same way) And try running it using --address 11.22.33.44 - with your address from ifconfig
torkve
commented
6 years ago @p1ratrulezzz I use dedicated server on hetzner. Tried your fork, but still no luck with or without --address and/or --nat-info.
However I was able to start proxy on two other servers, one of which is connecting telegram through the VPN over that very hetzner server. I didn't check if the proxy works well on them, but at least they successfully establish the connection and do not fall into busyloop.
The commands I used to start proxy:
./mtproto-proxy -v -p 2398 -H 7958 -M 1 -C 60000 --aes-pwd aes backend.conf --allow-skip-dh --address $HETZNER_IP -S $SECRET./mtproto-proxy -v -p 2398 -H 7958 -M 1 -C 60000 backend.conf --aes-pwd aes --address $TUN_IP --nat-info $TUN_IP:$HETZNER_IP --allow-skip-dh -S $SECRET./mtproto-proxy -v -p 2398 -H 7958 -M 1 -C 60000 backend.conf --aes-pwd aes --allow-skip-dh -S $SECRETp1ratrulezzz
commented
6 years ago Strange issue. So looks like it depends on some processor instruction somewhere and works only on some virtualizations. What is your cpu on VDS?
вт, 5 июн. 2018 г., 0:06 Vsevolod notifications@github.com:
@p1ratrulezzz https://github.com/p1ratrulezzz I use dedicated server on hetzner. Tried your fork, but still no luck with or without --address and/or --nat-info. However I was able to start proxy on two other servers, one of which is connecting telegram through the VPN over that very hetzner server. I didn't check if the proxy works well on them, but at least they successfully establish the connection and do not fall into busyloop. The commands I used to start proxy:
- (FAIL) Hetzner: ./mtproto-proxy -v -p 2398 -H 7958 -M 1 -C 60000 --aes-pwd aes backend.conf --allow-skip-dh --address $HETZNER_IP -S $SECRET
- (OK) VPN over hetzner: ./mtproto-proxy -v -p 2398 -H 7958 -M 1 -C 60000 backend.conf --aes-pwd aes --address $TUN_IP --nat-info $TUN_IP:$HETZNER_IP --allow-skip-dh -S $SECRET
- (OK) other server: ./mtproto-proxy -v -p 2398 -H 7958 -M 1 -C 60000 backend.conf --aes-pwd aes --allow-skip-dh -S $SECRET
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/TelegramMessenger/MTProxy/issues/56#issuecomment-394499377, or mute the thread https://github.com/notifications/unsubscribe-auth/AEyYFegHYHkESnDD033KvrtSb0wNGchNks5t5aFAgaJpZM4UXS_y .
torkve
commented
6 years ago Intel Core i7-950
shadowlmd
commented
6 years ago Same issue with conntrack here. Looks like mtproxy makes thousands of outgoing connections and for some reason they end up in TIME_WAIT state in conntrack filling it up until server cannot make new connections at all.
mgdz
commented
6 years ago worked on: model name : Intel(R) Xeon(R) CPU L5640 @ 2.27GHz (vmware) arubacloud not worked on: model name : Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz (kvm) vscale.io
p1ratrulezzz
commented
6 years ago @torkve, very strange, I have i7-980 and it works fine. Maybe need to check for some differences in instruction sets... Just buy a cloud server on hetzner (you can test for 1 hour and then remove the server and pay only for 1h)
torkve
commented
6 years ago I first assumed that it could be somehow related to asm intrinsics, but as far as I see, asm is used (except for memory barriers) only in crc32 calculation, and in case of error in crc32 calculation mtproxy would print corresponding error message.
So I tried to inspect full log. It is not very meaningful, but now I have an impression that state machine can be somehow broken or just race condition encountered. Here is a history of one fd for example:
[32709][2018-06-05 12:34:41.783546 local] Creating NEW connection to 149.154.165.250:8888
[32709][2018-06-05 12:34:41.783561 local] <59 send buffer was 16384, now 16777216
[32709][2018-06-05 12:34:41.783586 local] >59 receive buffer was 87380, now 16777216
[32709][2018-06-05 12:34:41.783613 local] <59 send buffer was 425984, now 16777216
[32709][2018-06-05 12:34:41.783623 local] >59 receive buffer was 425984, now 16777216
[32709][2018-06-05 12:34:41.783651 local] New connection 176.9.x.x:37772 -> 149.154.165.250:8888 (fd 59)
[32709][2018-06-05 12:34:41.783662 local] rpcc_init_outbound (59)
[32709][2018-06-05 12:34:41.783667 local] tcp_rpcc_default_check_perm(59): [176.9.x.x]:37772 -> [149.154.165.250]:8888
[32709][2018-06-05 12:34:41.783678 local] epoll_mod(6,0x00000007,59,59,8000200f)
[32709][2018-06-05 12:34:41.799705 local] fd=59 state=39 ready=1026 epoll_ready=4
[32709][2018-06-05 12:34:41.799732 local] END processing connection 59, flags=33554433
[32709][2018-06-05 12:34:41.799743 local] readv from 59: -1 read out of 262144
[32709][2018-06-05 12:34:41.799823 local] tcp_rpcc_default_check_perm(59): [176.9.x.x]:37772 -> [149.154.165.250]:8888
[32709][2018-06-05 12:34:41.799827 local] RPC connection #59: [176.9.x.x]:37772 -> [149.154.165.250]:8888 connected, crypto_flags = 10
[32709][2018-06-05 12:34:41.799837 local] tcp_rpc_conn_send: sending message of size 32 to conn fd=59
[32709][2018-06-05 12:34:41.800072 local] END processing connection 59, flags=33554449
[32709][2018-06-05 12:34:41.800084 local] send/writev() to 59: 44 written out of 44 in 1 chunks
[32709][2018-06-05 12:34:41.800090 local] socket_server_writer: written 44 bytes to 59, flags=0x02000011So the connection is being stopped, read fails, and then it is marked as connected and starts sending data.
P.s. I see no reason to test cloud server, since you confirm that everything works in your hetzner case.
p1ratrulezzz
commented
6 years ago https://github.com/TelegramMessenger/MTProxy/blob/cc7b7097a5bc1681d0f4c05d5f07576c8b279fc4/mtproto/mtproto-proxy.c line 1740 - rpc_target_choose_random_connections doesnt work I guess in our cases
torkve
commented
6 years ago It is code for forwarding packet from client connection, as far as I see. In my case there're no client connections yet (not even attempts). All the connections, including the one in log, are attempts to establish connection from proxy to telegram servers.
coolermister
commented
6 years ago I wonder where the developers are? because a large number of people have different problems with this proxy, and accordingly they can not help fight for Internet freedom! P.s. For developers. Guys, thank you for your work, I'm sure everyone here appreciates it. We will provide you with all information that you need, in order to fix bugs) Just don't forget about us
coolermister
commented
6 years ago Guys, can the problem be in the cpu instructions of the AES? For example, my CPU's don't have hardware aes i7-950 don't have it But i7-980 have it
torkve
commented
6 years ago Sounds like a good guess to me.
shadowlmd
commented
6 years ago Funny thing, I too have aes in cpu flags on server where the problem occurs. On two other servers where there is no problem I don't have aes in cpu flags.
l0rda
commented
6 years ago Not working on KVM Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl eagerfpu pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
UrsusArctos
commented
6 years ago I had exactly the same issue with rapid multiple disconnections, only this time it was KVM-based VPS with Genoo Linux on it and public routable IP addresses, no NAT and no firewals/filters so the problem was not in any kind of filtering. Moreover, MTProxy worked fine in my another Gentoo box at the same datacenter. I fixed the problem by upgrading the Linux kernel to 4.9.95.
torkve
commented
6 years ago @UrsusArctos your idea rocks. I had kernel 4.11.0-2 on Debian, after upgrade to 4.16.0-2 the problem disappeared. I cannot be absolutely sure that it was kernel, since I upgraded all the packages, but it can be that.
The other two working systems I tested earlier have stock 4.15.0-23/Ubuntu and rather old custom 3.10.69-25/Ubuntu, so I have no idea where the problem lies exactly.
shadowlmd
commented
6 years ago Ok, I can confirm that --nat-info <local_ip>:<global_ip> fixed the issue with conntrack overflow on my server. Server is indeed behind nat and only has local IP addresses on interfaces. During startup mtproxy picks one of them (for some reason, from tun0 instead of eth0) and maybe that was a problem, I am not sure.
The biggest question is why mtproxy requires something like this parameter at all. It is quite easy to figure out the proper network interface and global IP.
Revertron
commented
6 years ago Yep, I can confirm this too.
--nat-info <local_ip>:<global_ip> helps.
Andycar
commented
6 years ago Tried everything before I found this. Yes, --nat-info key really solved it!
abrekhov
commented
6 years ago Same issue, but --nat-info doesn't work. If there any other solutions? Ubuntu 16.06 x 64 server in Russia.
Revertron
commented
6 years ago 19 days of good work, rebooted server and problem arises again, and now --nat-info <local_ip>:<global_ip> doesn't help.
d-makarenko
commented
5 years ago Having same problem with VPS on OpenVZ. All of the suggested settings doesn't help.
shadowlmd
commented
5 years ago @d-makarenko, can you show the command line you are using to start mtproto-proxy and also the output of curl https://bsrealm.net/ip.php and ip a commands?
d-makarenko
commented
5 years ago Command:
./mtproto-proxy -u nobody -p 1500 -H 10234 -S 222343dbdccdef4a54ebab1c6b0a6781 --aes-pwd proxy-secret proxy-multi.conf -M 1External IP:
curl https://bsrealm.net/ip.php
23.94.173.93Interfaces:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: gre0: <NOARP> mtu 1476 qdisc noop state DOWN
link/gre 0.0.0.0 brd 0.0.0.0
3: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
4: venet0: <BROADCAST,POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/void
inet 127.0.0.1/32 scope host venet0
inet 23.94.173.93/32 brd 23.94.173.93 scope global venet0:0
5: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 100
link/none
inet 10.8.0.1/24 brd 10.8.0.255 scope global tun0Ouch! You'll have to change your proxy secret after this. Anyway, I see that your server is not behind NAT so the problem is different. Can you show the output of sudo conntrack -L | wc -l?
d-makarenko
commented
5 years ago Yep, my bad. When I'm starting mtproxy it starts to grow up to 65k and then I'm getting my server suspended by sp.
this is before mtproxy is started
[root@vps MTProxy]# cat /proc/net/nf_conntrack | wc -l
41
and here it is
[root@vps MTProxy]# cat /proc/net/nf_conntrack | wc -l
2647
[root@vps MTProxy]# cat /proc/net/nf_conntrack | wc -l
3676
[root@vps MTProxy]# cat /proc/net/nf_conntrack | wc -l
4559
[root@vps MTProxy]# cat /proc/net/nf_conntrack | wc -l
5320
[root@vps MTProxy]# cat /proc/net/nf_conntrack | wc -l
7176
[root@vps MTProxy]# cat /proc/net/nf_conntrack | wc -l
7174
[root@vps MTProxy]# cat /proc/net/nf_conntrack | wc -lThis is weird. Just wonder, what happens if you add --nat-info 23.94.173.93:23.94.173.93 to parameters?
d-makarenko
commented
5 years ago Yep, already tried that, sadly it didn't help. The strange thing is that it worked before, somewhere around fcb804c, but then it got broken, and even if I build from fcb804c, it still overflow conntrack table.
Other thing that was changed since then is proxy configuration file, but have no older revisions.
shadowlmd
commented
5 years ago Do you see a lot of connections in TIME_WAIT state in conntrack -L output? All of them to Telegram servers and 8888 port?
d-makarenko
commented
5 years ago Yes most of them in TIME_WAIT and to the telegram servers.
Hi guys! First of all sorry for my english) Becouse english isn't my native language) (If someone have interested in this, my native language is russian) I have really strange problem. I compiled the proxy sucsessfully. The server which will run proxy is behind my nat router(i have public static ip of course), and have address 10.10.158.233(my local network is 10.10.158.0/24) 8443 port looks outside. ip of router is 10.10.158.254. So, when i run proxy with command
mtproto-proxy -u nobody -H 8888 -p 8443 -S <here is the secret> -M 3 --aes-pwd proxy-secret proxy-multi.confit succesfully starts.But in secods after it's started, routing table of my router is overflowed. For example: Before
After run:
Limit on router is
net.netfilter.nf_conntrack_max=65536When it's running, i can't connect to it using local ip, and public ip too, from my android client. And after something about minute, router is going down) System is Ubuntu 16.04 x64. So, what i'm gonna do? Which additional information i can provide, to fix it?