docker run -ti --rm oracle/instantclient:18.3.0 --network api_overlay sqlplus "<login>/<password>@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(Host=<ip>)(Port=1521))(CONNECT_DATA=(SID=<sid>)))"
hangs. Changing mtu doesn't help.
tcpdump of successful connection:
sudo tcpdump -i vpn0 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vpn0, link-type RAW (Raw IP), capture size 262144 bytes
12:15:10.087467 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [S], seq 3956465851, win 21200, options [mss 1060,sackOK,TS val 408789 ecr 0,nop,wscale 7], length 0
12:15:10.128225 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [S.], seq 2090041496, ack 3956465852, win 14480, options [mss 1160,sackOK,TS val 2910434587 ecr 408789,nop,wscale 7], length 0
12:15:10.128299 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [.], ack 1, win 166, options [nop,nop,TS val 408799 ecr 2910434587], length 0
12:15:10.128539 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 1:217, ack 1, win 166, options [nop,nop,TS val 408800 ecr 2910434587], length 216
12:15:10.167537 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [.], ack 217, win 122, options [nop,nop,TS val 2910434626 ecr 408800], length 0
12:15:10.185461 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 1:9, ack 217, win 122, options [nop,nop,TS val 2910434644 ecr 408800], length 8
12:15:10.185566 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [.], ack 9, win 166, options [nop,nop,TS val 408814 ecr 2910434644], length 0
12:15:10.185614 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 217:433, ack 9, win 166, options [nop,nop,TS val 408814 ecr 2910434644], length 216
12:15:10.224829 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 9:50, ack 433, win 130, options [nop,nop,TS val 2910434684 ecr 408814], length 41
12:15:10.224999 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 433:592, ack 50, win 166, options [nop,nop,TS val 408824 ecr 2910434684], length 159
12:15:10.264401 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 50:177, ack 592, win 139, options [nop,nop,TS val 2910434723 ecr 408824], length 127
12:15:10.268475 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 592:626, ack 177, win 166, options [nop,nop,TS val 408834 ecr 2910434723], length 34
12:15:10.307915 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 177:366, ack 626, win 139, options [nop,nop,TS val 2910434767 ecr 408834], length 189
12:15:10.308490 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 626:710, ack 366, win 174, options [nop,nop,TS val 408844 ecr 2910434767], length 84
12:15:10.348210 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 366:392, ack 710, win 139, options [nop,nop,TS val 2910434807 ecr 408844], length 26
12:15:10.349603 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 710:956, ack 392, win 174, options [nop,nop,TS val 408855 ecr 2910434807], length 246
12:15:10.391478 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 392:880, ack 956, win 147, options [nop,nop,TS val 2910434850 ecr 408855], length 488
12:15:10.392187 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 956:1966, ack 880, win 183, options [nop,nop,TS val 408865 ecr 2910434850], length 1010
12:15:10.440238 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [.], seq 880:1928, ack 1966, win 163, options [nop,nop,TS val 2910434898 ecr 408865], length 1048
12:15:10.440295 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [.], seq 1928:2976, ack 1966, win 163, options [nop,nop,TS val 2910434898 ecr 408865], length 1048
12:15:10.440330 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 2976:3051, ack 1966, win 163, options [nop,nop,TS val 2910434898 ecr 408865], length 75
12:15:10.440372 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [.], ack 2976, win 216, options [nop,nop,TS val 408877 ecr 2910434898], length 0
12:15:10.440854 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 1966:2026, ack 3051, win 216, options [nop,nop,TS val 408878 ecr 2910434898], length 60
12:15:10.479335 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 2976:3051, ack 1966, win 163, options [nop,nop,TS val 2910434938 ecr 408877], length 75
12:15:10.479454 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [.], ack 3051, win 216, options [nop,nop,TS val 408887 ecr 2910434938,nop,nop,sack 1 {2976:3051}], length 0
12:15:10.480049 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 3051:3237, ack 2026, win 163, options [nop,nop,TS val 2910434939 ecr 408878], length 186
12:15:10.480120 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [.], ack 3237, win 232, options [nop,nop,TS val 408887 ecr 2910434939], length 0
12:15:10.480277 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 2026:2377, ack 3237, win 232, options [nop,nop,TS val 408887 ecr 2910434939], length 351
12:15:10.520270 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 3237:3680, ack 2377, win 179, options [nop,nop,TS val 2910434979 ecr 408887], length 443
12:15:10.520574 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 2377:2398, ack 3680, win 248, options [nop,nop,TS val 408898 ecr 2910434979], length 21
12:15:10.559853 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 3680:3860, ack 2398, win 179, options [nop,nop,TS val 2910435019 ecr 408898], length 180
12:15:10.560150 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 2398:2710, ack 3860, win 265, options [nop,nop,TS val 408907 ecr 2910435019], length 312
12:15:10.600069 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 3860:4049, ack 2710, win 194, options [nop,nop,TS val 2910435059 ecr 408907], length 189
12:15:10.600313 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 2710:3145, ack 4049, win 281, options [nop,nop,TS val 408917 ecr 2910435059], length 435
12:15:10.640784 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 4049:4690, ack 3145, win 210, options [nop,nop,TS val 2910435099 ecr 408917], length 641
12:15:10.641094 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 3145:3603, ack 4690, win 297, options [nop,nop,TS val 408928 ecr 2910435099], length 458
12:15:10.681713 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 4690:5046, ack 3603, win 226, options [nop,nop,TS val 2910435140 ecr 408928], length 356
12:15:10.681999 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 3603:3989, ack 5046, win 314, options [nop,nop,TS val 408938 ecr 2910435140], length 386
12:15:10.722164 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 5046:5286, ack 3989, win 242, options [nop,nop,TS val 2910435181 ecr 408938], length 240
12:15:10.722406 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 3989:4355, ack 5286, win 330, options [nop,nop,TS val 408948 ecr 2910435181], length 366
12:15:10.762352 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 5286:5708, ack 4355, win 258, options [nop,nop,TS val 2910435221 ecr 408948], length 422
12:15:10.762606 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 4355:4391, ack 5708, win 347, options [nop,nop,TS val 408958 ecr 2910435221], length 36
12:15:10.801942 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 5708:5725, ack 4391, win 258, options [nop,nop,TS val 2910435261 ecr 408958], length 17
12:15:10.802095 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 4391:4404, ack 5725, win 347, options [nop,nop,TS val 408968 ecr 2910435261], length 13
12:15:10.841247 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 5725:5742, ack 4404, win 258, options [nop,nop,TS val 2910435300 ecr 408968], length 17
12:15:10.884508 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [.], ack 5742, win 347, options [nop,nop,TS val 408989 ecr 2910435300], length 0
^C
46 packets captured
46 packets received by filter
0 packets dropped by kernel
when stuck:
sudo tcpdump -i vpn0 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vpn0, link-type RAW (Raw IP), capture size 262144 bytes
12:16:11.422616 IP 172.30.2.157.52186 > 10.4.0.32.1521: Flags [S], seq 1453371367, win 29200, options [mss 1460,sackOK,TS val 424123 ecr 0,nop,wscale 7], length 0
12:16:11.463151 IP 10.4.0.32.1521 > 172.30.2.157.52186: Flags [S.], seq 2123913164, ack 1453371368, win 14480, options [mss 1160,sackOK,TS val 2910495922 ecr 424123,nop,wscale 7], length 0
12:16:11.463253 IP 172.30.2.157.52186 > 10.4.0.32.1521: Flags [.], ack 1, win 229, options [nop,nop,TS val 424133 ecr 2910495922], length 0
12:16:11.463476 IP 172.30.2.157.52186 > 10.4.0.32.1521: Flags [P.], seq 1:217, ack 1, win 229, options [nop,nop,TS val 424133 ecr 2910495922], length 216
12:16:11.502416 IP 10.4.0.32.1521 > 172.30.2.157.52186: Flags [.], ack 217, win 122, options [nop,nop,TS val 2910495961 ecr 424133], length 0
12:16:11.523656 IP 10.4.0.32.1521 > 172.30.2.157.52186: Flags [P.], seq 1:9, ack 217, win 122, options [nop,nop,TS val 2910495982 ecr 424133], length 8
12:16:11.523740 IP 172.30.2.157.52186 > 10.4.0.32.1521: Flags [.], ack 9, win 229, options [nop,nop,TS val 424148 ecr 2910495982], length 0
12:16:11.523783 IP 172.30.2.157.52186 > 10.4.0.32.1521: Flags [P.], seq 217:433, ack 9, win 229, options [nop,nop,TS val 424148 ecr 2910495982], length 216
12:16:11.563449 IP 10.4.0.32.1521 > 172.30.2.157.52186: Flags [P.], seq 9:50, ack 433, win 130, options [nop,nop,TS val 2910496022 ecr 424148], length 41
12:16:11.563639 IP 172.30.2.157.52186 > 10.4.0.32.1521: Flags [P.], seq 433:592, ack 50, win 229, options [nop,nop,TS val 424158 ecr 2910496022], length 159
12:16:11.603058 IP 10.4.0.32.1521 > 172.30.2.157.52186: Flags [P.], seq 50:177, ack 592, win 139, options [nop,nop,TS val 2910496062 ecr 424158], length 127
12:16:11.607200 IP 172.30.2.157.52186 > 10.4.0.32.1521: Flags [P.], seq 592:626, ack 177, win 229, options [nop,nop,TS val 424169 ecr 2910496062], length 34
12:16:11.646395 IP 10.4.0.32.1521 > 172.30.2.157.52186: Flags [P.], seq 177:366, ack 626, win 139, options [nop,nop,TS val 2910496105 ecr 424169], length 189
12:16:11.646864 IP 172.30.2.157.52186 > 10.4.0.32.1521: Flags [P.], seq 626:710, ack 366, win 237, options [nop,nop,TS val 424179 ecr 2910496105], length 84
12:16:11.686621 IP 10.4.0.32.1521 > 172.30.2.157.52186: Flags [P.], seq 366:392, ack 710, win 139, options [nop,nop,TS val 2910496145 ecr 424179], length 26
12:16:11.687757 IP 172.30.2.157.52186 > 10.4.0.32.1521: Flags [P.], seq 710:956, ack 392, win 237, options [nop,nop,TS val 424189 ecr 2910496145], length 246
12:16:11.729563 IP 10.4.0.32.1521 > 172.30.2.157.52186: Flags [P.], seq 392:880, ack 956, win 147, options [nop,nop,TS val 2910496188 ecr 424189], length 488
12:16:11.730214 IP 172.30.2.157.52186 > 10.4.0.32.1521: Flags [P.], seq 956:1966, ack 880, win 245, options [nop,nop,TS val 424200 ecr 2910496188], length 1010
12:16:11.777368 IP truncated-ip - 6 bytes missing! 10.4.0.32.1521 > 172.30.2.157.52186: Flags [.], seq 880:2028, ack 1966, win 163, options [nop,nop,TS val 2910496235 ecr 424200], length 1148
12:16:11.777411 IP 10.4.0.32.1521 > 172.30.2.157.52186: Flags [P.], seq 2028:3051, ack 1966, win 163, options [nop,nop,TS val 2910496236 ecr 424200], length 1023
12:16:11.777934 IP 172.30.2.157.52186 > 10.4.0.32.1521: Flags [.], ack 880, win 261, options [nop,nop,TS val 424212 ecr 2910496188,nop,nop,sack 1 {2028:3051}], length 0
12:16:12.019546 IP truncated-ip - 6 bytes missing! 10.4.0.32.1521 > 172.30.2.157.52186: Flags [.], seq 880:2028, ack 1966, win 163, options [nop,nop,TS val 2910496478 ecr 424212], length 1148
12:16:12.503770 IP truncated-ip - 6 bytes missing! 10.4.0.32.1521 > 172.30.2.157.52186: Flags [.], seq 880:2028, ack 1966, win 163, options [nop,nop,TS val 2910496962 ecr 424212], length 1148
12:16:13.471513 IP truncated-ip - 6 bytes missing! 10.4.0.32.1521 > 172.30.2.157.52186: Flags [.], seq 880:2028, ack 1966, win 163, options [nop,nop,TS val 2910497930 ecr 424212], length 1148
12:16:15.407660 IP truncated-ip - 6 bytes missing! 10.4.0.32.1521 > 172.30.2.157.52186: Flags [.], seq 880:2028, ack 1966, win 163, options [nop,nop,TS val 2910499866 ecr 424212], length 1148
12:16:19.279620 IP truncated-ip - 6 bytes missing! 10.4.0.32.1521 > 172.30.2.157.52186: Flags [.], seq 880:2028, ack 1966, win 163, options [nop,nop,TS val 2910503738 ecr 424212], length 1148
^C
26 packets captured
26 packets received by filter
0 packets dropped by kernel
6 packets dropped by interface
Please help. I can see "IP truncated-ip - 6 bytes missing!" and same incoming packet from server when connection hangs.
And first packet from client suggest mss 1460, ignoring --opt com.docker.network.driver.mtu=1100
in case of overlay. When in bridged mode everything is ok.
Output of docker version:
Client:
Version: 18.09.2
API version: 1.39
Go version: go1.10.6
Git commit: 6247962
Built: Sun Feb 10 04:13:52 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.2
API version: 1.39 (minimum version 1.12)
Go version: go1.10.6
Git commit: 6247962
Built: Sun Feb 10 03:42:13 2019
OS/Arch: linux/amd64
Experimental: false
$ip a show dev vpn0
19: vpn0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1194 qdisc pfifo_fast state UP group default qlen 500
link/none
inet 172.30.2.157/24 brd 172.30.2.255 scope global vpn0
valid_lft forever preferred_lft forever
Expected behavior
I can connect to Oracle database over Cisco anyconnect from docker image with network driver overlay
Actual behavior
Connection hangs because of MTU
Steps to reproduce the behavior
$docker build -t oracle/instantclient:18.3.0 .
$docker run -ti --rm oracle/instantclient:18.3.0 sqlplus "<login>/<password>@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(Host=<ip>)(Port=1521))(CONNECT_DATA=(SID=<sid>)))"
works. same command with network
docker run -ti --rm oracle/instantclient:18.3.0 --network api_bridge sqlplus "<login>/<password>@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(Host=<ip>)(Port=1521))(CONNECT_DATA=(SID=<sid>)))"
works. Without setting mtu do driver hangs. If i try to connect with overlay
docker run -ti --rm oracle/instantclient:18.3.0 --network api_overlay sqlplus "<login>/<password>@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(Host=<ip>)(Port=1521))(CONNECT_DATA=(SID=<sid>)))"
hangs. Changing mtu doesn't help.
tcpdump of successful connection:
when stuck:
Please help. I can see "IP truncated-ip - 6 bytes missing!" and same incoming packet from server when connection hangs. And first packet from client suggest mss 1460, ignoring
--opt com.docker.network.driver.mtu=1100
in case of overlay. When in bridged mode everything is ok.Output of
docker version
:Output of
docker info
:Additional environment details (AWS, VirtualBox, physical, etc.)