search

docker / for-linux

Docker Engine for Linux
https://docs.docker.com/engine/installation/
756 stars 85 forks source link

Setting MTU in overlay #594

Open andribas404 opened 5 years ago

andribas404 commented 5 years ago

Expected behavior

I can connect to Oracle database over Cisco anyconnect from docker image with network driver overlay

Actual behavior

Connection hangs because of MTU

Steps to reproduce the behavior

$docker build -t oracle/instantclient:18.3.0 .

$docker run -ti --rm oracle/instantclient:18.3.0 sqlplus "<login>/<password>@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(Host=<ip>)(Port=1521))(CONNECT_DATA=(SID=<sid>)))"

works. same command with network

$docker network create -d bridge \
  --subnet=172.28.0.0/16 \
  --ip-range=172.28.5.0/24 \
  --attachable \
  --gateway=172.28.5.1 \
  --opt com.docker.network.driver.mtu=1100 \
  api_bridge

docker run -ti --rm oracle/instantclient:18.3.0 --network api_bridge sqlplus "<login>/<password>@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(Host=<ip>)(Port=1521))(CONNECT_DATA=(SID=<sid>)))"

works. Without setting mtu do driver hangs. If i try to connect with overlay

$docker network rm api_bridge
$docker network create -d overlay \
  --subnet=172.28.0.0/16 \
  --ip-range=172.28.5.0/24 \
  --attachable \
  --gateway=172.28.5.1 \
  --opt com.docker.network.driver.mtu=1100 \
  api_overlay

docker run -ti --rm oracle/instantclient:18.3.0 --network api_overlay sqlplus "<login>/<password>@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(Host=<ip>)(Port=1521))(CONNECT_DATA=(SID=<sid>)))"

hangs. Changing mtu doesn't help.

tcpdump of successful connection:

sudo tcpdump -i vpn0 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vpn0, link-type RAW (Raw IP), capture size 262144 bytes
12:15:10.087467 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [S], seq 3956465851, win 21200, options [mss 1060,sackOK,TS val 408789 ecr 0,nop,wscale 7], length 0
12:15:10.128225 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [S.], seq 2090041496, ack 3956465852, win 14480, options [mss 1160,sackOK,TS val 2910434587 ecr 408789,nop,wscale 7], length 0
12:15:10.128299 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [.], ack 1, win 166, options [nop,nop,TS val 408799 ecr 2910434587], length 0
12:15:10.128539 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 1:217, ack 1, win 166, options [nop,nop,TS val 408800 ecr 2910434587], length 216
12:15:10.167537 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [.], ack 217, win 122, options [nop,nop,TS val 2910434626 ecr 408800], length 0
12:15:10.185461 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 1:9, ack 217, win 122, options [nop,nop,TS val 2910434644 ecr 408800], length 8
12:15:10.185566 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [.], ack 9, win 166, options [nop,nop,TS val 408814 ecr 2910434644], length 0
12:15:10.185614 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 217:433, ack 9, win 166, options [nop,nop,TS val 408814 ecr 2910434644], length 216
12:15:10.224829 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 9:50, ack 433, win 130, options [nop,nop,TS val 2910434684 ecr 408814], length 41
12:15:10.224999 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 433:592, ack 50, win 166, options [nop,nop,TS val 408824 ecr 2910434684], length 159
12:15:10.264401 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 50:177, ack 592, win 139, options [nop,nop,TS val 2910434723 ecr 408824], length 127
12:15:10.268475 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 592:626, ack 177, win 166, options [nop,nop,TS val 408834 ecr 2910434723], length 34
12:15:10.307915 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 177:366, ack 626, win 139, options [nop,nop,TS val 2910434767 ecr 408834], length 189
12:15:10.308490 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 626:710, ack 366, win 174, options [nop,nop,TS val 408844 ecr 2910434767], length 84
12:15:10.348210 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 366:392, ack 710, win 139, options [nop,nop,TS val 2910434807 ecr 408844], length 26
12:15:10.349603 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 710:956, ack 392, win 174, options [nop,nop,TS val 408855 ecr 2910434807], length 246
12:15:10.391478 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 392:880, ack 956, win 147, options [nop,nop,TS val 2910434850 ecr 408855], length 488
12:15:10.392187 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 956:1966, ack 880, win 183, options [nop,nop,TS val 408865 ecr 2910434850], length 1010
12:15:10.440238 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [.], seq 880:1928, ack 1966, win 163, options [nop,nop,TS val 2910434898 ecr 408865], length 1048
12:15:10.440295 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [.], seq 1928:2976, ack 1966, win 163, options [nop,nop,TS val 2910434898 ecr 408865], length 1048
12:15:10.440330 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 2976:3051, ack 1966, win 163, options [nop,nop,TS val 2910434898 ecr 408865], length 75
12:15:10.440372 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [.], ack 2976, win 216, options [nop,nop,TS val 408877 ecr 2910434898], length 0
12:15:10.440854 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 1966:2026, ack 3051, win 216, options [nop,nop,TS val 408878 ecr 2910434898], length 60
12:15:10.479335 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 2976:3051, ack 1966, win 163, options [nop,nop,TS val 2910434938 ecr 408877], length 75
12:15:10.479454 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [.], ack 3051, win 216, options [nop,nop,TS val 408887 ecr 2910434938,nop,nop,sack 1 {2976:3051}], length 0
12:15:10.480049 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 3051:3237, ack 2026, win 163, options [nop,nop,TS val 2910434939 ecr 408878], length 186
12:15:10.480120 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [.], ack 3237, win 232, options [nop,nop,TS val 408887 ecr 2910434939], length 0
12:15:10.480277 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 2026:2377, ack 3237, win 232, options [nop,nop,TS val 408887 ecr 2910434939], length 351
12:15:10.520270 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 3237:3680, ack 2377, win 179, options [nop,nop,TS val 2910434979 ecr 408887], length 443
12:15:10.520574 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 2377:2398, ack 3680, win 248, options [nop,nop,TS val 408898 ecr 2910434979], length 21
12:15:10.559853 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 3680:3860, ack 2398, win 179, options [nop,nop,TS val 2910435019 ecr 408898], length 180
12:15:10.560150 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 2398:2710, ack 3860, win 265, options [nop,nop,TS val 408907 ecr 2910435019], length 312
12:15:10.600069 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 3860:4049, ack 2710, win 194, options [nop,nop,TS val 2910435059 ecr 408907], length 189
12:15:10.600313 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 2710:3145, ack 4049, win 281, options [nop,nop,TS val 408917 ecr 2910435059], length 435
12:15:10.640784 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 4049:4690, ack 3145, win 210, options [nop,nop,TS val 2910435099 ecr 408917], length 641
12:15:10.641094 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 3145:3603, ack 4690, win 297, options [nop,nop,TS val 408928 ecr 2910435099], length 458
12:15:10.681713 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 4690:5046, ack 3603, win 226, options [nop,nop,TS val 2910435140 ecr 408928], length 356
12:15:10.681999 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 3603:3989, ack 5046, win 314, options [nop,nop,TS val 408938 ecr 2910435140], length 386
12:15:10.722164 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 5046:5286, ack 3989, win 242, options [nop,nop,TS val 2910435181 ecr 408938], length 240
12:15:10.722406 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 3989:4355, ack 5286, win 330, options [nop,nop,TS val 408948 ecr 2910435181], length 366
12:15:10.762352 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 5286:5708, ack 4355, win 258, options [nop,nop,TS val 2910435221 ecr 408948], length 422
12:15:10.762606 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 4355:4391, ack 5708, win 347, options [nop,nop,TS val 408958 ecr 2910435221], length 36
12:15:10.801942 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 5708:5725, ack 4391, win 258, options [nop,nop,TS val 2910435261 ecr 408958], length 17
12:15:10.802095 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [P.], seq 4391:4404, ack 5725, win 347, options [nop,nop,TS val 408968 ecr 2910435261], length 13
12:15:10.841247 IP 10.4.0.32.1521 > 172.30.2.157.38998: Flags [P.], seq 5725:5742, ack 4404, win 258, options [nop,nop,TS val 2910435300 ecr 408968], length 17
12:15:10.884508 IP 172.30.2.157.38998 > 10.4.0.32.1521: Flags [.], ack 5742, win 347, options [nop,nop,TS val 408989 ecr 2910435300], length 0
^C
46 packets captured
46 packets received by filter
0 packets dropped by kernel

when stuck:

sudo tcpdump -i vpn0 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vpn0, link-type RAW (Raw IP), capture size 262144 bytes
12:16:11.422616 IP 172.30.2.157.52186 > 10.4.0.32.1521: Flags [S], seq 1453371367, win 29200, options [mss 1460,sackOK,TS val 424123 ecr 0,nop,wscale 7], length 0
12:16:11.463151 IP 10.4.0.32.1521 > 172.30.2.157.52186: Flags [S.], seq 2123913164, ack 1453371368, win 14480, options [mss 1160,sackOK,TS val 2910495922 ecr 424123,nop,wscale 7], length 0
12:16:11.463253 IP 172.30.2.157.52186 > 10.4.0.32.1521: Flags [.], ack 1, win 229, options [nop,nop,TS val 424133 ecr 2910495922], length 0
12:16:11.463476 IP 172.30.2.157.52186 > 10.4.0.32.1521: Flags [P.], seq 1:217, ack 1, win 229, options [nop,nop,TS val 424133 ecr 2910495922], length 216
12:16:11.502416 IP 10.4.0.32.1521 > 172.30.2.157.52186: Flags [.], ack 217, win 122, options [nop,nop,TS val 2910495961 ecr 424133], length 0
12:16:11.523656 IP 10.4.0.32.1521 > 172.30.2.157.52186: Flags [P.], seq 1:9, ack 217, win 122, options [nop,nop,TS val 2910495982 ecr 424133], length 8
12:16:11.523740 IP 172.30.2.157.52186 > 10.4.0.32.1521: Flags [.], ack 9, win 229, options [nop,nop,TS val 424148 ecr 2910495982], length 0
12:16:11.523783 IP 172.30.2.157.52186 > 10.4.0.32.1521: Flags [P.], seq 217:433, ack 9, win 229, options [nop,nop,TS val 424148 ecr 2910495982], length 216
12:16:11.563449 IP 10.4.0.32.1521 > 172.30.2.157.52186: Flags [P.], seq 9:50, ack 433, win 130, options [nop,nop,TS val 2910496022 ecr 424148], length 41
12:16:11.563639 IP 172.30.2.157.52186 > 10.4.0.32.1521: Flags [P.], seq 433:592, ack 50, win 229, options [nop,nop,TS val 424158 ecr 2910496022], length 159
12:16:11.603058 IP 10.4.0.32.1521 > 172.30.2.157.52186: Flags [P.], seq 50:177, ack 592, win 139, options [nop,nop,TS val 2910496062 ecr 424158], length 127
12:16:11.607200 IP 172.30.2.157.52186 > 10.4.0.32.1521: Flags [P.], seq 592:626, ack 177, win 229, options [nop,nop,TS val 424169 ecr 2910496062], length 34
12:16:11.646395 IP 10.4.0.32.1521 > 172.30.2.157.52186: Flags [P.], seq 177:366, ack 626, win 139, options [nop,nop,TS val 2910496105 ecr 424169], length 189
12:16:11.646864 IP 172.30.2.157.52186 > 10.4.0.32.1521: Flags [P.], seq 626:710, ack 366, win 237, options [nop,nop,TS val 424179 ecr 2910496105], length 84
12:16:11.686621 IP 10.4.0.32.1521 > 172.30.2.157.52186: Flags [P.], seq 366:392, ack 710, win 139, options [nop,nop,TS val 2910496145 ecr 424179], length 26
12:16:11.687757 IP 172.30.2.157.52186 > 10.4.0.32.1521: Flags [P.], seq 710:956, ack 392, win 237, options [nop,nop,TS val 424189 ecr 2910496145], length 246
12:16:11.729563 IP 10.4.0.32.1521 > 172.30.2.157.52186: Flags [P.], seq 392:880, ack 956, win 147, options [nop,nop,TS val 2910496188 ecr 424189], length 488
12:16:11.730214 IP 172.30.2.157.52186 > 10.4.0.32.1521: Flags [P.], seq 956:1966, ack 880, win 245, options [nop,nop,TS val 424200 ecr 2910496188], length 1010
12:16:11.777368 IP truncated-ip - 6 bytes missing! 10.4.0.32.1521 > 172.30.2.157.52186: Flags [.], seq 880:2028, ack 1966, win 163, options [nop,nop,TS val 2910496235 ecr 424200], length 1148
12:16:11.777411 IP 10.4.0.32.1521 > 172.30.2.157.52186: Flags [P.], seq 2028:3051, ack 1966, win 163, options [nop,nop,TS val 2910496236 ecr 424200], length 1023
12:16:11.777934 IP 172.30.2.157.52186 > 10.4.0.32.1521: Flags [.], ack 880, win 261, options [nop,nop,TS val 424212 ecr 2910496188,nop,nop,sack 1 {2028:3051}], length 0
12:16:12.019546 IP truncated-ip - 6 bytes missing! 10.4.0.32.1521 > 172.30.2.157.52186: Flags [.], seq 880:2028, ack 1966, win 163, options [nop,nop,TS val 2910496478 ecr 424212], length 1148
12:16:12.503770 IP truncated-ip - 6 bytes missing! 10.4.0.32.1521 > 172.30.2.157.52186: Flags [.], seq 880:2028, ack 1966, win 163, options [nop,nop,TS val 2910496962 ecr 424212], length 1148
12:16:13.471513 IP truncated-ip - 6 bytes missing! 10.4.0.32.1521 > 172.30.2.157.52186: Flags [.], seq 880:2028, ack 1966, win 163, options [nop,nop,TS val 2910497930 ecr 424212], length 1148
12:16:15.407660 IP truncated-ip - 6 bytes missing! 10.4.0.32.1521 > 172.30.2.157.52186: Flags [.], seq 880:2028, ack 1966, win 163, options [nop,nop,TS val 2910499866 ecr 424212], length 1148
12:16:19.279620 IP truncated-ip - 6 bytes missing! 10.4.0.32.1521 > 172.30.2.157.52186: Flags [.], seq 880:2028, ack 1966, win 163, options [nop,nop,TS val 2910503738 ecr 424212], length 1148
^C
26 packets captured
26 packets received by filter
0 packets dropped by kernel
6 packets dropped by interface

Please help. I can see "IP truncated-ip - 6 bytes missing!" and same incoming packet from server when connection hangs. And first packet from client suggest mss 1460, ignoring --opt com.docker.network.driver.mtu=1100 in case of overlay. When in bridged mode everything is ok.

Output of docker version:

Client:
 Version:           18.09.2
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        6247962
 Built:             Sun Feb 10 04:13:52 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.2
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       6247962
  Built:            Sun Feb 10 03:42:13 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

Containers: 4
 Running: 0
 Paused: 0
 Stopped: 4
Images: 360
Server Version: 18.09.2
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
 NodeID: t49u8p2l3jeop40rlzwtztmxx
 Is Manager: true
 ClusterID: 116m8szio1ossvo8z9zo810n5
 Managers: 1
 Nodes: 1
 Default Address Pool: 10.0.0.0/8  
 SubnetSize: 24
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 10
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: 192.168.88.253
 Manager Addresses:
  192.168.88.253:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
runc version: 09c8266bf2fcf9519a651b04ae54c967b9ab86ec
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.0-8-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.767GiB
Name: silverstone
ID: HLN2:7TO6:FRCJ:XTWH:WDUN:EP2P:75AR:LC7X:7AMJ:Y75O:XYXV:XN3C
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: andribas
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.)

$ip a show dev vpn0
19: vpn0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1194 qdisc pfifo_fast state UP group default qlen 500
    link/none 
    inet 172.30.2.157/24 brd 172.30.2.255 scope global vpn0
       valid_lft forever preferred_lft forever
andribas404 commented 5 years ago

from forums: https://forums.docker.com/t/cant-connect-to-oracle-db-on-external-network-from-container/53818