search

vmware / vic

vSphere Integrated Containers Engine is a container runtime for vSphere.
http://vmware.github.io/vic
Other
640 stars 174 forks source link

Deploying a multi-container wordpress application **With Network** on VCH via Admiral fails #3542

Closed bhravich closed 7 years ago

bhravich commented 7 years ago

VIC version:

vic-machine-linux version v0.8.0-7489-f116b8b

Deployment details:

./vic-machine-linux create --target 10.20.132.199 --image-store datastore1 --name bhu-docker-1 --user root --password 'password' --no-tlsverify --thumbprint 'xx:xx:xx:xx:xx:xx' --public-network 'VM Network' --volume-store=datastore1/vic-vol:default

 ### Installing VCH ####
 Using administrative user for VCH operation - use --ops-user to improve security (see -x for advanced help)
 Generating self-signed certificate/key pair - private key in bhu-docker-1/server-key.pem
 Configuring without TLS verify - certificate-based authentication disabled
 Validating supplied configuration
 Firewall status: DISABLED on "/ha-datacenter/host/prome-1s-dhcp199.eng.vmware.com/prome-1s-dhcp199.eng.vmware.com"
 Firewall configuration will be incorrect if firewall is reenabled on hosts:
   "/ha-datacenter/host/prome-1s-dhcp199.eng.vmware.com/prome-1s-dhcp199.eng.vmware.com"
 Firewall must permit dst 2377/tcp outbound to VCH management interface if firewall is reenabled
 License check OK
 DRS check SKIPPED - target is standalone host

 Creating Resource Pool "bhu-docker-1"
 Creating VirtualSwitch
 Creating Portgroup
 Creating directory [datastore1] vic-vol
 datastore root [datastore1] vic-vol already exists
 Datastore path is [datastore1] vic-vol
 Creating appliance on target
 Network role "client" is sharing NIC with "public"
 Network role "management" is sharing NIC with "public"
 Uploading images for container
   "bootstrap.iso"
   "appliance.iso"
 Waiting for IP information
 Waiting for major appliance components to launch
 Checking VCH connectivity with vSphere target
 Could not run VCH vSphere API target check due to ServerFaultCode: The guest authentication being used does not have sufficient permissions to perform the operation.
 Initialization of appliance successful

 VCH Admin Portal:
 https://10.20.132.44:2378

 Published ports can be reached at:
 10.20.132.44

 Docker environment variables:
 DOCKER_HOST=10.20.132.44:2376

 Environment saved in bhu-docker-1/bhu-docker-1.env

 Connect to docker:
 docker -H 10.20.132.44:2376 --tls info
 Installer completed successfully

Steps to reproduce:

  1. Deploy the VCH using the above mentioned command.
  2. Register the host on admiral.
  3. Upload an application template with the following specification (With Network Definition): 
    ---
name: "New-WP"
components:
wordpress:
type: "App.Container"
data:
  name: "wordpress"
  image: "wordpress:4.3.1"
  _cluster: 2
  env:
  - var: "WORDPRESS_DB_PASSWORD"
    value: "pass@word01"
  - var: "WORDPRESS_DB_NAME"
    value: "wordpress"
  - var: "WORDPRESS_DB_HOST"
    value: "mysql:3306"
  networks:
  - name: "new-net"
  publish_all: true
  ports:
  - container_port: "80"
  links:
  - service: "mysql"
    alias: "mysql"
  restart_policy: "no"
dependsOn:
- "mysql"
new-net:
type: "App.Network"
data:
  name: "new-net"
  external: false
mysql:
type: "App.Container"
data:
  name: "mysql"
  image: "mariadb:10.0.26"
  _cluster: 1
  env:
  - var: "MYSQL_ROOT_PASSWORD"
    value: "pass@word01"
  networks:
  - name: "new-net"
  publish_all: true
  ports:
  - container_port: "3306"
  restart_policy: "no"

    Actual behavior: The mysql container is sucessfully deployed however, the wordpress container goes offline after reporting the following error in the console: screen shot 2016-12-20 at 3 31 40 pm Expected behavior: The application should have successfully deployed. Logs: container-logs.zip

docker -H 10.20.132.44:2376 --tls ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                      PORTS                          NAMES
ae93a018cc43        wordpress:4.3.1     "/entrypoint.sh apach"   7 hours ago         Exited (1) 19 minutes ago   10.20.132.44:32776->80/tcp     wordpress-mcm368_30668812143
b891a58cc295        wordpress:4.3.1     "/entrypoint.sh apach"   7 hours ago         Exited (1) 43 minutes ago   10.20.132.44:32777->80/tcp     wordpress-mcm369_30668812143
10078bd3bff8        mariadb:10.0.26     "docker-entrypoint.sh"   7 hours ago         Up 44 minutes               10.20.132.44:32775->3306/tcp   mysql-mcm367_30668805634
22926c1659a5        wordpress:4.3.1     "/entrypoint.sh apach"   8 hours ago         Up About an hour            10.20.132.44:32770->80/tcp     wordpress-mcm360_30664839918
3ccac4bd5ba6        wordpress:4.3.1     "/entrypoint.sh apach"   8 hours ago         Up About an hour            10.20.132.44:32769->80/tcp     wordpress-mcm359_30664559422
0aef1a3a67b2        mariadb:10.0.26     "docker-entrypoint.sh"   8 hours ago         Up About an hour            10.20.132.44:32768->3306/tcp   mysql-mcm358_30664556128
docker -H 10.20.132.44:2376 --tls network ls
NETWORK ID          NAME                         DRIVER
3228c2b77bda        bridge                       bridge              
3e36c16d6114        new-net-mcm366_30668802410   bridge  
docker -H 10.20.132.44:2376 --tls network inspect new-net-mcm366_30668802410
[
    {
        "Name": "new-net-mcm366_30668802410",
        "Id": "3e36c16d61146e5ae4e5cc570dccc97b94cd34281f72890e7a2d2ee4b0f41a3a",
        "Scope": "",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "10078bd3bff8930a558180eac5fa412d0a1cb6b5069d294fcefd3cfe6953d96a": {
                "Name": "mysql-mcm367_30668805634",
                "EndpointID": "10078bd3bff8930a558180eac5fa412d0a1cb6b5069d294fcefd3cfe6953d96a",
                "MacAddress": "",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]
docker -H 10.20.132.44:2376 --tls inspect wordpress-mcm368_30668812143
[
    {
        "Id": "ae93a018cc43241fd6f04beda5ebba486023979126662c1bc75e88919dbcf26e",
        "Created": "2016-12-20T16:38:18Z",
        "Path": "/entrypoint.sh",
        "Args": [
            "apache2-foreground"
        ],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 1,
            "Error": "",
            "StartedAt": "2016-12-20T17:02:59Z",
            "FinishedAt": "2016-12-20T17:03:38Z"
        },
        "Image": "db54b37edfcffd257f0db461bb75cdd88656f65ed57375ea88f9d7afb0b6c66b",
        "ResolvConfPath": "",
        "HostnamePath": "",
        "HostsPath": "",
        "LogPath": "",
        "Name": "/wordpress-mcm368_30668812143",
        "RestartCount": 0,
        "Driver": "vSphere Integrated Containers v0.8.0-7489-f116b8b Backend Engine",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "bridge",
            "PortBindings": {
                "80/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": ""
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "vSphere Integrated Containers v0.8.0-7489-f116b8b Backend Engine",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": true,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "StorageOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 0,
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "BlkioIOps": 0,
            "BlkioBps": 0,
            "SandboxSize": 0
        },
        "GraphDriver": {
            "Name": "vSphere Integrated Containers v0.8.0-7489-f116b8b Backend Engine",
            "Data": null
        },
        "Mounts": null,
        "Config": {
            "Hostname": "ae93a018cc43",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "80/tcp": {}
            },
            "Tty": true,
            "OpenStdin": true,
            "StdinOnce": true,
            "Env": [
                "WORDPRESS_DB_PASSWORD=pass@word01",
                "WORDPRESS_DB_NAME=wordpress",
                "WORDPRESS_DB_HOST=mysql:3306",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "TERM=xterm",
                "PHP_INI_DIR=/usr/local/etc/php",
                "PHP_EXTRA_BUILD_DEPS=apache2-dev",
                "PHP_EXTRA_CONFIGURE_ARGS=--with-apxs2",
                "GPG_KEYS=0BD78B5F97500D450838F95DFE857D9A90D90EC1 6E4F6AB321FDC07F2C332E3AC2BF0BC433CFC8B3",
                "PHP_VERSION=5.6.16",
                "WORDPRESS_VERSION=4.3.1",
                "WORDPRESS_SHA1=b2e5652a6d2333cabe7b37459362a3e5b8b66221"
            ],
            "Cmd": [
                "apache2-foreground"
            ],
            "Image": "wordpress:4.3.1",
            "Volumes": {
                "/var/www/html": {}
            },
            "WorkingDir": "/var/www/html",
            "Entrypoint": [
                "/entrypoint.sh"
            ],
            "OnBuild": [],
            "Labels": null
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "80/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "32776"
                    }
                ]
            },
            "SandboxKey": "",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "",
                    "EndpointID": "192",
                    "Gateway": "172.16.0.1/16",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": ""
                },
                "new-net-mcm366_30668802410": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "mysql",
                        "wordpress",
                        "ae93a018cc43"
                    ],
                    "NetworkID": "",
                    "EndpointID": "192",
                    "Gateway": "172.18.0.1/16",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": ""
                }
            }
        }
    }
]

Additional details as necessary:

bhravich commented 7 years ago

The following are the details of the same application when deployed onto a regular docker host:

 docker ps
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS                                              NAMES
1418f9aba060        library/wordpress:4.3.1   "/entrypoint.sh apach"   11 hours ago        Up 11 hours         0.0.0.0:32781->80/tcp, 0.0.0.0:32782->80/tcp       wordpress-mcm324-31996297365
74dd480cb196        library/wordpress:4.3.1   "/entrypoint.sh apach"   11 hours ago        Up 11 hours         0.0.0.0:32773->80/tcp, 0.0.0.0:32774->80/tcp       wordpress-mcm318-31995892346
0ee461acd815        library/wordpress:4.3.1   "/entrypoint.sh apach"   11 hours ago        Up 11 hours         0.0.0.0:32771->80/tcp, 0.0.0.0:32772->80/tcp       wordpress-mcm319-31995892346
05d90b192370        library/mariadb:10.0.26   "docker-entrypoint.sh"   11 hours ago        Up 11 hours         0.0.0.0:32769->3306/tcp, 0.0.0.0:32770->3306/tcp   mysql-mcm317-31995881426
docker network ls
NETWORK ID          NAME                         DRIVER              SCOPE
c6aa97d0101b        bridge                       bridge              local               
c60ee7f256d4        host                         host                local               
079aa3dd5f7e        new-net-mcm316-31995872336   bridge              local               
bd40a3977d21        none                         null                local 
docker network inspect new-net-mcm316-31995872336
[
    {
        "Name": "new-net-mcm316-31995872336",
        "Id": "079aa3dd5f7ed903782f793ecbe632b6df761b830b3c1686ac81a9cf50219ffd",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1/16"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "05d90b192370811e6cee1492bea624270acb2ab35f506a3fb42bea42b4ebf947": {
                "Name": "mysql-mcm317-31995881426",
                "EndpointID": "85490f07a3194ce21ee64c5ed0ea0118e57234d97a4c4b35944b7825ff1c9f90",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            },
            "0ee461acd815d677742dec76f0dcc70efca02740e01e5b571985f6199be2f904": {
                "Name": "wordpress-mcm319-31995892346",
                "EndpointID": "9c757db8a94aa03436dcbaef39f2862de852ad17ba9fc78a87f0fc8c2af7f2a6",
                "MacAddress": "02:42:ac:12:00:03",
                "IPv4Address": "172.18.0.3/16",
                "IPv6Address": ""
            },
            "1418f9aba0600d5617bc5943e5a54b728bbd3802878fe860583e8308b6d1767d": {
                "Name": "wordpress-mcm324-31996297365",
                "EndpointID": "6b16b874536de4379faf1c390e14e38f9ecb810e398064e94990b2c32b32132a",
                "MacAddress": "02:42:ac:12:00:05",
                "IPv4Address": "172.18.0.5/16",
                "IPv6Address": ""
            },
            "74dd480cb196bf2c09134fc629a3981ddd69c434b0d55a38e8de362145764458": {
                "Name": "wordpress-mcm318-31995892346",
                "EndpointID": "f3bf1dfc6dcf402a980ea5c774aec0dbddc653abcc427a7bddc2a74a3acc7277",
                "MacAddress": "02:42:ac:12:00:04",
                "IPv4Address": "172.18.0.4/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]
docker inspect wordpress-mcm324-31996297365
[
    {
        "Id": "1418f9aba0600d5617bc5943e5a54b728bbd3802878fe860583e8308b6d1767d",
        "Created": "2017-01-05T07:51:51.215827108Z",
        "Path": "/entrypoint.sh",
        "Args": [
            "apache2-foreground"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 3336,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2017-01-05T07:51:51.46603354Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:5e5ac19dc8641c465379a7443a62688ea517b9874ad48ce5cc28c9efdd56aca2",
        "ResolvConfPath": "/var/lib/docker/containers/1418f9aba0600d5617bc5943e5a54b728bbd3802878fe860583e8308b6d1767d/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/1418f9aba0600d5617bc5943e5a54b728bbd3802878fe860583e8308b6d1767d/hostname",
        "HostsPath": "/var/lib/docker/containers/1418f9aba0600d5617bc5943e5a54b728bbd3802878fe860583e8308b6d1767d/hosts",
        "LogPath": "/var/lib/docker/containers/1418f9aba0600d5617bc5943e5a54b728bbd3802878fe860583e8308b6d1767d/1418f9aba0600d5617bc5943e5a54b728bbd3802878fe860583e8308b6d1767d-json.log",
        "Name": "/wordpress-mcm324-31996297365",
        "RestartCount": 0,
        "Driver": "overlay",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {
                "80/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": ""
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": true,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Name": "overlay",
            "Data": {
                "LowerDir": "/var/lib/docker/overlay/41ece09c8d79bc4afa5fa8153bde5c314fe0c1a0023b654821df0c474d4d7a76/root",
                "MergedDir": "/var/lib/docker/overlay/f4a3c2c134b3823d561b7bf9ea892de3923d5f9340b61ad6126a72fac498a233/merged",
                "UpperDir": "/var/lib/docker/overlay/f4a3c2c134b3823d561b7bf9ea892de3923d5f9340b61ad6126a72fac498a233/upper",
                "WorkDir": "/var/lib/docker/overlay/f4a3c2c134b3823d561b7bf9ea892de3923d5f9340b61ad6126a72fac498a233/work"
            }
        },
        "Mounts": [
            {
                "Name": "262b21bba6760a2d0384b67aa0a265487a49ef3e10b645736cd0a34524d49d2c",
                "Source": "/var/lib/docker/volumes/262b21bba6760a2d0384b67aa0a265487a49ef3e10b645736cd0a34524d49d2c/_data",
                "Destination": "/var/www/html",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "1418f9aba060",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "80": {},
                "80/tcp": {}
            },
            "Tty": true,
            "OpenStdin": true,
            "StdinOnce": false,
            "Env": [
                "WORDPRESS_DB_PASSWORD=pass@word01",
                "WORDPRESS_DB_NAME=wordpress",
                "WORDPRESS_DB_HOST=mysql:3306",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "PHP_INI_DIR=/usr/local/etc/php",
                "PHP_EXTRA_BUILD_DEPS=apache2-dev",
                "PHP_EXTRA_CONFIGURE_ARGS=--with-apxs2",
                "GPG_KEYS=0BD78B5F97500D450838F95DFE857D9A90D90EC1 6E4F6AB321FDC07F2C332E3AC2BF0BC433CFC8B3",
                "PHP_VERSION=5.6.16",
                "WORDPRESS_VERSION=4.3.1",
                "WORDPRESS_SHA1=b2e5652a6d2333cabe7b37459362a3e5b8b66221"
            ],
            "Cmd": [
                "apache2-foreground"
            ],
            "Image": "library/wordpress:4.3.1",
            "Volumes": {
                "/var/www/html": {}
            },
            "WorkingDir": "/var/www/html",
            "Entrypoint": [
                "/entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "c35f2e7074cf0671c990fded1c82c6c061dbebbf77b0b240db27d58a967c9177",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "80/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "32781"
                    },
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "32782"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/c35f2e7074cf",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "new-net-mcm316-31995872336": {
                    "IPAMConfig": null,
                    "Links": [
                        "mysql:mysql"
                    ],
                    "Aliases": [
                        "wordpress",
                        "1418f9aba060"
                    ],
                    "NetworkID": "079aa3dd5f7ed903782f793ecbe632b6df761b830b3c1686ac81a9cf50219ffd",
                    "EndpointID": "6b16b874536de4379faf1c390e14e38f9ecb810e398064e94990b2c32b32132a",
                    "Gateway": "172.18.0.1",
                    "IPAddress": "172.18.0.5",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:12:00:05"
                }
            }
        }
    }
]
bhravich commented 7 years ago

Adding the mysql container details on regular docker host as well:

docker inspect mysql-mcm317-31995881426
[
    {
        "Id": "05d90b192370811e6cee1492bea624270acb2ab35f506a3fb42bea42b4ebf947",
        "Created": "2017-01-05T07:45:26.951885119Z",
        "Path": "docker-entrypoint.sh",
        "Args": [
            "mysqld"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 1219,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2017-01-05T07:45:27.166903234Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:561bf343a362583af0c162790c820cd4743de5e57719ae50011a22f155116d10",
        "ResolvConfPath": "/var/lib/docker/containers/05d90b192370811e6cee1492bea624270acb2ab35f506a3fb42bea42b4ebf947/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/05d90b192370811e6cee1492bea624270acb2ab35f506a3fb42bea42b4ebf947/hostname",
        "HostsPath": "/var/lib/docker/containers/05d90b192370811e6cee1492bea624270acb2ab35f506a3fb42bea42b4ebf947/hosts",
        "LogPath": "/var/lib/docker/containers/05d90b192370811e6cee1492bea624270acb2ab35f506a3fb42bea42b4ebf947/05d90b192370811e6cee1492bea624270acb2ab35f506a3fb42bea42b4ebf947-json.log",
        "Name": "/mysql-mcm317-31995881426",
        "RestartCount": 0,
        "Driver": "overlay",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {
                "3306/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": ""
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": true,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Name": "overlay",
            "Data": {
                "LowerDir": "/var/lib/docker/overlay/eeedb0e4ef9f662d8a3a5a6cd2678b6a03781fdf8a8536022d456ca033a4713f/root",
                "MergedDir": "/var/lib/docker/overlay/70b4338cc841767ebf33cc8862088590a79c1a6701f570b472ebf44ecf071809/merged",
                "UpperDir": "/var/lib/docker/overlay/70b4338cc841767ebf33cc8862088590a79c1a6701f570b472ebf44ecf071809/upper",
                "WorkDir": "/var/lib/docker/overlay/70b4338cc841767ebf33cc8862088590a79c1a6701f570b472ebf44ecf071809/work"
            }
        },
        "Mounts": [
            {
                "Name": "3b9628956c7e136e1b03a340d10b675d4e31c819d3596a4f9a1d9b4970638169",
                "Source": "/var/lib/docker/volumes/3b9628956c7e136e1b03a340d10b675d4e31c819d3596a4f9a1d9b4970638169/_data",
                "Destination": "/var/lib/mysql",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "05d90b192370",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "3306": {},
                "3306/tcp": {}
            },
            "Tty": true,
            "OpenStdin": true,
            "StdinOnce": false,
            "Env": [
                "MYSQL_ROOT_PASSWORD=pass@word01",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "GOSU_VERSION=1.7",
                "MARIADB_MAJOR=10.0",
                "MARIADB_VERSION=10.0.26+maria-1~jessie"
            ],
            "Cmd": [
                "mysqld"
            ],
            "Image": "library/mariadb:10.0.26",
            "Volumes": {
                "/var/lib/mysql": {}
            },
            "WorkingDir": "",
            "Entrypoint": [
                "docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "45d587da32e768348eca2cc869f4ae1ceb2ee6763ef531088424414a66d3f984",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "3306/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "32769"
                    },
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "32770"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/45d587da32e7",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "new-net-mcm316-31995872336": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "mysql",
                        "05d90b192370"
                    ],
                    "NetworkID": "079aa3dd5f7ed903782f793ecbe632b6df761b830b3c1686ac81a9cf50219ffd",
                    "EndpointID": "85490f07a3194ce21ee64c5ed0ea0118e57234d97a4c4b35944b7825ff1c9f90",
                    "Gateway": "172.18.0.1",
                    "IPAddress": "172.18.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:12:00:02"
                }
            }
        }
    }
]
chengwang86 commented 7 years ago

@bhravich @hmahmood By comparing the network configs of the containers when deployed by vic vs. regular docker, it seems that the containers were added to the bridge network under vic whereas they were not under regular docker.

Since the alias "mysql" was not added to the bridge network, and the dns requests of the wordpress container used the bridge network, the wordpress container was not able to find the mysql container.

I will investigate into why the containers were added to the bridge network. What we expect is that the containers only connect to the user-defined network "new-net", which is true under regular docker.

hickeng commented 7 years ago

@chengwang86 I think that Admiral is calling network connect for new_net instead of creating the container directly on it. Can you confirm the following with regular docker host & VCH:

  1. regular compose - does this use network connect or set the network directly on creation
  2. admiral compose - is my hypothesis correct about admiral calling network connect

The aliases are added for both networks:

time="2016-12-20T16:32:12Z" level=debug msg="parsing alias :mysql" 
time="2016-12-20T16:32:12Z" level=debug msg="adding alias bridge:mysql-mcm363_30668450124 -> mysql-mcm363_30668450124" 
time="2016-12-20T16:32:12Z" level=debug msg="adding alias bridge:7870a8214173 -> mysql-mcm363_30668450124" 
time="2016-12-20T16:32:12Z" level=debug msg="adding alias new-net-mcm362_30668446875:mysql-mcm363_30668450124 -> mysql-mcm363_30668450124" 
time="2016-12-20T16:32:12Z" level=debug msg="adding alias new-net-mcm362_30668446875:7870a8214173 -> mysql-mcm363_30668450124" 
time="2016-12-20T16:32:12Z" level=debug msg="adding alias new-net-mcm362_30668446875:mysql -> mysql-mcm363_30668450124"

@hmahmood @caglar10ur says that you may have address this?

hmahmood commented 7 years ago

@hickeng the aliases are added correctly. The problem seems to be that the same calls from Admiral result in different behavior in VIC vs. docker. It is very likely that we are not processing the parameters to the network connect or docker create call properly.

hickeng commented 7 years ago

@bhravich Is the output from the regular docker example also deployed via Admiral, or using compose directly?

bhravich commented 7 years ago

@hickeng The output from the regular docker examples are deployed via Admiral. Not through compose.

chengwang86 commented 7 years ago

Update: @hickeng @hmahmood

I just confirmed with the admiral team: admiral takes three steps:

  1. network create --> new-net-mcm...
  2. container create ---> create container mysql
  3. connect container to network ---> network connect new-net-mcm mysql

However, admiral uses docker rest api instead of cli to achieve these steps. So in our docker-personality log, I find 

time="2016-12-13T00:58:07Z" level=debug msg="Calling POST /v1.19/networks/create" 
time="2016-12-13T00:58:07Z" level=debug msg="form data: {\"Driver\":\"bridge\",\"Name\":\"new-net-mcm636_29984275416\"}" 
... ...
time="2016-12-13T00:58:11Z" level=debug msg="Calling POST /v1.19/containers/create?name=mysql-mcm637_29984278239" 
time="2016-12-13T00:58:11Z" level=debug msg="form data: {\"Config\":{},\"Env\":[\"MYSQL_ROOT_PASSWORD=pass@word01\"],\"ExposedPorts\":{\"3306\":{}},\"HostConfig\":{\"CpuShares\":0,\"Memory\":0,\"PortBindings\":{\"3306/tcp\":[{}]},\"PublishAllPorts\":true,\"RestartPolicy\":{\"Name\":\"no\"}},\"Image\":\"mariadb:10.0.26\",\"Name\":\"mysql-mcm637_29984278239\",\"NetworkingConfig\":{\"EndpointsConfig\":{\"new-net-mcm636_29984275416\":{\"Aliases\":[\"mysql\"]}}},\"OpenStdin\":true,\"Tty\":true,\"__allocation_request\":\"true\",\"__composition_context_id\":\"cc1755b65437fb5d950f8\",\"__referer\":\"/request/composition-sub-tasks/cc1755b65437fb5d950f8-new-net\"}" 
... ...
time="2016-12-13T00:58:13Z" level=debug msg="Calling POST /v1.19/networks/new-net-mcm636_29984275416/connect" 
time="2016-12-13T00:58:13Z" level=debug msg="form data: {\"Container\":\"a86094d4633eda53167829630ff56a444c19190f569013325f689aeb12fce917\",\"EndpointConfig\":{\"Aliases\":[\"mysql\"]},\"Id\":\"new-net-mcm636_29984275416\"}"

However, when I just manually run docker cmds following the above steps, I would obtain a container which is connected to both the bridge and the new-net-mcm networks, no matter on regular docker or vic.

chengwang86 commented 7 years ago

If I simply send the above form data for "docker create" to the docker daemon via 

curl -H "Content-Type: application/json"  -d form-data -X POST http:/v1.23/containers/create

then on regular docker, I get a container with NeworkMode=default, which is only connected to the user-defined network.

On vic, I get a container with NetworkMode=bridge, which is only connected to the bridge network (not the user-defined network).

In both cases, the user-defined network pre-exists before I create the container.

Note that in the form-data, the networkmode is not set. However,

Seems like on vic the container would be connected to the bridge network if either of these happens: (1) networkmode is not specified, (2) networkmode=default, or (3) networkmode=bridge, regardless of the NetworkConfig.

Possible fix 1: if admiral could set NetworkMode={user-defined network} in the form data, it might work. No changes at vic engine.

Possible fix 2: Change the vic engine code. If the NetworkMode is not specified in the form-data while the user-defined network is specified, connect the container to the user-defined network without the bridge network.

@hmahmood

shadjiiski commented 7 years ago

@chengwang86 could you try to remove the links from your template and see if this fixes anything? I am not sure whether VIC supports the links construct (VIC team to confirm), but for docker they are deprecated and are also not needed when networking is used. Furthermore, if you look closely, you will see that for your wordpress container VIC reports null links whereas docker reports mysql:mysql. Also, VIC has added a mysql network alias for the wordpress container (along with the wordpress and the container-id aliases). This shouldn't be there (see inspect output for the wordpress container on docker).

Seems like on vic the container would be connected to the bridge network if either of these happens: (1) networkmode is not specified, (2) networkmode=default, or (3) networkmode=bridge, regardless of the NetworkConfig.

You may want to check #2557

Possible fix 1: if admiral could set NetworkMode={user-defined network} in the form data, it might work. No changes at vic engine.

This is not an option: what should be the NetworkMode value if a container is connected to more than one user-defined network? (I think it is default for docker in this cases, i.e. none of the networks) Also, using NetworkMode in place of NetworkConfig will not let you override the default network settings (e.g. subnet, iprange, etc.)

chengwang86 commented 7 years ago

@shadjiiski vic engine does support link. See this example https://github.com/vmware/vic/issues/3439

You are right that "fix 1" is not an option. I'm discussing with @hmahmood to modify our vic code and mimic docker network behavior. Thanks for the informative comment.

stuclem commented 7 years ago

I have added the following to the 1.1.1 release notes:

@bhravich is this OK? Does this need to go into the 1.2 notes too? Thanks!