hcsshim::CreateComputeSystem The object already exists

Just started to debug into an issue that we're starting to see after integrating Consul into a compose stack under LCOW (PR at https://github.com/puppetlabs/pupperware/pull/37).

Opening an issue as I investigate further in case there's something obviously wrong that we're doing given current LCOW limitations, etc.

Error Message

The error would appear to originate somewhere within https://github.com/Microsoft/hcsshim/blob/4dd3466fb47933ff5df526e7751cdf1f6315bfcf/internal/hcs/system.go#L78-L125

Creating network "pupperware_default" with the default driver
Creating pupperware_consul_1   ... error
Creating pupperware_postgres_1 ...

ERROR: for pupperware_consul_1  Cannot start service consul: hcsshim::CreateComputeSystem 63bc7e31451c982a1d61a2d58241a159bbb8a14742d9a41e5cb6b493c51ae0be: The object already exists.
(extra info: {"SystemType":"container","Name":"63bc7e31451c982a1d61a2d58241a159bbb8a14742d9a41e5cb6b493c51ae0be","Owner":"docker","LayerFolderPath":"C:\\ProgramData\\docker\\lcow\\63bc7e31451c982a1d61a2d58241a159bbb8a14742d9a41e5cb6b493c51ae0be","Layers":[{"ID":"78af33f1-41e4-5040-9a34-379c196d67c6","Path":"C:\\ProgramData\\docker\\lcow\\aac25173134312d04813f489bf421cc26516319f192cde9323dbd5712552ed79\\layer.vhd"},{"ID":"0a9de556-e086-5de0-8aa5-0dc7eda966d2","Path":"C:\\ProgramData\\docker\\lcow\\fcb92930ba1aaee670bf6e859320c8a9513c888b18075a985179e1c4a24b6ae6\\layer.vhd"},{"ID":"7d2d735a-9338-5b93-b94d-89be3b86d501","Path":"C:\\ProgramData\\docker\\lcow\\f3c6e475d30a14fdf7027f90fb926b70820044ab4862f9c8e2217a05b1adfab0\\layer.vhd"},{"ID":"f5fe9097-b284-52f0-ab16-5008d8ac6cf0","Path":"C:\\ProgramData\\docker\\lcow\\287b109960a042c925b25c465992da9585aaef01b54219bb40b6d18c42e6550c\\layer.vhd"},{"ID":"41a83db1-3517-5e45-bffc-cd490fbc9dfa","Path":"C:\\ProgramData\\docker\\lcow\\682656916f0fc1eee3c16b4d22ec1b5d5c09157894ab0e6ff0ff31d043b11fad\\layer.vhd"},{"ID":"ed240024-c2f1-5bd8-aa0c-94dc834fc25b","Path":"C:\\ProgramData\\docker\\lcow\\65707d2700015a042ce0a8eaa85ba8fe49d63a33665075ca063f1e4dfe8f25af\\layer.vhd"}],"MappedDirectories":[{"HostPath":"c:\\users\\puppet\\pupperware\\volumes\\consul","ContainerPath":"/tmp/gcs/63bc7e31451c982a1d61a2d58241a159bbb8a14742d9a41e5cb6b493c51ae0be/binds/consul/data","ReadOnly":false,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":true,"LinuxMetadata":true},{"HostPath":"C:\\ProgramData\\docker\\volumes\\f4f9204837acd657171075673458a598dcbeb3dbc65bd3cd1972042755fa9a8a\\_data","ContainerPath":"/tmp/gcs/63bc7e31451c982a1d61a2d58241a159bbb8a14742d9a41e5cb6b493c51ae0be/binds/consul/data","ReadOnly":false,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":true,"LinuxMetadata":true}],"HvPartition":true,"EndpointList":["cf7a299c-41de-44d3-9b84-26c16dbabfb9"],"HvRuntime":{"ImagePath":"C:\\Program Files\\Linux Containers","LinuxInitrdFile":"initrd.img","LinuxKernelFile":"kernel"},"AlCreating pupperware_postgres_1 ... done

ERROR: for consul  Cannot start service consul: hcsshim::CreateComputeSystem 63bc7e31451c982a1d61a2d58241a159bbb8a14742d9a41e5cb6b493c51ae0be: The object already exists.
(extra info: {"SystemType":"container","Name":"63bc7e31451c982a1d61a2d58241a159bbb8a14742d9a41e5cb6b493c51ae0be","Owner":"docker","LayerFolderPath":"C:\\ProgramData\\docker\\lcow\\63bc7e31451c982a1d61a2d58241a159bbb8a14742d9a41e5cb6b493c51ae0be","Layers":[{"ID":"78af33f1-41e4-5040-9a34-379c196d67c6","Path":"C:\\ProgramData\\docker\\lcow\\aac25173134312d04813f489bf421cc26516319f192cde9323dbd5712552ed79\\layer.vhd"},{"ID":"0a9de556-e086-5de0-8aa5-0dc7eda966d2","Path":"C:\\ProgramData\\docker\\lcow\\fcb92930ba1aaee670bf6e859320c8a9513c888b18075a985179e1c4a24b6ae6\\layer.vhd"},{"ID":"7d2d735a-9338-5b93-b94d-89be3b86d501","Path":"C:\\ProgramData\\docker\\lcow\\f3c6e475d30a14fdf7027f90fb926b70820044ab4862f9c8e2217a05b1adfab0\\layer.vhd"},{"ID":"f5fe9097-b284-52f0-ab16-5008d8ac6cf0","Path":"C:\\ProgramData\\docker\\lcow\\287b109960a042c925b25c465992da9585aaef01b54219bb40b6d18c42e6550c\\layer.vhd"},{"ID":"41a83db1-3517-5e45-bffc-cd490fbc9dfa","Path":"C:\\ProgramData\\docker\\lcow\\682656916f0fc1eee3c16b4d22ec1b5d5c09157894ab0e6ff0ff31d043b11fad\\layer.vhd"},{"ID":"ed240024-c2f1-5bd8-aa0c-94dc834fc25b","Path":"C:\\ProgramData\\docker\\lcow\\65707d2700015a042ce0a8eaa85ba8fe49d63a33665075ca063f1e4dfe8f25af\\layer.vhd"}],"MappedDirectories":[{"HostPath":"c:\\users\\puppet\\pupperware\\volumes\\consul","ContainerPath":"/tmp/gcs/63bc7e31451c982a1d61a2d58241a159bbb8a14742d9a41e5cb6b493c51ae0be/binds/consul/data","ReadOnly":false,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":true,"LinuxMetadata":true},{"HostPath":"C:\\ProgramData\\docker\\volumes\\f4f9204837acd657171075673458a598dcbeb3dbc65bd3cd1972042755fa9a8a\\_data","ContainerPath":"/tmp/gcs/63bc7e31451c982a1d61a2d58241a159bbb8a14742d9a41e5cb6b493c51ae0be/binds/consul/data","ReadOnly":false,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":true,"LinuxMetadata":true}],"HvPartition":true,"EndpointList":["cf7a299c-41de-44d3-9b84-26c16dbabfb9"],"HvRuntime":{"ImagePath":"C:\\Program Files\\Linux Containers","LinuxInitrdFile":"initrd.img","LinuxKernelFile":"kernel"},"AllowUnqualifiedDNSQuery":true,"ContainerType":"linux","TerminateOnLastHandleClosed":true})
ERROR: Encountered errors while bringing up the project.

Is there a good way to get a more detailed stack trace of failures like this? I've checked in Hyper-V-Compute and Hyper-V-Worker -> Operational event logs but only see Information messages.

docker-compose 1.24.0-rc1 invocation

> docker-compose --version
docker-compose version 1.24.0-rc1, build 0f3d4dda

> docker-compose -f .\docker-compose.yml -f .\docker-compose.windows.yml up

local `.env`

CONSUL_IP=172.22.0.100
PUPPERWARE_SUBNET=172.22.0.0/16

docker-compose.yml

version: '3'

services:
  consul:
    hostname: consul
    image: consul
    ports:
      - 8500:8500 # Consul HTTP
      - 53
    environment:
      CONSUL_BIND_INTERFACE: 'eth0'
      CONSUL_ALLOW_PRIVILEGED_PORTS: 'true'
      CONSUL_LOCAL_CONFIG: '{ "ports": { "dns": 53 }, "recursors": [ "8.8.8.8" ] }'
      CONSUL_IP:
    volumes:
      - ./volumes/consul:/consul/data/
    networks:
      default:
        ipv4_address: $CONSUL_IP

  puppet:
    hostname: puppet
    image: puppet/puppetserver
    ports:
      - 8140:8140
    environment:
      # DNS_ALT_NAMES must be set before starting the stack the first time,
      # and must list all the names under which the puppetserver can be
      # reached. 'puppet' must be one of them, otherwise puppetdb won't be
      # able to get a cert. Add other names as a comma-separated list
      - DNS_ALT_NAMES=puppet,puppet.service.consul,${DNS_ALT_NAMES:-}
      - CONSUL_ENABLED=true
      - CONSUL_IP
      - PUPPETDB_SERVER_URLS=https://puppetdb:8081
    depends_on:
      - consul
    dns:
      - $CONSUL_IP
    volumes:
      - ./volumes/code:/etc/puppetlabs/code/
      - ./volumes/puppet:/etc/puppetlabs/puppet/
      - ./volumes/serverdata:/opt/puppetlabs/server/data/puppetserver/

  postgres:
    environment:
      - POSTGRES_PASSWORD=puppetdb
      - POSTGRES_USER=puppetdb
      - POSTGRES_DB=puppetdb
    expose:
      - 5432
    # volumes can only be overriden by *target*, not *source*
    # so define them in .override.yml and .windows.yml since target varies
    # either /var/lib/postgresql/data/ on Linux or c:\data on Windows
    # https://docs.docker.com/compose/extends/#adding-and-overriding-configuration

  puppetdb:
    hostname: puppetdb
    image: puppet/puppetdb
    environment:
      - PUPPETDB_PASSWORD=puppetdb
      - PUPPETDB_USER=puppetdb
      - CONSUL_ENABLED=true
      - CONSUL_IP
    ports:
      - 8080
      - 8081
    depends_on:
      - postgres
      - puppet
    dns:
      - $CONSUL_IP
    volumes:
      - ./volumes/puppetdb/ssl:/etc/puppetlabs/puppet/ssl/

networks:
  default:
    ipam:
      config:
        - subnet: $PUPPERWARE_SUBNET

docker-compose.windows.yml

version: '3'

services:
  puppet:
    networks:
      default:
        aliases:
         - puppet.${AZURE_DOMAIN}

  postgres:
    image: stellirin/postgres-windows:9.6
    environment:
      - PGDATA=c:\data
    volumes:
      - ./volumes/puppetdb-postgres/data:c:\data
      - ./postgres-custom:c:\docker-entrypoint-initdb.d

If anything stands out for anyone, I would appreciate hearing about it! Thanks!

microsoft / hcsshim