search

docker / for-win

Bug reports for Docker Desktop for Windows
https://www.docker.com/products/docker#/windows
1.86k stars 290 forks source link

How to debug container that fails to start #676

Open gregpakes opened 7 years ago

gregpakes commented 7 years ago

Expected behavior

I am trying to start a container and I get the following error:

ERROR: for XXX Cannot start service XXX: container ba3f80d447b30049d074e9b3ae99db0b9e9c1b28ed75aa0d09f09b6c9dc68a03 encountered an error during Start: failure in a Windows system call: The compute system exited unexpectedly. (0xc0370106)

How can I get more logs as to why the container failed to start?

Actual behavior

Container fails to start

Information

The only logs I can find:

[16:14:12.445][WindowsDockerDaemon][Info   ] time="2017-04-26T16:14:12.445699100+10:00" level=debug msg="Result: {\"Error\":-1070137082,\"ErrorMessage\":\"The compute system exited unexpectedly.\"}" 
[16:14:12.445][WindowsDockerDaemon][Info   ] time="2017-04-26T16:14:12.445699100+10:00" level=error msg="libcontainerd: failed to start container: container ba3f80d447b30049d074e9b3ae99db0b9e9c1b28ed75aa0d09f09b6c9dc68a03 encountered an error during Start: failure in a Windows system call: The compute system exited unexpectedly. (0xc0370106)"
friism commented 7 years ago

Can you please post docker version and docker info and what container you're trying to run?

gregpakes commented 7 years ago

Output from docker info

PS C:\Users\greg.pakes> docker info
Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 11
Server Version: 17.03.1-ce
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: l2bridge l2tunnel nat null overlay transparent
Swarm: inactive
Default Isolation: hyperv
Kernel Version: 10.0 15063 (15063.0.amd64fre.rs2_release.170317-1834)
Operating System: Windows 10 Pro
OSType: windows
Architecture: x86_64
CPUs: 8
Total Memory: 31.82 GiB
Name: iSAMS-LAP-GREG3
ID: H5EZ:EK42:BS6Y:FC3X:TRWA:YDP5:5RCT:ZZ3U:VJ7K:GTOC:C4Y5:ZDXK
Docker Root Dir: C:\ProgramData\docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: -1
 Goroutines: 17
 System Time: 2017-04-27T07:25:16.2773095+10:00
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8

Output from docker version

Client:
 Version:      17.03.1-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Tue Mar 28 00:40:02 2017
 OS/Arch:      windows/amd64

Server:
 Version:      17.03.1-ce
 API version:  1.27 (minimum version 1.24)
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Tue Mar 28 00:40:02 2017
 OS/Arch:      windows/amd64
 Experimental: true

The container is built from a dockerfile based on microsoft/iis. The dockerfile simply does some IIS configuration. The docker build command completes successfully, but the container will not start.

Note: The container starts fine if I don't do the IIS config.

gregpakes commented 7 years ago

This issue is still happening and makes docker basically unusable.

I have narrowed the issue down to the volumes command in docker compose. If I try and map a directory on the host machine to the container, I will get this issue 90% of the time. But if I just keep on trying (change nothing), it will eventually work.

So it takes me about 30 mins to get my containers up and running.

The real issue is the "black-box ness". I simply have no idea how to debug this. The logs are basically useless.

gregpakes commented 7 years ago

Just for clarity, I am aware of other issues with mapping volumes and this does not seem related.

tusharm commented 7 years ago

Hi,

I see the same error on my Windows 10 machine. I am using docker-compose; when the volume mounts are commented out, containers start. Otherwise, they fail most of the time with the error given by the OP. This is how my compose file looks like:

version: '3.3'

services:
  monolith:
    container_name: monolith
    image: monolith-build:latest
    build:
      context: ./build
    volumes:
      - "C:\\dev\\Project\\app:C:\\website\\app"
    ports:
      - "80"
      - "8172" 
      - "4020-4021"

Output from docker version:

Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:40:09 2017
 OS/Arch:      windows/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.24)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:50:27 2017
 OS/Arch:      windows/amd64
 Experimental: true

Output from docker info:

Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 13
Server Version: 17.09.0-ce
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: ics l2bridge l2tunnel nat null overlay transparent
 Log: awslogs etwlogs fluentd json-file logentries splunk syslog
Swarm: inactive
Default Isolation: hyperv
Kernel Version: 10.0 15063 (15063.0.amd64fre.rs2_release.170317-1834)
Operating System: Windows 10 Pro
OSType: windows
Architecture: x86_64
CPUs: 8
Total Memory: 15.9GiB
Name: tusharm-bc-mbp
ID: SX3I:HWQZ:JQ6W:ZZ5C:7LDO:YJ2C:IPBD:A6RN:LOFP:DDBV:EBLA:T5CD
Docker Root Dir: C:\ProgramData\Docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: -1
 Goroutines: 26
 System Time: 2017-10-17T14:54:10.7328206+11:00
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
zlepper commented 6 years ago

Also seeing this problem here.

Docker info:

Containers: 2
 Running: 1
 Paused: 0
 Stopped: 1
Images: 10
Server Version: 17.09.0-ce
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: ics l2bridge l2tunnel nat null overlay transparent
 Log: awslogs etwlogs fluentd json-file logentries splunk syslog
Swarm: inactive
Default Isolation: hyperv
Kernel Version: 10.0 16299 (16299.15.amd64fre.rs3_release.170928-1534)
Operating System: Windows 10 Pro
OSType: windows
Architecture: x86_64
CPUs: 8
Total Memory: 47.95GiB
Name: DESKTOP-SQ4EPKR
ID: N42W:AHEJ:TYKI:OQ6T:DLPC:PWUA:YRSH:EZ75:HTMI:B74L:I2DO:OC2O
Docker Root Dir: C:\ProgramData\Docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: -1
 Goroutines: 59
 System Time: 2017-11-05T13:43:00.4276253+01:00
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Docker version:

Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:40:09 2017
 OS/Arch:      windows/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.24)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:50:27 2017
 OS/Arch:      windows/amd64
 Experimental: true

Dockerfile for troublesome container:

FROM microsoft/iis:nanoserver

WORKDIR C:/inetpub/wwwroot

VOLUME //C:/inetpub/wwwroot
ghost commented 6 years ago

I got the same problem on Windows 10 with nanoserver based images. It disappeared as soon as i moved all VOLUME statements to the end of the Dockerfile.

atrauzzi commented 6 years ago

Experiencing this as well any time my compose file has volumes.

christianacca commented 6 years ago

Same here with docker-compose

docker-compose.yml

version: '3.4'

services:

  series5-db:
    image: microsoft/mssql-server-windows-express
    environment:
      - sa_password=This1sthesapassword1
      - ACCEPT_EULA=Y
    volumes:
      - db-data:C:\data
    networks:
      - series5-net

  series5-spa:
    build: .
    image: series5/spa
    ports:    
      - "8080:80"
      - "877:877/udp"
    volumes:
      - C:\Git\Series5\output\web-logs:C:\inetpub\sites\Series5\Spa\App_Data\logs
    depends_on:
      - series5-db
    networks:
      - series5-net

volumes:
  db-data:
  # web-logs:

networks:
  series5-net:
    driver: nat

Environment

docker info output

Containers: 5
 Running: 2
 Paused: 0
 Stopped: 3
Images: 40
Server Version: 17.12.0-ce
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: ics l2bridge l2tunnel nat null overlay transparent
 Log: awslogs etwlogs fluentd gelf json-file logentries splunk syslog
Swarm: inactive
Default Isolation: hyperv
Kernel Version: 10.0 16299 (16299.15.amd64fre.rs3_release.170928-1534)
Operating System: Windows 10 Pro
OSType: windows
Architecture: x86_64
CPUs: 4
Total Memory: 7.886GiB
Name: W10CCROWHURSTLT
ID: 3EQ5:TYLN:ZC43:DTZF:CZAP:WGT7:DM4F:VEHM:ETXJ:MQY7:ELB7:Y6GR
Docker Root Dir: C:\ProgramData\Docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: -1
 Goroutines: 44
 System Time: 2018-01-13T16:20:09.4193165Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

docker version output

Client:
 Version:       17.12.0-ce
 API version:   1.35
 Go version:    go1.9.2
 Git commit:    c97c6d6
 Built: Wed Dec 27 20:05:22 2017
 OS/Arch:       windows/amd64

Server:
 Engine:
  Version:      17.12.0-ce
  API version:  1.35 (minimum version 1.24)
  Go version:   go1.9.2
  Git commit:   c97c6d6
  Built:        Wed Dec 27 20:15:52 2017
  OS/Arch:      windows/amd64
  Experimental: true
christianacca commented 6 years ago

Just found the reason for the failure in my specific case: I accidentally baked into the image a non-empty folder that my volume was then mapping to.

This issue still stands however: how to debug this type of generic failure?

docker-robott commented 6 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale comment. Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle stale

dtretyakov commented 6 years ago

/lifecycle frozen

dazinator commented 6 years ago

I am experiencing this issue - I have no mounted volumes. I have rebuilt the image with docker-compose build --no-cache --force-rm

docker-compose up Removing infrastructuredefinition_teamcityagent1_1 Recreating d9f5e27c9409_infrastructuredefinition_teamcityagent1_1 ... error

ERROR: for d9f5e27c9409_infrastructuredefinition_teamcityagent1_1 Cannot start service teamcityagent1: container 62835df52fee3048ca562a67ef1ff815cecb144a96e61bb652cc461d592ca353 encountered an error during Start: failure in a Windows system call: The compute system exited unexpectedly. (0xc0370106)

ERROR: for teamcityagent1 Cannot start service teamcityagent1: container 62835df52fee3048ca562a67ef1ff815cecb144a96e61bb652cc461d592ca353 encountered an error during Start: failure in a Windows system call: The compute system exited unexpectedly. (0xc0370106)

My docker-compose.yml:

version: '2.2'
services:  
  teamcityagent1:  
    build: 
      context: .
      dockerfile: './TeamCityBuildAgent/DOCKERFILE'
    image: reach_teamcitybuildagent  
    restart: always
    # privileged: true # not supported on windows yet but this would allow docker in docker scenario which build agent would find useful.
    environment:
      SERVER_URL: https://teamcity.foo.co.uk:444
      #AGENT_NAME: dockerhost1   
    #volumes:         
      #- C:/DockerMount/TeamCityAgent:C:/BuildAgent/conf
      #- C:/DockerMount/sources:C:/sources       
    mem_limit: 2g
dazinator commented 6 years ago

Ok so my issues seems transient, as once I did all of the following it cleared:

  1. Restarted docker
  2. Freed disk space
  3. Re build the image

I have no idea what the problem was.

Pete1138 commented 6 years ago

/remove-lifecycle stale

Pete1138 commented 6 years ago

I have a very simple console app and docker file, but when I run it I get "encountered an error during CreateProcess: failure in a Windows system call: the system cannot find the file specified". I don't know how to get any more information (such as which file is missing).

FROM microsoft/windowsservercore
WORKDIR /service
COPY /bin .
ENTRYPOINT ["dotnet","TestService.exe"]
EXPOSE 8888
ctgarvey commented 6 years ago

I am experiencing this issue.:

I can successfully run a plain vanilla container based on microsoft/mssql-server-windows-express no problem, and can successfully build a new image with a dockerfile as per below. But i cannot get my new container to run. It bombs out with: Error response from daemon: container 294ee2e229db73591aa0b0a1d6bb34715d6caa75bc905b6f6ad0c880f73851d5 encountered an error during Start: failure in a Windows system call: The compute system exited unexpectedly. (0xc0370106).

Im not using mount. I am a bit confused about the interplay between the original CMD in the microsoft dockerfiles, so i just copied that again ain an attempt to get container running , as my new Dockerfile needs a CMD statement:

CMD .\start -sa_password $env:sa_password -ACCEPT_EULA $env:ACCEPT_EULA -Verbose

Dockerversion: docker info Containers: 17 Running: 1 Paused: 0 Stopped: 16 Images: 91 Server Version: 18.03.1-ce Storage Driver: windowsfilter (windows) lcow (linux) Windows: LCOW: Logging Driver: json-file Plugins: Volume: local Network: ics l2bridge l2tunnel nat null overlay transparent Log: awslogs etwlogs fluentd gelf json-file logentries splunk syslog Swarm: inactive Default Isolation: hyperv Kernel Version: 10.0 17134 (17134.1.amd64fre.rs4_release.180410-1804) Operating System: Windows 10 Enterprise N OSType: windows Architecture: x86_64 CPUs: 12 Total Memory: 15.9GiB Name: Teamcity ID: I7KV:EYIG:QRQ4:7YNI:RJZD:GWNZ:NNML:XC5A:CDDS:RBG5:YBM6:VXDH Docker Root Dir: C:\ProgramData\Docker Debug Mode (client): false Debug Mode (server): true File Descriptors: -1 Goroutines: 93 System Time: 2018-06-29T15:41:31.4975322+01:00 EventsListeners: 0 Registry: https://index.docker.io/v1/ Labels: Experimental: true Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

Windows version: windows 10 enterprise N 1803

Docker file:

escape=`

FROM microsoft/mssql-server-windows-express:latest AS XXXSQL SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]

ARG SPIKE_VERSION

ARG PowershellScriptPath

ARG BuildWorkingDirectory

ARG SpikeDBConnInitialCatalog

ARG SpikeDBconnIntegratedSecurity

ARG SpikeDBConnUserId

ARG SpikeDBConnPassword

ARG SpikeDBConnectionString

ARG SpikeSisterIISContainerIP

ARG SPIKE_VERSION=1.66 ARG PowershellScriptPath=c:\powershellscripts\ ARG BuildWorkingDirectory=e:\spikecontainerstaging\sqlcontainer\ ARG SpikeDBConnInitialCatalog=spike6687 ARG SpikeDBconnIntegratedSecurity=False ARG SpikeDBConnUserId=sa ARG SpikeDBConnPassword=SpikeXXXXXX999! ARG SpikeDBConnectionString="NotUsed" ARG SpikeSisterIISContainerIP=127.0.0.1 ARG Spikeattach_dbs="not applicable"

label image ID

LABEL name = "XXXSQL" LABEL version="0.0.1-beta" LABEL vendor="XXX" LABEL release-date="08-06-2018" LABEL containerID = "TBA"

set environment variables

ENV sa_password=${SpikeDBConnPassword} ACCEPT_EULA=Y SPIKE_VERSION=${SPIKE_VERSION} attach_dbs=${Spikeattach_dbs} env_SPIKE_VERSION=${SPIKE_VERSION} env_PowershellScriptPath=${PowershellScriptPath} env_BuildWorkingDirectory=${BuildWorkingDirectory} env_SpikeDBConnInitialCatalog=${SpikeDBConnInitialCatalog} env_SpikeDBconnIntegratedSecurity=${SpikeDBconnIntegratedSecurity} env_SpikeDBConnUserId=${SpikeDBConnUserId} env_SpikeDBConnPassword=${SpikeDBConnPassword} env_SpikeDBConnectionString=${SpikeDBConnectionString} env_SpikeSisterIISContainerIP=${SpikeSisterIISContainerIP}

RUN Get-Childitem Env:

DOCKER expose these external public ports to the IIS container running on the same container default network and the host for SQL admin

EXPOSE 1433 1434 3389 5985 5986 EXPOSE 1433/udp 1434/udp 5985/udp 5986/udp

set working directory in containers - needs \ !

WORKDIR c://

RUN Get-Location ; New-Item -Path c:\sqlserverdbbackup -Type Directory -Force; New-Item -Path c:\dockerpowershellscripts -Type Directory -Force; New-Item -Path c:\restoredsqldatabase -Type Directory -Force; New-Item -Path c:\sqlserverscripts -Type Directory -Force;

WORKDIR c://

COPY ["./DockerPowershellScripts/" , "/dockerpowershellscripts/"] COPY ["./SQLserverDBBackup/" , "/sqlserverdbbackup/"] COPY ["./SQLServerScripts/*" , "/sqlserverscripts/"]

ENV sa_password="XXXX9999999!" RUN Get-Childitem Env: ;

this line is essentially copied form the original MS image dockerfiles, except with attach_dbs removed.

CMD .\start -sa_password $env:sa_password -ACCEPT_EULA $env:ACCEPT_EULA -Verbose

cowlinator commented 5 years ago

I was also having this exact issue: docker: Error response from daemon: container XYZ encountered an error during Start: failure in a Windows system call: The compute system exited unexpectedly. (0xc0370106).

I found 2 solutions for my case:

jibinjose087 commented 5 years ago

I tried with windowsservercore:1709 image aswell but getting the same error while trying to start the container.

C:\Program Files\Docker\Docker\Resources\bin\docker.exe: Error response from daemon: container 12345678edbfert encountered an error during Start: failure in a Windows system call: The compute system exited unexpectedly. (0xc0370106)

dockerfile

FROM mcr.microsoft.com/windows/servercore:1709 SHELL ["powershell", "-Command"] WORKDIR /application COPY . . VOLUME C:/application CMD cmd

OS: Windows 10 Docker: Docker for windows 2.0.0

Any solution for this issue?

cowlinator commented 5 years ago

@jibinjose087 , I was able to repro your particular case, and I was able to fix it by changing COPY . . to COPY * /

jibinjose087 commented 5 years ago

@cowlinator Thank you for your quick reply, I tried the solution that you have mentioned and I am able to start the container and mount the volume only if it is empty. If the volume in our case C:/application contains folders or files then I am getting the following error.

========= C:\Program Files\Docker\Docker\Resources\bin\docker.exe: Error response from daemon: container 1233445edfersbtre12121 encountered an error during Start: failure in a Windows system call: The compute system exited unexpectedly.

I have removed all the entries from docker file and added only few steps to identify the issue. The output of docker build and run is as follows.

Step 1/7 : FROM microsoft/windowsservercore Step 2/7 : SHELL ["powershell", "-Command"] Step 3/7 : WORKDIR /application Step 4/7 : COPY * /application/ Step 5/7 : ENV ABC='Test' Step 6/7 : VOLUME C:/application Step 7/7 : CMD cmd ---> 1234567abcd Successfully built 1234567abcd

PS C:\TestFolder> docker container run -it --name test -v "C:\TestData:C:/application" 1234567abcd C:\Program Files\Docker\Docker\Resources\bin\docker.exe: Error response from daemon: container 1233445edfersbtre12121 encountered an error during Start: failure in a Windows system call: The compute system exited unexpectedly. (0xc0370106). PS C:\TestFolder>

I tried with servercore:1709 also but getting the same error. I have tried all the possible combination of C:/application aswell but no luck. My objective is to create a windows docker image and need to mount one volume so I can have the output files on my localsystem. If you have some insights then please let me know

OS: Windows 10, Version 1709, OS Build: 16299

cowlinator commented 5 years ago

@jibinjose087 , for this particular instance, you are correct, it is because the volume mount target is not empty. Changing COPY * /application/ to COPY * /application2/ will fix the problem. (Or, you could entirely remove the COPY command, and place the host files that you want to be inside the container into C:\TestData instead).

It's unfortunate that docker waits until docker run to error, when the problem should have been evident at build time.

jibinjose087 commented 5 years ago

@cowlinator , Yes you are correct the volume mount happens only when the container folder is empty. In my case I am running test cases inside the container so as per the code /application/ABC/TestData folder contains some files which are required for running test cases and once the test is completed the results are also getting generated inside the same folder. If I am able to mount the folder /application/ABC/TestData locally then I can easily get the reports otherwise I need to login to container to get the data. The last line of the Dockerfile CMD command is

CMD nunit3-console.exe test.dll

so when we run the container it will get executed and I can't copy the files to any other folder before exiting the container Keeping data on host machine and changing the report generation folder is difficult due to some restrictions. Is there any alternative solution you can think of to solve this issue?

jplwood commented 5 years ago

I've isolated this error to being caused by trying to mount a volume against a folder in the container/image that already contains data. Supposedly this should still be possible re their documentation. I see people talking about issues with volumes related to this error, anyone have references to discussions about that?

Also ditto to the fact that we need a way to see internally what's causing this error!

EriksonBahr commented 5 years ago

My case was an environment variable declared as myVar theVarValue, changing to myVar=theVarValue made it.