Docker compose fails to start container with mapped volumes : Fails error during hcsshim::System::Start: failure in a Windows system call: The virtual machine or container exited unexpectedly. (0xc0370106)

markddrake commented 3 years ago

[X ] I have tried with the latest version of Docker Desktop
[X ] I have tried disabling enabled experimental features
[X ] I have uploaded Diagnostics
Diagnostics ID:ECFBB5DA-D5F0-468D-82B7-B62EA973BCCF/20211001065102

Actual behavior

Docker Compose fails to start container

C:\yadamu>docker-compose --file=docker\dockerfiles\docker-compose-windows.yml up -d
Creating network "YADAMU-NET" with the default driver
Creating volume "YADAMU_01-SHARED" with default driver
Creating volume "dockerfiles_MSSQL14-01-DATA" with default driver
Creating volume "dockerfiles_MSSQL14-01-LOG" with default driver
Creating MSSQL14-01 ... error

ERROR: for MSSQL14-01  Cannot start service MSSQL14-01: container a2e579e8e5e8aea455cff46f85793f6bd87601b5d444f51fe1ebe50f65ac919f encountered an error during hcsshim::System::Start: failure in a Windows system call: The virtual machine or container exited unexpectedly. (0xc0370106)

ERROR: for MSSQL14-01  Cannot start service MSSQL14-01: container a2e579e8e5e8aea455cff46f85793f6bd87601b5d444f51fe1ebe50f65ac919f encountered an error during hcsshim::System::Start: failure in a Windows system call: The virtual machine or container exited unexpectedly. (0xc0370106)
ERROR: Encountered errors while bringing up the project.

Expected behavior

Container should start

Information

version: "2.4"
services:

  MSSQL14-01: 
      container_name: MSSQL14-01
      image:  yadamu/mssql:2014
      mem_limit: '16gb'
      shm_size: '4gb'
      networks:
        - YADAMU-NET
      ports: 
        - "1435:1433"
      environment:
        ACCEPT_EULA: Y
        MSSQL_MEMORY_LIMIT_MB: 16384
        SA_PASSWORD: oracle#1
      volumes:
        - YADAMU_01-SHARED:c:\docker\shared
        - MSSQL14-01-DATA:C:\MSSQL12.MSSQLSERVER\MSSQL\DATA
        #- MSSQL14-01-LOG:C:\MSSQL12.MSSQLSERVER\MSSQL\LOG
      restart: always

networks: 
  YADAMU-NET:
     name: YADAMU-NET

volumes:
  YADAMU_01-SHARED:
    name: YADAMU_01-SHARED
#    driver: local
#    driver_opts:
#      type: none
#      o: bind
#     device: c:\docker\shared
  MSSQL14-01-DATA:
  MSSQL14-01-LOG:

Note if I comment out the line - MSSQL14-01-DATA:C:\MSSQL12.MSSQLSERVER\MSSQL\DATA the container will start

Windows Version: 10 21H1
Docker Desktop Version: 3.6.0
WSL2 or Hyper-V backend? Hyper V - Windows Container
Are you running inside a virtualized Windows e.g. on a cloud server or a VM: No

Steps to reproduce the behavior

See Above.

In my Image the folders C:\MSSQL12.MSSQLSERVER\MSSQL\DATA exist and contain data.

I am not sure how I can provide the 18G Image that is : yadamu/mssql:2014

lorenrh commented 3 years ago

Can you please share the output of docker-compose -v?

markddrake commented 3 years ago

FYI


C:\yadamu>docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  compose: Docker Compose (Docker Inc., v2.0.0-beta.6)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 2
  Running: 0
  Paused: 0
  Stopped: 2
 Images: 2
 Server Version: 20.10.7
 Storage Driver: windowsfilter (windows) lcow (linux)
  Windows:
  LCOW:
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics internal l2bridge l2tunnel nat null overlay private transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
 Swarm: inactive
 Default Isolation: hyperv
 Kernel Version: 10.0 19043 (19041.1.amd64fre.vb_release.191206-1406)
 Operating System: Windows 10 Pro Version 2009 (OS Build 19043.1237)
 OSType: windows
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.89GiB
 Name: YADAMU-DB3
 ID: YUMN:T4JT:IBBW:XTIK:EEGC:KQLR:AHI5:O2T7:NIW6:4HQ5:P2OY:MZLN
 Docker Root Dir: C:\ProgramData\Docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

C:\yadamu>
``

markddrake commented 3 years ago

C:\yadamu>docker-compose version
docker-compose version 1.29.2, build 5becea4c
docker-py version: 5.0.0
CPython version: 3.9.0
OpenSSL version: OpenSSL 1.1.1g  21 Apr 2020

C:\yadamu>

markddrake commented 3 years ago

Also, problem will reproduce with this docker compose file which uses an image from the docker repository

version: "2.4"
services:

  SS2014-01: 
      container_name: SS2014-01
      image:  dbafromthecold/sqlserver2014dev:sp2
      mem_limit: '16gb'
      shm_size: '4gb'
      networks:
        - YADAMU-NET
      ports: 
        - "1435:1433"
      environment:
        ACCEPT_EULA: Y
        MSSQL_MEMORY_LIMIT_MB: 16384
        SA_PASSWORD: oracle#1
      volumes:
        - YADAMU_01-SHARED:c:\docker\shared
        - SS2014-01-DATA:c:\Program Files\Microsoft SQL Server\MSSQL12.MSSQLSERVER\MSSQL\DATA
        - SS2014-01-LOG:c:\Program Files\Microsoft SQL Server\MSSQL12.MSSQLSERVER\MSSQL\Log
      restart: always

networks: 
  YADAMU-NET:
     name: YADAMU-NET

volumes:
  YADAMU_01-SHARED:
    name: YADAMU_01-SHARED
#    driver: local
#    driver_opts:
#      type: none
#      o: bind
#     device: c:\docker\shared
  SS2014-01-DATA:
  SS2014-01-LOG:

My original example was derived from this one, but attempted to remove the spaces from the target path names in case they were the cause of the error.

C:\yadamu>docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

C:\yadamu>docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

C:\yadamu>docker images
REPOSITORY                        TAG       IMAGE ID       CREATED         SIZE
yadamu/mssql                      2014      e6e12e199f9a   9 hours ago     18.9GB
mcr.microsoft.com/windows         2004      9e30cb60b773   12 months ago   12.9GB
dbafromthecold/sqlserver2014dev   sp2       d1dfe544c4fc   4 years ago     18.7GB

C:\yadamu>docker-compose --file=docker\dockerfiles\docker-compose-ss2014.yml up -d
Creating SS2014-01 ... error

ERROR: for SS2014-01  Cannot start service SS2014-01: container dd14ba823dff5264cdfbf659313fdfc6dd7a25235d5afa53a9eb8e3d1492e3a1 encountered an error during hcsshim::System::Start: failure in a Windows system call: The virtual machine or container exited unexpectedly. (0xc0370106)

ERROR: for SS2014-01  Cannot start service SS2014-01: container dd14ba823dff5264cdfbf659313fdfc6dd7a25235d5afa53a9eb8e3d1492e3a1 encountered an error during hcsshim::System::Start: failure in a Windows system call: The virtual machine or container exited unexpectedly. (0xc0370106)
ERROR: Encountered errors while bringing up the project.

markddrake commented 3 years ago

I think this error was caused by the fact the SQLSERVER service was already running when DOCKER attempted to mount the volumes.... I have got round it by stopping the service until the volumes have been copied.

Maximys commented 2 years ago

@markddrake , can you provide more detailed info about applied steps for fix this issue?

markddrake commented 2 years ago

It's an Ugly Hack... There appears to be a second problem which is on a Windows container the original contents of a volume mount are NOT copied to the mounted volume the first time the volume is mounted. Proceeding under that assumption here's my docker file..

# escape=`
FROM yadamu/svr-2019:vs22
#
RUN New-Item -ItemType "directory" C:\INSTALL; `
    New-Item -ItemType "directory" C:\INSTALL\CU
#
# Unpack SQL Server
#
WORKDIR c:/INSTALL
COPY SQLServer2019-DEV-x64-ENU.exe .
COPY SQLServer2019-DEV-x64-ENU.box .
RUN Start-Process .\SQLServer2019-DEV-x64-ENU.exe  -Wait -ArgumentList """/x:SQLServer2019-DEV-x64-ENU""", """/u"""; `
    Remove-Item SQLServer2019-DEV-x64-ENU.exe; `
    Remove-Item SQLServer2019-DEV-x64-ENU.box
#
# Install SQL Server RTM
#
RUN C:\INSTALL\SQLServer2019-DEV-x64-ENU\SETUP.exe /Q /ACTION=Install /UPDATEENABLED=FALSE /FEATURES=SQLEngine /INSTANCENAME=MSSQLSERVER /SQLSVCACCOUNT='NT AUTHORITY\System' /SQLSYSADMINACCOUNTS='BUILTIN\ADMINISTRATORS' /TCPENABLED=1 /NPENABLED=0 /IACCEPTSQLSERVERLICENSETERMS=1 /SECURITYMODE=SQL /SAPWD="""oracle#1""" 
#
WORKDIR c:/
COPY prepareMounts.ps1 .
RUN Set-Service  -Name """MSSQLSERVER""" -StartupType """Manual"""; `
    Stop-Service -Name """MSSQLSERVER"""; `
    c:\prepareMounts.ps1; `
    Remove-Item c:\prepareMounts.ps1
#
COPY Start.ps1 .
#ENTRYPOINT [ "powershell","c:\\Start.ps1","-ACCEPT_EULA $ENV:ACCEPT_EULA","-SA_PASSWORD $ENV:SA_PASSWORD"]
ENTRYPOINT powershell "c:\\Start.ps1 -ACCEPT_EULA $ENV:ACCEPT_EULA -SA_PASSWORD $ENV:SA_PASSWORD
#ENTRYPOINT ["powershell","wait-event"]

The first part of the workaround is to set the startup type of the service to manual, stop the service and then run prepareMounts.ps1. prepareMounts.ps1 is shown below

Compress-Archive -Path 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA' -DestinationPath 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\Data.zip'
Compress-Archive -Path 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\Log' -DestinationPath 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\Log.zip'
rmdir -Recurse  'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA'
rmdir -Recurse  'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\Log'

All it does is zip up the contents of the folders I intend to mount and stores them somewhere outside the of the locations that will become volumes. In this case I am creating volumes for the Data and Log folders.

When the container starts it executes startup.ps1

# start the service

$STATUS = (Get-Service MSSQLSERVER).StartType
Write-Output "MSSQLSERVER: Start Type $STATUS"

if ($STATUS -eq "Manual") {
  Write-Output "MSSQLSERVER: Restoring DATA and Log Folders"
  Expand-Archive -Path "C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA.zip" -DestinationPath "C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\"  
  Expand-Archive -Path "C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\Log.zip"  -DestinationPath "C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\"  
  Write-Output "MSSQLSERVER: Restored DATA and Log Folders"
  Set-Service -Name "MSSQLSERVER" -StartupType "Automatic"
  Write-Output "MSSQLSERVER: Set StartType Automatic"
  Write-Output "MSSQLSERVER: Starting."
  start-service MSSQLSERVER
}

Write-Output "MSSQLSERVER: Started."

if($sa_password -ne "_")
{
    Write-Output "MSSQLSERVER: Updating sa credentials."
    $sqlcmd = "ALTER LOGIN sa with password=" +"'" + $sa_password + "'" + ";ALTER LOGIN sa ENABLE;"
    & sqlcmd -Q $sqlcmd
}

Write-Output "MSSQLSERVER: Container Ready. Monitoring Logs."

$lastCheck = (Get-Date).AddSeconds(-2) 
while ($true) 
{ 
    Get-EventLog -LogName Application -Source "MSSQL*" -After $lastCheck | Select-Object TimeGenerated, EntryType, Message   
    $lastCheck = Get-Date 
    Start-Sleep -Seconds 2 
}

If the service startup state is manual, it unzips the archives created by prepareMounts.ps1 into the newly mounted volumes, sets the service startup back to automatic, and starts the service. This appears to be working for my use case.

markddrake commented 2 years ago

The full set of scripts can be found here..

https://github.com/markddrake/YADAMU---Yet-Another-DAta-Migration-Utility/tree/master/docker/rdbms/mssql/docker/windows

docker-robott commented 2 years ago

Issues go stale after 90 days of inactivity. Mark the issue as fresh with /remove-lifecycle stale comment. Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle stale

markddrake commented 2 years ago

I suspect the issue appears to be caused by a service starting and locking the mount point before docker can mount the volume and copy the content. If that is true most cases could be worked around by delaying service startup

docker-robott commented 2 years ago

Closed issues are locked after 30 days of inactivity. This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle locked

docker / for-win