Docker stop working after a Windows Backup...

Stonewall70 commented 4 years ago

Docker stop working after a Windows Backup. Docker works perfect until the backup starts.

OS: Windows Server 2019 Essentials Docker: Docker Desktop for Windows stable version 2.4.0.0

Reproducible: Yes

Works before: Docker Desktop for Windows stable version 2.1.0.5 (latest version it works on)

Error: docker container ls -a error during connect: Get http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.40/containers/json?all=1: open //./pipe/docker_engine: The system cannot find the file specified. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.

docker-robott commented 3 years ago

Issues go stale after 90 days of inactivity. Mark the issue as fresh with /remove-lifecycle stale comment. Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle stale

Stonewall70 commented 3 years ago

The same issue exist in version Docker Desktop Community 3.0.0 for Windows (Linux containers).

vargasisaac1982 has the same problem with Windows server 2016 standard and Linux containers. For more information see: https://forums.docker.com/t/docker-stop-working-on-windows-server-backup-activation/82949

/remove-lifecycle stale

maxiride commented 3 years ago

I'm having the same error message and just stumbled upon this issue. I initially followed the lead on this ticker #4495 but after investigating I can finally reduce the variables involved and confirm that also for me Windows Backup is breaking Docker CE. Just after the backup process ends Docker is unresponsive throwing the "file not found" error. I wonder if also all the people in the other ticket have backups running.

Docker version 20.10.5, build 55c4c88 running using Hyper-V and not WSL
docker-compose version 1.28.5, build c4eb3a1f
Windows Server 2019 Essential

I'm really not good at navigating the Event log of windows but I found this error Hyper-V related following the backup based on the timestamps:

DockerDesktopVM
E96510AD-9D30-4460-989D-30D57F7DD87B
%%2147946719
0x800710DF

The very crude workaround has been to set a scheduled task to kill and start Docker Desktop each night after the backup is complete.

danielvandenberg95 commented 3 years ago

The very crude workaround has been to set a scheduled task to kill and start Docker Desktop each night after the backup is complete.

@maxiride Could you possibly post the configuration for your task here? I've tried to do the same but haven't managed to find the right commands. Something always seems to keep running with me. If it would be possible perhaps even export your tasks and drop the XMLs here?

maxiride commented 3 years ago

@danielvandenberg95 Appended to the XMLs the extension .txt as Github doesn't support attaching .xml, just rename the files back. The kill task invokes a bat file which I provided below.

Is it unsafe? Absolutely Yes.
Should anyone do it? No.
Why I do it? There is only one non-critical non-essential container running and the service has been configured to halt its tasks a bit before the scheduled task so that it isnt killed in the middle of an operatio.
I hope this issue get some attention by the maintainer, at least to guide us on how to provide better debug information on the case.

Docker Kill.xml.txt Docker Start.xml.txt

bat kill

taskkill /IM "Docker Desktop.exe" /F

docker-robott commented 3 years ago

Issues go stale after 90 days of inactivity. Mark the issue as fresh with /remove-lifecycle stale comment. Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle stale

danielvandenberg95 commented 3 years ago

Still an issue. /remove-lifecycle stale /lifecycle frozen

averkiev commented 2 years ago

I see this issue as well. Docker is running on Windows Server 2022. Docker Desktop version 4.2.0 /remove-lifecycle stale

protivakid commented 2 years ago

Still an issue for me as well on Server 2016 and starting Server backup. If I try "docker container ls" I see my containers are still up and reporting as healthy however I am unable to access the containers ports from the host. Restarting the server after solves the issue but each night that server backup starts I encounter this again. Need a fix!!

And just to clarify, for me it doesn't look like docker actually goes down, it looks like the port forwarding from the host to the containers is what crashes as soon as Windows/Server Backup begins. When opening the Docker Logs I see this:

[21:25:28.072][VpnKit ][Error ] vpnkit.exe: Flow read_thread caught: An established connection was aborted by the software in your host machine. [21:25:28.073][VpnKit ][Error ] vpnkit.exe: Flow read_thread caught: An established connection was aborted by the software in your host machine. [21:25:28.073][VpnKit ][Error ] vpnkit.exe: Socket.read: An established connection was aborted by the software in your host machine. [21:25:28.073][VpnKit ][Error ] vpnkit.exe: port-forwarding: EOF reading frame length [21:25:28.073][VpnKit ][Error ] vpnkit.exe: port-forwarding: dispatcher shutting down [21:25:28.073][VpnKit ][Error ] vpnkit.exe: Socket.read: An established connection was aborted by the software in your host machine. [21:25:28.073][VpnKit ][Error ] vpnkit.exe: port-forwarding: EOF reading frame length [21:25:28.073][VpnKit ][Error ] vpnkit.exe: port-forwarding: dispatcher shutting down [21:25:28.073][VpnKit ][Error ] vpnkit.exe: Vmnet.Server.listen: read EOF so closing connection [21:25:28.073][VpnKit ][Info ] vpnkit.exe: Vmnet.Server.disconnect [21:25:28.073][VpnKit ][Info ] vpnkit.exe: Vmnet.Server.listen returning Ok() [21:25:28.073][VpnKit ][Info ] vpnkit.exe: TCP/IP stack disconnected [21:25:29.242][Moby ][Info ] [ 182.741856] hv_utils: TimeSync IC version 4.0 [21:25:32.969][Moby ][Info ] [ 186.435894] sd 0:0:0:0: [storvsc] Sense Key : 0x6 [current] [21:25:32.990][Moby ][Info ] [ 186.477283] sd 0:0:0:0: [storvsc] ASC=0x3f ASCQ=0x2 [21:25:33.177][Moby ][Info ] [ 186.498057] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automa [21:25:58.208][VpnKit ][Error ] vpnkit.exe: Socket.Stream: caught Forwarder.Multiplexer.Make(Flow).Multiplexer_failed

After this I have to restart docker to be able to reach the containers through the defined ports.

/remove-lifecycle stale /lifecycle frozen

rtuite commented 1 year ago

I'm also having issues. It appears that Docker is crashing when my nightly backup job runs (Windows backup in my home lab). I noticed this morning that my Docker Desktop "Containers" tab was empty, the container I set up last night was gone. I tried to restart Docker but got an error that it could not be started. I checked and Windows backup was still running. I waited for it to complete, restarted docker, and my container returned and started like normal.

It looks like when it runs the Hyper-V backup job it causes the virtualization to crash. Not sure if anything can be done about it aside from the scuffed stop/start suggestion earlier.

protivakid commented 1 year ago

I'm also having issues. It appears that Docker is crashing when my nightly backup job runs (Windows backup in my home lab). I noticed this morning that my Docker Desktop "Containers" tab was empty, the container I set up last night was gone. I tried to restart Docker but got an error that it could not be started. I checked and Windows backup was still running. I waited for it to complete, restarted docker, and my container returned and started like normal.

It looks like when it runs the Hyper-V backup job it causes the virtualization to crash. Not sure if anything can be done about it aside from the scuffed stop/start suggestion earlier.

I ended up writing a little script that would wait for the backup to complete and then restart the whole server. Docker would then re-start on boot. Happy to share if that helps anyone.

maxiride commented 1 year ago

@protivakid please share how you scripted the reboot, it would help a lot.

protivakid commented 1 year ago

Here is a hastily written copy of my powershell script that you can pull apart to suit your needs. Basically you schedule a task to run the script on system boot. It sits idle for a bit and then starts checking that the docker container is up by hitting the service the container is running.

The magic happens when it fails to get a response and then runs the command "wbadmin GET STATUS". This command will print the output of the backup progress the whole time the server backup is running. This prevents the script from checking over and over while the backup runs AND prevents the server from restarting too soon before the backup can finish.

After the backup is done, the script will check again and if the container is still down will reboot the server. Upon re-boot the script will check again and if the container is STILL down after reboot the script will end. This prevents it from restarting the server over and over.

You will need to modify to suit your needs and there are probably better ways to handle this but it works for me. I'm not a powershell expert but this got the job done. It uses statuscake to alert me if things go down and logs all events to a file on the server. DockerCheckScript.txt

docker / for-win

Docker stop working after a Windows Backup... #8948