Windows Update KB4074588 causes permission denied on port mapping

[sarvasana]

Expected behavior

When I start a docker container, using docker run, I expect it to be able to start and map ports to my machine.

Actual behavior

It fails with the following error:

73eb86324647935af64b41e4b2015bc1460425a71eeb8e6982e601fb7d6cfabc docker: Error response from daemon: driver failed programming external connectivity on endpoint (011417fa62b227b7b1e154e6dd120e4830e5716a050dc7b5d4fa925b28468326): Error starting userland proxy: Bind for 0.0.0.0:27017: unexpected error Permission denied.

Diagnostics: D669B56E-9D5A-448F-97BE-1654EC3EB6CE/2018-02-17_11-39-34

Information

Windows Update KB4074588 https://support.microsoft.com/en-us/help/4074588/windows-10-update-kb4074588

Steps to reproduce the behavior

1. Install KB4074588
2. Start a container (this one had mongodb on port 27017)
3. See the message
4. Uninstall KB4074588
5. Start the container again
6. See the container running.

[JonasJes]

I had same problem today, but no error message when starting a container. I just couldn't connect to containers using localhost. Until I found that KB4074588 was installed today, i removed it and rebooted, and now it works.

[rickbutton]

Confirmed for me too. Uninstalling KB4074588 fixed the problem for me. I didn't see an error when I created the container.

For anyone else, is this the normal level of stability when working with Docker on Windows? We switched to using a docker container for a dependency in dev literally today and it has been rough.

[carlfischer1]

@rickbutton in general we would not expect issues with Patch Tuesday updates. Can you / @youfoundkris upload a diagnostic? (Diagnose and Feedback from the whale tray icon)

cc @PatrickLang @gtardif

[rzd-lumo]

Not just docker problems. This KB4074588 update caused Window to reject almost every type of USB peripheral or network printer that I tried to attach to my laptop. The USB HW (including a genuine Microsoft keyboard and mouse) were fine but Device Manager had a cryptic error message stating driver was not available. I uninstalled KB4074588 and everything went back to normal. However, now Windows update is trying to force me to reinstall KB4074588.

[sarvasana]

@carlfischer1 @PatrickLang @gtardif

D669B56E-9D5A-448F-97BE-1654EC3EB6CE/2018-02-17_11-39-34

[JonasJes]

@rzd-lumo https://www.top-password.com/blog/block-specific-updates-in-windows-10/

[rzd-lumo]

Thanks for the info on how to block the updates. This will come in handy in the future. The really strange thing is that the second time KB4074588 was installed it did not cause any damage. All my devices and drivers were recognized and they are working properly. The mysteries of Microsoft's updates never cease to amaze me. I hope your docker problems get resolved just as easily.

[JonasJes]

@rzd-lumo Also after a reboot of Docker or the PC?

[sarvasana]

I have re-installed the KB and it breaks again after a reboot. That is how I managed to produce the diagnostic. I have disabled windows update completely for now.

[pgayvallet]

Hi

Thanks for the feedback.

We are actively looking for a workaround / fix right now.

[mjohnson007us]

We the same issues on many PCs in our corporation. All have been USB HID issues with the mnouse and KB not working. They show up as other devices. we remote into the PC and manually load the files from another folder location NOT the driverstore and so far we can gett he mice and KB working. But this is a major bug and if the restore doesn't work there are many out of luck. Hopefully MS identifies and fixes the bug.

[mlmeyers]

I have been encounter a similar issue with port forwarding going out to lunch with this update.

Odd that I do not get the permission error, but suspect it is related to the same.

https://github.com/docker/for-win/issues/1737

@MS If you want windows container adoption releasing major breaking changes is not going to get you there.

[daschott]

We've reproduced and investigated these issues now, thank you for your patience and sorry for the delay.

There was a bug in Windows Server 1607 and 1709, where WinNAT did not reserve the correct ports with the TCPIP stack. The reason was, WinNAT and TCPIP interpreted the port number in a different byte order than intended.

This was causing ports which were already reserved in TCPIP to be reused when publishing ports externally for a container. After installing KB4074588, you may see a perceived regression, as the ports which were working earlier may now be blocked, if they are already reserved.

We are actively working to publish a KB to get the word out about this. In the meantime, can you check if the port you like to publish is already reserved on the container host using the following command(s)? netsh interface ipv4 show excludedportrange protocol=tcp netsh interface ipv4 show excludedportrange protocol=udp

[artiomchi]

Hi @daschott

I just tested this, and it seems like the port is in fact reserved on the host. See below:

PS C:\Users\artio> netsh interface ipv4 show excludedportrange protocol=tcp

Protocol tcp Port Exclusion Ranges

Start Port    End Port
----------    --------
        80          80
       443         443
      5357        5357
     37408       37408
     58699       58708
     58709       58718
     61157       61166
     61167       61176

* - Administered port exclusions.

PS C:\Users\artio> docker run --rm -d -p 8080:80 nginx
4a0e695f8ab568d2181a8b0c8a838747e9862df48d3036a63cc75527127f4469
PS C:\Users\artio> netsh interface ipv4 show excludedportrange protocol=tcp

Protocol tcp Port Exclusion Ranges

Start Port    End Port
----------    --------
        80          80
       443         443
      5357        5357
      8080        8080
     37408       37408
     58699       58708
     58709       58718
     61157       61166
     61167       61176

* - Administered port exclusions.

[mlmeyers]

@daschott for the sake of clarity. Are you expecting the port to show in the excluded range before start-up of the container and that is at the root of the issue? If so that is not what I see, the port does not show up as excluded until after start of the container.

[artiomchi]

@daschott I've just tested it on another machine that had the update uninstalled. On this machine, the ports are correctly exposed, and I can access the website on localhost:8080. Both machines are running Win 10 1709 and Docker 18.03

I'm adding the output from this machine below for reference

It's not that exciting, but you can see the difference

``` PS C:\Users\artio> netsh interface ipv4 show excludedportrange protocol=tcp Protocol tcp Port Exclusion Ranges Start Port End Port ---------- -------- 80 80 443 443 5357 5357 37408 37408 49675 49684 49696 49705 49752 49761 49762 49771 50846 50855 50856 50865 * - Administered port exclusions. PS C:\Users\artio> docker run --rm -d -p 8080:80 nginx 66ba6e1cf3c6b5076ac638206a2085b6b70c4413eb572a0d12a9564dd2de914f PS C:\Users\artio> netsh interface ipv4 show excludedportrange protocol=tcp Protocol tcp Port Exclusion Ranges Start Port End Port ---------- -------- 80 80 443 443 5357 5357 36895 36895 37408 37408 49675 49684 49696 49705 49752 49761 49762 49771 50846 50855 50856 50865 * - Administered port exclusions. ```

[daschott]

@artiomchi Thank you for demonstrating that without the KB, the incorrect port of 36895 (instead of 8080) is assigned.

@mlmeyers To answer your immediate question, yes, if you are running into a port mapping issue, we expect the port to be already allocated in some way on the TCP/IP stack, and appear in the list before starting the container. To be explicit, are you saying this is not working (e.g. using docker run -p 8080:80) Invoke-WebRequest http://<ExternalIpOfContainerHost>:8080

Please show similar output as artiomchi has shown above. Steps:

1. netsh interface ipv4 show excludedportrange protocol=tcp
2. docker run command used to map external host IP
3. netsh interface ipv4 show excludedportrange protocol=tcp

[mlmeyers]

@daschott

It appears I am seeing different behavior but still believe it is somehow related to the same. The port is not reserved prior to container start. I am able to access ports via the container IP. I am not able to access ports via localhost. In my example I am working with a Mongo container but holds true for all that I have tested. This issue was definitely introduced with KB4074588.

See for more details: https://github.com/docker/for-win/issues/1737

netsh interface ipv4 show excludedportrange protocol=tcp

Protocol tcp Port Exclusion Ranges

Start Port End Port

---

 50377       50386
 62331       62340
 63128       63137
 63138       63147
 63148       63157
 63256       63265
 65203       65212
 65223       65232
 65233       65242
 65243       65252

• • Administered port exclusions.

docker run --rm -d -p 27017:27107 ngi-mongodb-nano d9a71ea665141de042932ae9461d927344af6dafc45e519a224821d32233e3b5

netsh interface ipv4 show excludedportrange protocol=tcp

Protocol tcp Port Exclusion Ranges

Start Port End Port

---

 27017       27017
 50377       50386
 62331       62340
 63128       63137
 63138       63147
 63148       63157
 63256       63265
 65203       65212
 65223       65232
 65233       65242
 65243       65252

• • Administered port exclusions.

PS C:> netstat -an | findstr 27017 PS C:>

[artiomchi]

@mlmeyers You seem to be having exactly the same behaviour as I'm having in my first report - i.e. the machine with the KB patch installed, where the docker port mappings aren't working :)

[artiomchi]

@daschott Damn, I didn't even notice! You're absolutely right. Now I realise what you meant by interpreting the port number in a different byte order!

[mlmeyers]

@daschott and @artiomchi just to be clear so that I am not adding confusion my testing is with KB4074588 applied.

That said I am confused @daschott The issue is not a "perceived regression" but appears a regression in my case. Based on your previous comments it seems you are relating the issue to binding to ports already reserved. In my instance after applying the KB the issue is encountered but I am not binding to already reserved ports.

Can you confirm ?

[mlmeyers]

@artiomchi Sorry to bother but if time allows can you confirm that on a machine with the KB applied forwarding does not work even when binding to a port not already reserved. Hopefully to prove out that I am not a special case. I suspect we are both encountering the issue related to my other post.

https://github.com/docker/for-win/issues/1737

I am suspecting that @daschott is speaking specifically to the access denied error not to the fact that localhost binding is just broken after KB4074588.

[artiomchi]

@mlmeyers Please check my two console dumps above - you can see the allocated ports before and after starting the container. In both cases the port that I want to map on the host was not allocated before running the container. Which is exactly the case you're having too

I think we're either misunderstanding the issue in the KB, or @daschott's post was a little misleading.

The way I understand it, when the docker engine is trying to expose the port, it requests it from the OS. Pre-patch, when the OS received the request it would allocate the wrong port - in my case port 8080 is requested, which is 90 1F in hex. WinNAT would reserve the wrong port though - port 36895, which is 1F 90 in hex - so you can see that it read the bytes in the reverse order.

At this point, my guess is that since after that call, the desired port was not allocated, docker has some workaround code to force allocate said port, and was able to handle the port forwarding.

When the issue was found, the KB was released which patches WinNAT's behaviour. Except in this case the port allocation code in docker would end up crashing, since it was a workaround around a bug in the first place, and it would try allocate a port that was just allocated for it shortly before.

Of course this is just my guess about how it works and why it's crashing..

Nonetheless, based on your output it seems like we all are experiencing the same issue, you included.

P.S. It's possibly not relevant, since it's not using LCOW, but I switched to Linux Containers just to get the output when I run these commands there. Seems like the port is not listed at all in the netsh command

Finally, I also ran the netstat command to see the output. Here's what I got

# Running Linux Containers
PS C:\>netstat -an | findstr 8080
  TCP    0.0.0.0:8080           0.0.0.0:0              LISTENING
  TCP    [::1]:8080             [::]:0                 LISTENING

# Running Windows Containers WITH KB
PS C:\>netstat -an | findstr 8080

# Running Windows Containers WITHOUT the KB installed
PS C:\>netstat -an | findstr 8080
  TCP    127.0.0.1:8080         0.0.0.0:0              LISTENING

... huh!

[mlmeyers]

@artiomchi thanks for all the details. I suspect you are spot on. As far as I can tell localhost binding is completely broken post this patch. MS may have resolved the defect with reserving ports but that seems to have broken docker localhost binding completely.

@daschott has any one at MS confirmed the localhost binding works post the KB install when investigating this issue? Can't imagine you would want to post a KB on this issue if users are going to find binding still does not work.

@artiomchi from what I gather the localhost forwarding for docker on windows is handled by another process | project. If you look you will find a process running com.docker.localhost-forwarder.exe. When this process gets the request to bind the port it gets an exception for some unknown reason post the KB install. Unfortunately the full exception is not thrown so not much more information to go on. The below is returned in my docker logs as an example 'I suspect you will find the same'.

[21:42:21.217][LocalhostForwarder][Info ] TransortErrorException 1 [21:42:21.217][LocalhostForwarder][Info ] Failed to forward tcp port 27019 to 172.29.42.128:27017 [21:42:21.217][LocalhostForwarder][Info ] Reconciliation failed: invalid argument

[mlmeyers]

@daschott or @pgayvallet It seems very likely that anyone one running Docker for Windows is unable to bind localhost ports post this update. A response of some kind would be very much appreciated.

[Kesmy]

@mlmeyers Just to confirm, from the conversation on #1756 , that yes, I was attempting to bind to several different unused and unbound ports with the patch installed, and was unable to bind to any of them.

Once the patch was removed, binding worked properly to all of the previously attempted ports.

[daschott]

@mlmeyers @artiomchi @Kesmy Thank you for your patience and detailed reports on the issues you're having with this KB. I believe there was a misunderstanding since Windows has no localhost mapping support on RS1-RS3, so apologies for that! However, Docker CE has some logic to make this work, which we suspect that this KB broke. @artiomchi you observations align with this theory :)

@carlfischer1 Can you please get someone from Docker to investigate compatibility issues between KB4074588 and Docker's workaround code for pre-RS4 localhost mapping support?

[mlmeyers]

@daschott Thanks for the response and clarification. Seems we are all on the same page now :)

[mlmeyers]

@carlfischer1 possible to get an update on this ?

[carlfischer1]

@mlmeyers joint investigation continues. the only workaround available is to uninstall KB4074588, which is understood to not be plausible in all situations.

[mlmeyers]

@carlfischer1 thanks for letting me know it is actively being looked at.

[simonferquel]

@daschott I originally wrote the localhost-forwarder hack for pre-RS4. What it does is really simple:

• It listen to docker events for container creation/removal
• From there it proxify TCP/UDP traffic from 127.0.0.1: to :.

So when you run docker run -p 8080:8080 some-aspnet-image, it sees the container creation, and binds on 127.0.0.1:8080 and pipe everything to :8080. (UDP is a bit more complex, but works in a similar fashion).

The problem with the KB fixing port reservation is that we can't bind anymore, and as localhost forwarding is not there pre-RS4, user experience is degraded.

IMO, I don't think the problem is in the KB, it is more that HNS makes a reservation spanning all IPs including 127.0.0.1 for any port published by a container, even in pre-RS4, where localhost forwarding is not there.

[simonferquel]

@daschott I tried to hack things a bit with LookupPersistentTcpPortReservation. I am able to get a reservation token for the container public port, but whenever I call WSAIoctl with command SIO_ASSOCIATE_PORT_RESERVATION with this token on my socket, I get a WSAEACCESS error (even if the process is running as admin). Do you have any insight why it might fail like this ?

[simonferquel]

And BTW, DeletePersistentTcpPortReservation works (and then we can bind the port). I am now creating a PR with that workaround in place (however, it requires the localhost-forwarder to run as Admin instead of as a user). I have no idea if it could break some obscure HNS behavior though...

[daschott]

@simonferquel Sounds good, thanks for the update -- I think I see what you're doing. Perhaps there is a way to avoid the port reservation for 127.0.0.1 in first place so that you don't have to delete anything. I'll make the HNS folks aware of your efforts.

[gtardif]

The latest Docker for Windows Edge release (18.03.0-rc3) contains a fix for this, released just a few minutes ago

[mlmeyers]

@gtardif and @daschott Thanks for picking this up, will kick the tires and let you know how it shakes out.

[artiomchi]

I can confirm that the new release has fixed the issue, and the ports are now correctly forwarded!

[mlmeyers]

Yep all looks good from my side, ports are binding to localhost again post update, Thanks!

[ghost]

Does it require some special settings in docker to make it work?

I am trying to bind the ports for interactive shell in docker for win (or docker in general) (WITHOUT DEAMON (-d switch))

docker run -p 3000:5001 -it microsoft/dotnet:2.1-sdk-alpine

# in container
# optional: note down output of ifconfig
dotnet new mvc -n wapp && dotnet run -p wapp

Application gets created, built and started

Hosting environment: Development Content root path: /wapp Now listening on: https://localhost:5001 Now listening on: http://localhost:5000 Application started. Press Ctrl+C to shut down.

When I try http://localhost:3000 or http://localhost:5001 or http://<ip-of-container>:5001 in host browser, MS Edge on Windows 10, I get not found (INET_E_RESOURCE_NOT_FOUND).

Testing with docker version 18.03.0-ce-rc3-win56 (16433)

[artiomchi]

That's actually not an issue - the behaviour will be the same no matter what version or configuration of docker you're running.

In your case, the app is explicitly ONLY listening on http://localhost:5000 and https://localhost:5001 instead of http://*:5000 and https://*:5001

This is why you can't access it even when using the container IP.

P.S. Also, you're trying to load http://localhost:3000 while you should be trying https://localhost:3000, since you mapped port 5001 to 3000

[artiomchi]

P.S. You can test this by changing your commands to the following:

docker run -p 3000:5001 -it -e "ASPNETCORE_URLS=http://*:5000;https://*:5001" microsoft/dotnet:2.1-sdk-alpine
dotnet new mvc -n wapp && rm wapp/Properties/launchSettings.json && dotnet run -p wapp

Then try access the app at https://localhost:3000

I've opened an issue regarding the asp.net mvc template in 2.1 here: aspnet/templating#360

[ghost]

@artiomchi, thanks! you are absolutely right. I changed /wapp/Properties/launchSettings.json like:

- "ASPNETCORE_URLS": "https://localhost:5001;http://localhost:5000"
+ "ASPNETCORE_URLS": "https://*:5001;http://localhost:5000"

and with https://locahost:3000 in browser it worked. I asked this question few days ago: https://github.com/docker/for-win/issues/1787, and was suspecting even then that it's something I am doing wrong.. 😄

Requesting ASP team to provide command line option so we don't need to fiddle with JSON files.. https://github.com/aspnet/Hosting/issues/1326

[ghost]

Wow, nice workaround for un-overridable-by-default launchSettings environment variables. 👍

[ghost]

Btw, we can install dotnet on raw / non-Microsoft-dotnet alpine box as well using the SDK tarball:

docker run -it alpine

# in container
apk update
apk add curl libstdc++ libunwind libintl icu-libs openssl

curl -O https://dotnetcli.blob.core.windows.net/dotnet/Sdk/master/dotnet-sdk-latest-alpine.3.6-x64.tar.gz
mkdir -p /usr/bin/dotnet
tar -xzvf dotnet-sdk-latest-alpine.3.6-x64.tar.gz  -C /usr/bin/dotnet
export PATH=$PATH:/usr/bin/dotnet
rm dotnet-sdk-latest-alpine.3.6-x64.tar.gz

nugetConfDir="/root/.nuget/NuGet"
mkdir -p $nugetConfDir
cat > $nugetConfDir/NuGet.Config <<-"EOF"
<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
    <add key="dotnet-core" value="https://dotnet.myget.org/F/dotnet-core/api/v3/index.json" />
  </packageSources>
</configuration>
EOF

then to the mvc app

dotnet new mvc -n wapp && dotnet run -p wapp

However there is one obsolete package being pulled by MVC in preivew2 image (that maybe fixed by https://github.com/aspnet/templating/pull/349#issuecomment-373362743), but preview1 tarball is the one that Microsoft/dotnet docker is using which we can fetch as well (by tweaking the blob url above). Console apps are running with preview 2

[Kesmy]

@rickbutton Related to your ask, though it is unrelated to this bug: Yes, our team has been dealing with a variety of issues, usually related to the lack of feature parity between Linux and Windows containers but also myriad networking problems, and even a complete lack of parity between Windows 10 (development) and Windows Server (runtime), for the past year+.

We've also recently discovered that to continue receiving updates to container support on Windows Server, we'll need to pave our servers, as it is impossible to update a Windows Server to 1709.

[sarvasana]

Something from Docker for Windows Edge release (18.03.0-rc3) did not make it into 18.03.1-ce-win65 (17513).

The problem is there in 18.03.1-ce-win65 (17513).

[frankyfish]

Same here. As @youfoundkris still facing the issue. Trying to forward Postgres port during run results in error. docker run -d --name postgres -p5432:5432 -e POSTGRES_PASSWORD=postgres postgres:9 docker: Error response from daemon: driver failed programming external connectivity on endpoint postgres (a3d5b2526be3b6219122e64346b75044cd46f6c7b3d272932cd7aa0a6f9c87fe): Error starting userland proxy: mkdir /port/tcp:0.0.0.0:5432:tcp:172.17.0.2:5432: input/output error. Docker version: 18.03.1-ce-win65 (17513) Win10: version 1803, build 17134.112 Attempt to set port via Kitematic results in inability of container to run normally after restart (no logs can be seen). UPD: Tried the same command works like a charm on laptop with older OS build: Docker version: 18.03.1-ce-win65 (17513) Win10: version 1709, build 16299.492

[docker-robott]

Closed issues are locked after 30 days of inactivity. This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle locked