Support for sharing unix sockets

[BouncyLlama]

Expected behavior

When mounting a directory containing unix sockets the sockets should function the same as they do on a Linux host.

Actual behavior

The socket is 'there', but non-functional.

Information

After reading several forum threads, it appears that there is a workaround with socat over TCP, but this is rather slow.

The documentation has this to say: 'Socket files and named pipes only transmit between containers and between OS X processes -- no transmission across the hypervisor is supported, yet' Hopefully this is a planned feature already, but I did not see any existing issues open in this tracker for this particular issue, although it relates to #410 which asks specifically for SSH_AUTH_SOCK to be supported.

Host OS: Mac OSX 10.10.5

Steps to reproduce the behavior

1. mount a directory containing unix sockets like so: '-v "/directorywithsockets:/otherdirectory"'
2. attempt to send data to/from the host/container via the socket

[aleohl]

Any updates on this issue? Ran into it today and it took a while to find this issue...

[monoflash]

They just closed the problem without solving it. No issue no problem :)

... unix socket still does not work on macOS! Issue opened two years and five months ago!

[jpcope]

If socket mounting is not supported then a PR that calls this out when the user attempts to do so can save us all time and effort. A PR that calls out this exact issue in github would also force more attention.

Anyone have time to submit a PR? Clearly there must be a larger issue under the hood that keeps this issue from being resolved. Surely a warning or nag is better than blind optimism and wasted time?

[fbsb]

Too bad osxfs and docker-for-mac are not open-source 😞

[BretFisher]

D4Mac is built from many Moby tools:

https://github.com/moby/datakit https://github.com/linuxkit/linuxkit https://github.com/moby/hyperkit

[BretFisher]

Mentioned here: https://github.com/docker/for-mac#component-projects

[dhull]

Regarding @domdom82 's netcat solution, I found the for the container side to reliably work I had to add the -k option to the second netcat invocation on the container side:

nc docker.for.mac.localhost 12345 <myfifo | nc -Ulk /tmp/ssh-agent.sock >myfifo

[Alan-R]

@dsheets

Here's a use case for UNIX domain sockets and containers that leverages UNIX domain sockets and containers together in a unique and IMHO way cool, security-improving way.

UNIX domain sockets allow the server side to authenticate the client. No other type of socket or IPC allows this. This is an amazing way to pass secrets (credentials) into containers - except the Mac under Docker :-(.

See https://www.peerlyst.com/posts/sharing-secrets-with-containers-using-custodia-alan-robertson and https://www.peerlyst.com/posts/the-authproxy-method-of-sharing-secrets-safely-with-containers-alan-robertson for reasons why this is an important security feature.

FWIW: Custodia is a key part of Red Hat's identity management solution. The authproxy method is much simpler and easily implemented.

[Alan-R]

@jpcope It's not the mount that can detect the problem - as the most reliable way to do that is to mount the directory which contains the socket. There's nothing special about the directory. If you mount the directory containing the socket, then the application in the host can crash and restart (creating a new socket inode) and not have to restart all the containers which might want to use it.

[alexlanz]

Any updates on this? Can't wait to see this feature. It is the only reason that stops my team for using Docker for development and maybe later also in production!

[goodevilgenius]

Here's my use case:

I'm using Google Cloud SQL in a project. They provide a proxy that creates a socket that allows communication with the server.

I run the proxy on my host, and can connect through the socket. But when I try to connect through the socket within a container, that fails (of course).

My only solution right now is to create the socket within the container itself, but because this project is using multiple containers, that means I have to create it multiple times, which is not ideal.

[matthew-hollick]

Adding my voice to the others. I am evaluating nginx/unit for future use. The official unit docker image presents a unix socket for control. Given that I do not work with sockets on a daily basis it took me ages to figure out that it was docker not working and not my invocation of curl/socat/nc.

Maybe if this is not going to work on MacOS it could be made more obvious that it is not working? Not sure how, maybe documentation (I searched but could not find reference to this bug other than this github issue.) maybe just prevent directories containing sockets from being exposed on MacOS?

[grimmy]

I agree about making it more obvious. A simple os.Stat on the source of the volume mount could check if it's a socket and then direct the user to here. Would probably help avoid duplicate tickets as well.

[Alan-R]

@grimmy That would catch a small percentage of the cases. The more robust way to share a socket is to share the directory it's in -- or will be in - as I noted in a comment above. Personally, I'd rather see them put the effort into fixing the darn thing.

[grimmy]

@Alan-R hmm I actually haven't tried mounting the parent directory. If that works though that would fix my issue trying to get the auth socket for ssh-agent into a container.

[Alan-R]

That doesn't fix that - it fixes other issues - mainly now you can restart the server. Of course this doesn't matter if you can't use it - which you can't on the Mac.

On Tue, Apr 24, 2018, at 4:53 PM, Gary Kramlich wrote:

@Alan-R[1] hmm I actually haven't tried mounting the parent directory. If that works though that would fix my issue trying to get the auth socket for ssh-agent into a container.> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub[2], or mute the thread[3].>

-- Alan Robertson alanr@unix.sh

Links:

1. https://github.com/Alan-R
2. https://github.com/docker/for-mac/issues/483#issuecomment-384104725
3. https://github.com/notifications/unsubscribe-auth/ABM0o2J2nsRPMQMTLgspTBaVXtR6fdVTks5tr6zegaJpZM4Jx1jX

[glensc]

mounting socket or dir won't work for mac, because unix socket is kind of in os memory, but docker and mac is not the same kernel. you'll get the socket on the other side, but it's not "connected" because logically speaking those sockets are in two different machines.

[Alan-R]

Of course it won't work. That's the whole point of this issue ;-).

The point is that it works perfectly on Linux - and should also work on MacOS. Docker is not a VM. If that's how MacOS implemented it (as a VM), then that's broken - and it's not Docker at all. I would imagine that dozens of things would be broken if they implemented a VM for each Docker instance. I suppose they could implement a single VM for all the Docker instances.

[goodevilgenius]

@Alan-R Docker for Mac is implemented with a single VM for all Docker instances. That's necessary because Docker is meant to run on a Linux kernel, but macOS uses a Mach kernel.

[udondan]

Correct, that's how it is. One single VM. See https://docs.docker.com/docker-for-mac/docker-toolbox/#the-docker-for-mac-environment

Still a VM and therefore not the same Kernel as the Host.

[Alan-R]

So, it sounds like this isn't possible, and will remain in it's current state (broken/unfixed) forever?

It's not the data transmission between the OSes that I care about, it's the socket calls to authenticate who the user is, and then to look in /proc to get the cgroup and then do a docker inspect. It sounds like all that stuff will never work - given that architecture. Docker processes show up on the host. I haven't looked, but I assume they don't on MacOS.

I'm curious why @dsheets stated it's on the roadmap if it's actually impossible to do.

[udondan]

I'm pretty sure it is possible. You can still forward sockets per netcat or socat. Somebody just has to look into it.

[fbsb]

Theoretically it's possible to share sockets between the mac host and the docker vm (hyperkit) as it works with the docker socket itself.

The difference between the docker socket and user mounted sockets is that the docker socket gets mounted via the vpnkit by hyperkit while volumes are ultimately shared to the vm by the osxfs (which has not been opensourced yet).

For this to work, osxfs has to detect whether a volume is a socket and connect it via a network socket inside the machine (just like socat works).

As a workaround one could run socat inside a container connecting a unix socket to a network port and then on the host another socat would connect the host unix socket to the container port. While this is very ugly and error prone it might work.

[udondan]

For reference:

https://docs.docker.com/docker-for-mac/osxfs/#file-types

Symlinks, hardlinks, socket files, named pipes, regular files, and directories are supported. Socket files and named pipes only transmit between containers and between macOS processes -- no transmission across the hypervisor is supported, yet. Character and block device files are not supported.

[Alan-R]

@udondan That would be insecure and would not support the system calls to identify the caller on the other end. It would eliminate the main advantage of UNIX domain sockets over named pipes.

On Linux, you can get the pid, uid, gid, and security context of your caller. On other BSD systems, you can get most of those things. On this implementation, those things aren't meaningful. So, you can no longer identify who is sending you messages.

On a Linux system, you can also get the cgroup from /proc and use it to figure out which container the caller was coming from, and then do a docker inspect on that container. You can't do that here.

So, full compatibility would be impossible without reimplementing Docker directly on the host OS. This is possible, but a huge amount of work. Containers don't care about the host OS - except for the system calls - which is pretty limited compared to the whole system. Docker, on the other hand, needs things like cgroups and much more OS capability.

The workaround I suppose is to create your sockets in other containers, and run whatever application you had in mind in a container - maybe one with full OS privileges and the Linux host /proc mounted directly on the container /proc (or something like that). An ugly hack that still might not work on MacOS.

[udondan]

Oh yes, sure, it does. Most popular solution probably is https://github.com/uber-common/docker-ssh-agent-forward

[Alan-R]

@udondan I agree you could get data forwarding working without a huge trouble (which I said earlier). But getting these other OS calls working and giving meaningful results would be impossible - because the process space and UID and GID space is separate between the host and the docker instances.

[udondan]

Unfortunately true. But it's the best we have at the moment. 😿

[Alan-R]

I would go further and say it's the best that there ever will be without a completely new implementation of Docker that doesn't involve a VM.

The best workaround is to just start a Linux VM on your Mac and do everything in your VM and forget MacOS. Alternatively, move all your host work into a Docker container, and figure out how to get access to all the Docker VM resources you need - which is kind of the half-way version of the first alternative.

[tvon]

Two things I think would be useful in this discussion:

1. What the community should expect as far as an upstream solution. Roadmaps change, priories change, that is the nature of the business. Is this still on a roadmap, or is it expected for an upcoming release, or has it been de-prioritized and unlikely to show up anytime soon in a release?
2. Any technical information on the core issue. Is this a HyperKit limitation? A Hypervisor Framework limitation? Is a technical solution even possible on the OSS side of Docker for Mac? Is there a project that this is built upon that has discussion relevant to this issue?

[tomgeorge]

So... I found something interesting in this for my use case (sharing docker.sock for local development, not really concerned about the security of the socket), but I have no idea what the implications are of this.

On the OS X host, /var/run/docker.sock is owned by root:daemon, and when you run a container with -v /var/run/docker.sock:/var/run/docker.sock, docker.sock is owned by root:root.

I was looking into trying something with socat to forward this into the container (which I don't fully understand, either), and just for grins I did the following

docker run -v /var/run/docker.sock:/var/run/docker.sock -it jpetazzo/dind /bin/bash
# docker.sock is owned by root:root at this point
chown root:daemon /var/run/docker.sock
usermod -Ums /bin/bash -G daemon tom
su tom
docker run hello-world

That seemed to work. Running docker in OS X still worked after exiting the container, and on subsequent docker runs the permission change persisted until I restarted docker for mac.

I don't really know why this works, or if I'm going to hose some part of my system, but it worked for me, I can now run docker on the host and in a container.

Docker for mac Version 18.03.1-ce-mac65 (24312) OS X 10.13.4

[4ntoine]

So are Unix sockets supported by docker eventually (host: macOs, container: linux)?

[Alan-R]

You can do some things (described above) which allow the two to inter-operate if all you need is data connectivity. But, if you need full Linux sockets APIs (like I described earlier) - that is unlikely to ever happen.

[trinitronx]

It seems that the most prevalent use case for this feature is for developers to develop and test containers locally on a single-user MacOS laptop. In this case, the connectivity is the main feature needed while the extra security & user identification provided natively on Linux are probably less useful. Most common examples being:

• Mounting Docker socket for launching other containers & interacting with Docker daemon from a container
• Mounting SSH Agent Socket for SSH connections from a container

Correct me if I'm wrong, but I've never heard of any company using a production environment running containers hosted on macOS Server. Therefore security features are lower priority given that the developer will usually trust themselves, or at least take responsibility if they do things that accidentally mess up the host system somehow. In this case, the "customer" (e.g. developers) for the feature really just wants working sockets through host volume mounts to work just like on Linux. It could be that some use cases for using sockets that require these extra features will not work, but at least basic connectivity is satisfied.

[Alan-R]

I agree with @trinitronx about why people use this and why the security features don't matter to most people - unless like me, they're testing and developing security sensitive software which needs this feature.

Given that the Docker OS environment and the MacOS environment share nothing but the contents of regular files, and have no pid space in common, and even have different maximum values for process ids, sharing that kind of security information would require a very radical and costly rearchitecture of the MacOS docker implementation, or a switch of MacOS to become Linux-based. IMHO, neither seems very likely - exactly for the reasons @trinitronx stated. Although the former is somewhat more likely than the latter ;-).

[lox]

One of the primary reasons we have Docker for Mac is to test and develop Docker workflows that will run on Linux environments. Without parity around how sockets are handled, it's very difficult to work with any applications that mount in an ssh-agent, or the myriad of other use cases folks above have described. The alternative is developing custom alternate implementations for when the host is macOS, which kind of devalues some of the core benefits of docker and docker-compose based workflows.

[zachmullen]

mounting socket or dir won't work for mac, because unix socket is kind of in os memory, but docker and mac is not the same kernel. you'll get the socket on the other side, but it's not "connected" because logically speaking those sockets are in two different machines.

How is that fundamentally different from the stdin, stdout, and stderr pipes? I'd like to be able to use named pipes, which should in theory work almost the exact same as the process standard pipes in a hypervisor environment. Can someone explain to me why docker on MacOS supports stdout/stderr/stdin, but not named pipes? Is there some critical difference I'm missing?

[kler]

Docker for Mac is running inside a virtual machine, with only network connectivity to the outside.

On Wed, 1 Aug 2018, 00:53 Zach Mullen, notifications@github.com wrote:

mounting socket or dir won't work for mac, because unix socket is kind of in os memory, but docker and mac is not the same kernel. you'll get the socket on the other side, but it's not "connected" because logically speaking those sockets are in two different machines.

How is that fundamentally different from the stdin, stdout, and stderr pipes? I'd like to be able to use named pipes, which should in theory work almost the exact same as the process standard pipes in a hypervisor environment. Can someone explain to me why docker on MacOS supports stdout/stderr/stdin, but not named pipes? Is there some critical difference I'm missing?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/docker/for-mac/issues/483#issuecomment-409393840, or mute the thread https://github.com/notifications/unsubscribe-auth/AA699EXJ7XuQ-gTRv4Sj1SG4Vlx28Gq6ks5uMOADgaJpZM4Jx1jX .

[docker-robott]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale comment. Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle stale

[href]

/remove-lifecycle stale

[ghost]

http://collabnix.com/how-docker-for-mac-works-under-the-hood/

Docker for Mac does not use docker-machine to provision its VM. The Docker Engine API is exposed on a socket available to the Mac host at /var/run/docker.sock. This is the default location Docker and Docker Compose clients use to connect to the Docker daemon, so you to use docker and docker-compose CLI commands on your Mac.

So.. if it can expose that socket, how come it can't handle SSH_AUTH_SOCK ?

[djpadz]

@bryanhuntesl said:

So.. if it can expose that socket, how come it can't handle SSH_AUTH_SOCK ?

Because the docker socket is being exposed by a daemon running under macOS. The challenge with SSH_AUTH_SOCK is that it needs to be exposed inside of a running container, which is actually running inside of a VM. So, the socket would be caught by the VM's kernel, instead of the macOS kernel. It's like trying to mount a socket from one machine to another via NFS. The socket file exists, but its metadata doesn't make any sense to the kernel on the remote machine.

[BlinkyStitt]

It looks like docker 2 added “docker build —ssh AUTHSOCK” which covers some of the cases needed here.

On Nov 21, 2018, at 12:55 PM, Dj Padzensky notifications@github.com wrote:

@bryanhuntesl said:

So.. if it can expose that socket, how come it can't handle SSH_AUTH_SOCK ?

Because the docker socket is being exposed by a daemon running under macOS. The challenge with SSH_AUTH_SOCK is that it needs to be exposed inside of a running container, which is actually running inside of a VM. So, the socket would be caught by the VM's kernel, instead of the macOS kernel. Think of like trying to mount a socket from one machine to another via NFS.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[tamsky]

@WyseNynja writes

It looks like docker 2 added “docker build —ssh AUTHSOCK”

I couldn't find this in any release notes. Can you share a link to this new feature?

[tamsky]

I think this is it... I had been looking at docker/for-mac release notes.

https://github.com/docker/cli/pull/1014

[BlinkyStitt]

docker/cli#1014 is something else (though it may also be useful for the people in this thread).

I was talking about https://medium.com/@tonistiigi/build-secrets-and-ssh-forwarding-in-docker-18-09-ae8161d066

[ghost]

@bryanhuntesl said:

So.. if it can expose that socket, how come it can't handle SSH_AUTH_SOCK ?

Because the docker socket is being exposed by a daemon running under macOS. The challenge with SSH_AUTH_SOCK is that it needs to be exposed inside of a running container, which is actually running inside of a VM. So, the socket would be caught by the VM's kernel, instead of the macOS kernel. It's like trying to mount a socket from one machine to another via NFS. The socket file exists, but its metadata doesn't make any sense to the kernel on the remote machine.

Thanks for clarifying.

[jeantil]

docker/cli#1014 is something else (though it may also be useful for the people in this thread).

Is it possible to connect to the docker4mac host vm using this ? (export DOCKER_HOST=ssh://localhost:????), then it should be possible to use ssh agent forwarding to share the osx based agent with a container ?

I am very very very interested in a solution for this. I have been hardening my security practices for the last couple years and this is one of the last issue with my setup.

I cannot copy my ssh private key nor share it through a volume (I use gpg-agent with ssh-agent mode and the corresponding gpg private key is safely stored on a write-only yubikey) therefore most workarounds to share my ssh credentials with docker containers are unapplicable for me.

Since it is also impossible to map a USB device to a container (as far as my research indicates) I was forced to create a less secure ssh key to be used in containers which is not so nice and means I have abandoned docker for some use cases (managing my ansible environment for instance).

[adamnovak]

Is this the right issue for tracking cross-hypervisor FIFO support? Or is this issue only for Unix sockets and their special features, and plain text streams need their own issue?

It seems like plain streams have a relatively successful workaround of "just use a network connection instead". Is that going to be (or has it been) rolled in as a workaround in Docker for Mac itself, or should I spend time implementing it in my project?

[bevensteven]

Does anybody know if Hashicorp Nomad's found a way around this? I'm running a nomad dev agent locally on a mac that runs a Docker job but can't get the logs corresponding to this job through Nomad.

The error message I get is: Error reading file: Unexpected response code: 500 (log entry for task "<taskname>" and log type "stdout" not found)