Open hairyhenderson opened 9 years ago
Some clarification - this actually doesn't work with boot2docker-cli either. I had built the ISO locally at first, so it never had to actually download it. So I guess this is a more general issue. Anyone had any experience with this?
I'm not quite sure how this would work. I guess it all depends on how "net/http" reads CA certs from the system, I'd assume it'll use the keychain on OS X and the CA store on Linux...
Yeah, I did some dtruss
ing and found that it's opening the various keychains in the right order (local, followed by System, followed by System Roots). The certs are in the System keychain. Other (non-golang) commands work fine, such as curl
. I dtruss
ed curl and it looked like it was opening the same keychains... Obviously I have no idea what net/http
is doing with those keychains, but... ;)
We are experiencing this problem as well. It seems to me this is caused by the cross-compilation on linux. If I go run
a simple go file that creates a http client this same way as getClient it works but not when building the docker-machine binary with script/build
Ok thanks. I'll try the same for me and see if we can get a fix.
On Tuesday, February 10, 2015, Fabian Ruff notifications@github.com wrote:
We are experiencing this problem as well. It seems to me this is caused by the cross-compilation on linux. If I go run a simple go file that creates a http client this same way as getClient https://github.com/docker/machine/blob/master/utils/b2d.go#L23 it works but not when building the docker-machine binary with script/build
— Reply to this email directly or view it on GitHub https://github.com/docker/machine/issues/491#issuecomment-73735377.
Apparently this problem is pretty old news. See https://github.com/boot2docker/boot2docker-cli/pull/13 or https://github.com/docker/docker/issues/3683 for example.
Looking at go's source there are two different ways for reading the root certificates for darwin depending on whether cgo is enabled or not:
Looking at the /System/Library/Keychains/SystemRootCertificates.keychain
on my machine it does only contain the Apple compiled list of root CAs shipped with OS X and is read only.
Commit https://github.com/golang/go/commit/4f234814831c48a3bbc2b9a2d00242fad890facf mentions that there are some differences:
The set of certs fetched via exec'ing
security
is not quite identical to the certs fetched via the cgo call. The cgo fetch includes any trusted root certs that the user may have added; exec does not. The exec fetch includes an Apple-specific root cert; the cgo fetch does not. Other than that, they appear to be the same.
When building natively on OS X this is a non issue but when cross-compiling for darwin it seems to be the CA handling is partly broken. There just seems to be no easy way of making go pick up user supplied trusted root CAs from the Keychain.
Should we vendor a set of root CA certs like Swarm does? These could be updated periodically.
@ehazlett: how would that solve this issue? I need to add CA certs that should never be out in the wild...
@hairyhenderson sorry i was thinking of a different CA issue. perhaps allowing the ability to pass a root CA cert for http or something. Not sure, just trying to throw out some ideas :)
That's probably the only way
From @databus23 's comments, it seems like this has to do with golang's cross-compilation on Linux... Maybe a fix would be to compile the OS X binary on OS X instead? I'll admit to having close-to-zero clue about golang, or how docker-machine gets built, so maybe it's not feasible...
@hairyhenderson Would be worth a test. Would you mind trying this build: https://public.evanhazlett.com/docker-machine/test/docker-machine-f9278b99-darwin-amd64
Built using godep go build
from OS X.
@ehazlett d'oh. Same error as before.
hmm ok
Building natively on OS X definitely makes the problem go away. I tested it before and I just tried the linked binary above. It works for me. I have our corporate CA certificate in the System Keychain and it does not work with cross-compiled binaries but does work whenever I compile natively. @hairyhenderson Can you re-check where you have installed the self-signed certificate?
@ehazlett I don't like the idea that the user has to specifically pass in the certificate. Ideally it should work with all trusted certificates from the keychain as it does when building natively.
I see three ways to solve this issue:
brew install docker-machine
I think 1. is the cleanest and most straight forward but its hard to automate and probably requires manual building for the foreseeable future. 2. adds some requirements for running machine: homebrew has to be installed and when installing machine brew also needs to pull down the go package to build the thing. 3. is the most fiddly but could provide a working binary with the current build workflow via docker.
I added gonative to the docker-machine Dockerfile
briefly and was able to cross compile a darwin binary that worked with custom certificates so 3. really does work btw.
@databus23 - I have my CAs in the System Keychain as well - i.e. /Libraries/Keychains/System.keychain
When I have a bit more time (probably not until Sunday or Monday evening, unfortunately), I can try to compile natively myself and see if that works differently...
This is an odd bug, we should trace it through and report it.
@databus23 completely agree with using system certs.
@sthulb I believe this is a bug that can't be fixed easily. There just seems to be no way of getting a list of all trusted certificates without interfacing with the Security Framework c library provided by Apple. The command line utility /usr/bin/security
provides no way of getting certificates and there trust settings from the System Keychain. I spend half day with it and gave up on it.
I guess we'll just have to make OS X builds on OS X from now on :)
Wow, very sorry to hear about this @hairyhenderson and @databus23. Thank you for being so diligent with assessment of the issue. I definitely want to help you get a solution to this.
I'm very against changing the default build process to native. gox
is fast and easy to use and is set up well now in our build toolchain as it is. It would slow down contribution and releases significantly to not be able to cross-compile so fast and easily (and in a container) - it's not uncommon for me to chuck some of those binaries onto other platforms I want to test machine on where don't have Go tools installed at all. It also might introduce cross-platform differences which could cause bugs. Most of the issues that we would usually need cgo
for e.g. home directory we are already working around in various ways.
So this is what I propose to mitigate the issue (short-term):
script/native_build
script so that users which need to can compile their own natively (this should be relatively fast to put together) if they need it.brew
recipe which will compile docker-machine
from source natively. The official install would still be to curl
a binary into $PATH
, but this would be the "easy way out" for users having this issue. Any volunteers for this? Also, this won't mitigate the problem on Windows, I'm curious if @jeffmendoza has any ideas.How does that sound to everyone? @hairyhenderson @databus23 which sounds best to you?
I think a brew recipe is the best short term solution for OSX from the user perspective. The only hassle is that this also needs to be updated/published for every new release otherwise users who need custom certificates will be stuck on older versions.
@nathanleclaire I'm not sure this problem exists for cross compiled windows binaries.
@nathanleclaire I agree that changing the build progress is not a good idea. Its really slick the way it is. That being said I still think that the solution proposed in the linked blog post would be better solution in the long run. It still allows for cross compiling with gox in a container but without the limitations cross-compiling usually brings regarding the go stdlib.
In general you guys should be aware that shipping cross-compiled darwin binaries that do anything with https might blow up for users which use certificates Apple didn't put in the SystemRoots Keychain.
@nathanleclaire - if there's a way to add a cert to machine's trusted cert list, would it be possible to get machine to add /Libraries/Keychains/System.keychain
by default? Or is that not a good idea (or impossible) for some reason?
@hairyhenderson To my knowledge this is not possible without negative security implications. Using /usr/bin/security -a -p find-certificate /Libraries/Keychains/System.keychain
to dump all certificates from the System.keychain
is a security risk because not all certificates in the keychain are actually trusted. In fact you can have certificates in the system keychain which you explicitly distrusted. As far as I can tell there is no way of sorting out certificates by their trust settings using /usr/bin/security
. The only viable way seems to be the Security Framework
c-binding which is not available when cross-compiling.
@databus23 ah - I see... sounds like we need some other tool that's aware of trust settings.
@hairyhenderson I think the options outlined in my above post are better solutions then trying to reimplemented the certificate trust setting evaluation as done by the Security Framework of OSX. Specifically option 3. allows to still cross-compile using a docker container while having the exact same code path for x509 certificate handling as when compiling natively.
@databus23 I was looking into this recently and I don't believe the security
command provides a way to list certificates anyways. I just tried your command under Yosemite and it complained about -a
being unsupported.
I think the only way to programmatically get a list of certificates is to go to the low-level SecItemCopyMatching
function.
@md5 It sure does. This is how the go stdlib itself fetches the certificates for the SystemRootCertificates.keychain
in a nocgo environment since v1.3.
@databus23 Thanks. Looks like the command is actually /usr/bin/security find-certificate -a -p /Libraries/Keychains/System.keychain
I realize now that I was looking into enumerating passwords, not certificates.
@databus23 I like the idea of using gonative
as it stays in the docker container build flow. The only thing that concerns me is the "Open Issue" of that it hasn't been tested on Windows but I think we could give it a shot.
@ehazlett I might be misunderstanding this, but I think this statement means cross-compiling with gonative directly on windows is not tested. In docker-machine's case the container build flow ensures we always compile on linux. The author of gonative writes in the accompanying blog post:
I’ve tested targeting Windows/Linux/Darwin and they all work correctly!
@databus23 correct. Ah ok -- perhaps I misunderstood -- I read it as "the cross compiled binary hasn't been tested on Windows". I did see the comment from the blog so I think it's worth it to try.
@hairyhenderson, is there any solution now to retrieve a custom (corporate) boot2docker.iso from a server with https (and own company certificate)?
@FraBle Yes, just compile docker-machine natively. brew install docker-machine
will do.
@databus23, thanks for your super fast reply. I tried the Homebrew version:
frank >>> brew install docker-machine
==> Downloading https://homebrew.bintray.com/bottles/docker-machine-0.6.0.el_capitan.bottle.tar.gz
Already downloaded: /Library/Caches/Homebrew/docker-machine-0.6.0.el_capitan.bottle.tar.gz
==> Pouring docker-machine-0.6.0.el_capitan.bottle.tar.gz
==> Caveats
Bash completion has been installed to:
/usr/local/etc/bash_completion.d
==> Summary
🍺 /usr/local/Cellar/docker-machine/0.6.0: 5 files, 36.8M
frank >>> rm -rf /Library/Caches/Homebrew/docker-machine-0.6.0.el_capitan.bottle.tar.gz
frank >>> brew uninstall docker-machine
Uninstalling /usr/local/Cellar/docker-machine/0.6.0... (5 files, 36.8M)
frank >>> brew install docker-machine
==> Downloading https://homebrew.bintray.com/bottles/docker-machine-0.6.0.el_capitan.bottle.tar.gz
######################################################################## 100.0%
==> Pouring docker-machine-0.6.0.el_capitan.bottle.tar.gz
==> Caveats
Bash completion has been installed to:
/usr/local/etc/bash_completion.d
==> Summary
🍺 /usr/local/Cellar/docker-machine/0.6.0: 5 files, 36.8M
frank >>> docker-machine create dev -d virtualbox --virtualbox-boot2docker-url https://<mycompany_github>/api/v3/repos/monsoon-docker/boot2docker-sap-iso/releases/latest --engine-registry-mirror https://mycompany_docker-mirror>
Running pre-create checks...
Creating machine...
Error creating machine: Error in driver during machine creation: Get https://<mycompany_github>/api/v3/repos/monsoon-docker/boot2docker-sap-iso/releases/latest: x509: certificate signed by unknown authority
The certificate for out Github is imported in the System Keychain.
ps: in terms of dependencies:
==> Dependencies
Build: go ✔, automake ✔
@FraBle Thats odd. It should work if you have the SAP Global Root CA in your keychain. Are you sure you are invoking the right docker-machine
binary? What does which docker-machine
say? We can continue this in-house just ping me via Lync. :)
@FraBle What is output of which docker-machine
? Of ls -lah $(which docker-machine)
?
@nathanleclaire , @databus23
frank >>> which docker-machine
/usr/local/bin/docker-machine
frank >>> ls -lah $(which docker-machine)
lrwxr-xr-x 1 <myuser> 49 Feb 29 14:42 /usr/local/bin/docker-machine -> ../Cellar/docker-machine/0.6.0/bin/docker-machine
Yup, looks symlinked to the Homebrew version correctly.
So this definitely should work. I'm using the same homebrew version against the same internal github enterprise installation. Has to be something with your local setup.
@databus23, you have mail in your corporate mailbox :)
@FraBle I'm on CET so i'll look into tomorrow. Cheers
@databus23 :+1:
Wow, this was a blast from the past :wink: -- makes me thankful I'm no longer working for the company I was at when I logged this :grin:
Seriously though, it seems we probably need to update some docs somewhere to cover this use-case...
So I'm getting this error:
I need a custom
boot2docker.iso
because our internal HTTPS all uses an internal corporate Root CA for signing (so, basically self-signed). I had this working fine with boot2docker-cli (see boot2docker/boot2docker#347), but I want to try out docker-machine with the same ISO.I'm on OS X, and I've installed the Root CA certs in such a way that most apps recognize it (browsers, etc...). Is there some special other way to install my certs such that docker-machine will be OK with them?
Thanks!