Closed JoeyAcc closed 7 years ago
Thanks for the report! So it looks like you're on Linux, and I'm going to initially assume that you're using rustup and/or the official toolchains. In that case the SSL library in use is OpenSSL and it's linked statically, so we're not using the system SSL implementation. This means that we have to ourselves instruct OpenSSL where to find certificates, and the crate which does this isn't entirely principled about how it does so.
So my guess as to what's happening here is that the probing for ssl certs is perhaps choosing a different location than where the actual certificates are living. Do you know where the valid SSL certificates are located? Does some combination of SSL_CERT_{FILE,DIR}
work for instructing Cargo where the SSL certificates are? Once we sort that out I figure we can determine what patch needs to be applied to Cargo
Thank you for writing/maintaining an awesome project, Cargo is hands-down one of the best build tools I've ever used (measured by frustration and anger when trying to get things done, or rather lack thereof :) ).
Linux: ✓ (At least the build server, but that's what matters here)
Rustup: ✓
Official Rust toolchains: ✓ (We use nightlies as the sbr-xquery-interpreter-api
project uses some unstable Rust features)
The cert installation procedure is as follows:
company_cert.pem
file to /etc/ssl/certs/
sudo dpkg-reconfigure ca-certificates
, which takes care of actually making the cert known to the system as a whole (apologies if it's redundant, I want the info to be as complete as I can make it).BTW, I did notice that /etc/ssl/certs/
is not in this list, I don't know what the exact consequences of that are?
As for SSL_CERT_{FILE,DIR}
, specifically executing both export SSL_CERT_DIR=/etc/ssl/certs
and export SSL_CERT_FILE=/etc/ssl/certs/company_cert.pem
has an effect on whether or not the build can fetch normally. Withholding either results in fetch issues from github.com
(rather than GitHub Enterprise, which surprised me):
Updating registry `https://github.com/rust-lang/crates.io-index`
error: failed to load source for a dependency on `neon`
Caused by:
Unable to update registry https://github.com/rust-lang/crates.io-index
Caused by:
failed to fetch `https://github.com/rust-lang/crates.io-index`
To learn more, run the command again with --verbose.
To get things building in the meantime I had to write a workaround, which works but is somewhat brittle: It involves invoking a bash script that manually clones the Rust dependencies of sbr-validate-npm
, then edits the corresponding dependencies in Cargo.toml to point to those local clones, and only then starts the build.
If there's any other info I can provide, please let me know and I'll add it.
Interesting! So to confirm, it was possible to clone from GitHub Enterprise when you set the environment variables? I think openssl-probe should be inferring SSL_CERT_DIR=/etc/ssl/certs
as down near the bottom it'll append the certs
dir. The openssl-probe crate won't, though, infer the value you mentioned for SSL_CERT_FILE
.
If you configure these env vars and it makes GH Enterprise works but breaks the actual GH, that's also bad though! I forget the precise mechanism that the DIR and FILE env vars configure, but IIRC you only really need to configure one or the other. The FILE one is "here's a bundle of valid certs" (I think) whereas the DIR one is "here's a giant dir of valid certs". It may be the case that openssl-probe is inferring a value for the FILE variable but that's causing breakage if there's custom certs in the dir too?
We're definitely quite interested in finding a fix here, just gotta figure out the best way!
It was possible to clone manually i.e. using command line git. This is also what my workaround build script does: first it manually clones the dependencies using git clone
, then it updates the relevant Cargo.toml
files to point to those newly-cloned dependencies, and only then it builds with neon build
(which delegates a part to cargo build
).
At this moment, trying to update the dependencies using cargo fails on those certs, no matter what I do.
And yeah it would be more intuitive that I'd only need one of SSL_CERT_*
env vars, but I tried all 4 combinations and setting both SSL_CERT_FILE
and SSL_CERT_DIR
is the only thing that actually does the job so far.
Ah sorry I'm personally no expert on SSL certificate configuration, nor how the directories/filesystems work on distros. At this point I'm not sure what's going on, but it seems likely to be related to ssl configuration?
It's very likely related, but I'm not sure about the exact nature of that relation (e.g. causal or just a correlation). As you may have guessed by now we use certs at work here, and pretty much everything else does what it's supposed to.
It's only when Cargo is introduced to the mix that I start getting SSL errors like the one above. It's this last part that makes me look at Cargo for the root cause.
Is there any other information I could provide to help debug this issue?
Unfortunately without a way to reproduce locally I may not be of much more help in debugging. You could try strace on a working tool to see what files it accesses and then try strace on cargo to see what the difference is maybe?
👋 Why is cargo statically linking against its own openssl? This seems problematic, is this something that can be fixed? Is there something that I can do to help?
@ethomson right now one of the primary mechanisms to ship Cargo is with the precompiled binaries that we ship ourselves as part of the rust-lang/rust project. These binaries are intended to be "extra compatible" in the sense that they're compiled with very old toolchains (to have low glibc requirements) and statically link dependencies (to not have runtime requirements).
It's expected, however, that if you install Cargo through a package manager (e.g. via a linux distro) you'd get a dynamically linked OpenSSL rather than a statically linked one (as you'd expect).
So in that sense I think it's unfortunately just a limitation of shipping "compatible" precompiled binaries. We could in theory ship one with a shim that dynamically uses whatever is available at runtime, but in general it's much easier on our end to just statically link it and then probe for certs at runtime.
So what's the plan for this issue? Are you going to teach cargo where to find the system certificates in a way that's compatible with the default system OpenSSL? Is there even a way to probe that?
Also, for what it's worth, I would have never expected that anybody statically linked OpenSSL. Especially something that I trust to install software on my machine. I understand the dependency problem you're facing but with security libraries, it seems like loading the system one is ideal, so that your end users have a predictable upgrade path. But that's just my $0.02.
We've currently got a crate to probe for standard cert dirs across linux distributions and it currently configures requisite env vars to pass the informationa long to OpenSSL.
To tackle this issue there's some reason that logic isn't working, so it'd basically be to start debugging there. Although without the ability to reproduce I may unfortunately not be super helpful :(
Apologies for the late response, other responsibilities got in the way. Let me try this from another angle, by describing what exactly I want to achieve:
We have a Jenkins build server as well as a GitHub Enterprise install here.
In addition, we have the top-level consumer app (sbr-viewer-ws)
← rust-backed npm module (sbr-validate-npm)
← multiple rust dependencies (regular rust crates e.g. sbr-xquery-interpreter-api)
dependency chain mentioned at the top.
Here's the issue: cargo has issues downloading rust project dependencies on the build server due to somehow not being able to access the cert (which is definitely installed, btw), while on every developer workstation (all of which use the same certificate to access GHE) cargo works without any hitch. What we need is for cargo to work the same on the build server as on the developer workstations.
Does that help any to clarify the situation?
@JoeyAcc thanks for the explanation! Do you know the difference in configuration between the jenkins server and your local machines? Is the certificate installed in a different location or perhaps the method of installation differs? Unfortunately I don't know a whole lot about debugging OpenSSL loading certificates from the filesystem, so without access to strace
and the ability to poke around I can only try to offer help from afar :(
That's ok, I'm already very happy that you're willing to help me solve this issue (as that is not a given for maintainers in general my experience) :)
There are indeed some differences between the build server and the dev workstations:
Jenkins Build Server |
Dev Workstations |
|
---|---|---|
OS |
Ubuntu 16.04 | Wndows 10 |
cert install method |
manual* | certmgr.msc |
cert installed location |
/etc/ssl/certs | ???** |
Those are the important differences that I can think of ATM.
I do have a question: What exactly would you apply strace
to? Perhaps I can replicate the steps locally and then provide the results.
*Manual here means that I manually copied the cert file to /usr/local/share/ca-certificates/accept2017.crt
, and then I ran sudo update-ca-certificates
as outlined here, which (probably among other things) makes a symlink to the cert in /etc/ssl/certs
named accetp2017.pem
. Initially I had not installed the .key
file to /etc/ssl/private
as outlined here, but I just tried exactly that and the result is the same failure as in my OP.
** I have no idea where certmgr.msc
actually stores its certificates.
Hm ok that's definitely quite odd, that sounds like it's all you'd need to do to install the cert on Ubuntu. I know though that the cert store on Windows is a little easier to install to than on Ubuntu, so that may at least explain a bit of the difference here!
I think the next thing to do may be to run strace
over cargo
itself. That'll tell us which cert files Cargo's actually trying to read from the filesystem. Could you ssh directly into the Jenkins server to run something like strace -f cargo build
? That should hopefully fail the same way your works are failing and provide some useful output to crawl over.
Below is the strace
output I got. There is indeed some kind of error in there.
Ok so early on there we see:
[pid 17115] stat("/var/ssl", 0x7ffd8200f610) = -1 ENOENT (No such file or directory)
[pid 17115] stat("/usr/share/ssl", 0x7ffd8200f610) = -1 ENOENT (No such file or directory)
[pid 17115] stat("/usr/local/ssl", 0x7ffd8200f610) = -1 ENOENT (No such file or directory)
[pid 17115] stat("/usr/local/openssl", 0x7ffd8200f610) = -1 ENOENT (No such file or directory)
[pid 17115] stat("/usr/local/share", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 17115] stat("/usr/lib/ssl", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 17115] stat("/usr/ssl", 0x7ffd8200f610) = -1 ENOENT (No such file or directory)
[pid 17115] stat("/etc/openssl", 0x7ffd8200f610) = -1 ENOENT (No such file or directory)
[pid 17115] stat("/etc/pki/tls", 0x7ffd8200f610) = -1 ENOENT (No such file or directory)
[pid 17115] stat("/etc/ssl", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 17115] stat("/data/data/com.termux/files/usr/etc/tls", 0x7ffd8200f610) = -1 ENOENT (No such file or directory)
notably, these three directories exist:
[pid 17115] stat("/usr/local/share", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 17115] stat("/usr/lib/ssl", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 17115] stat("/etc/ssl", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
Oddly though we don't see sycalls after this happens. These stat
syscalls correspond to this loop but the try
functions are never invoked because apparently env vars like SSL_CERT_FILE
and SSL_CERT_DIR
are already set?
@JoeyAcc are those env vars already set by jenkins? If so, maybe those aren't configured right? Something I saw was probing for /etc/ssl/certs/244b5494.0
which looks like some sort of compiled database?
Around the time of the failed verification we see:
stat("/etc/ssl/certs/c43a77d9.0", 0x7ffd81fed1e0) = -1 ENOENT (No such file or directory)
so presumably that file should exist and have the custom GH enterprise certificate?
Indeed the env var SSL_CERT_FILE
was set to /home/accept/cert/accept2017.crt
, and SSL_CERT_DIR
was set to /etc/ssl/certs
.
Additionally, the file /etc/ssl/certs/244b5494.0
did not exist (it seems to be generated by the update-ca-certificates
cmd).
However, I just unset
them both and ran update-ca-certificates
(after which /etc/ssl/certs/244b5494.0
does exist) and strace
again, and still got a very similar result from it:
@JoeyAcc ah ok yeah so that strace
looks like it is indeed doing the probing logic (no env vars set as you have done) and it looks like it's looking at /usr/lib/ssl/certs/ca-certificates.crt
for certificates. Do you know if that file is updated with the update-ca-certificates
command?
(are there maybe multiple installations of OpenSSL CA certificates on this system?)
My colleague tells me that it's very likely that update-ca-certificate
indeed does update /usr/lib/ssl/certs/ca-certificates.crt
. Additionally, both he and I think that there is only 1 OpenSSL installation i.e. the one that shipped with the Ubuntu image (the "duplicates" are old versions as reported by apt-cache
, which is part of Ubuntu's package manager APT
), which is to say neither of us installed any additional OpenSSL instances:
accept@vm-jenkins2:~$ apt-cache show openssl
Package: openssl
Priority: standard
Section: utils
Installed-Size: 934
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Debian OpenSSL Team <pkg-openssl-devel@lists.alioth.debian.org>
Architecture: amd64
Version: 1.0.2g-1ubuntu4.8
Depends: libc6 (>= 2.15), libssl1.0.0 (>= 1.0.2g)
Suggests: ca-certificates
Filename: pool/main/o/openssl/openssl_1.0.2g-1ubuntu4.8_amd64.deb
Size: 491720
MD5sum: ea3ad9b613c675a31cf9e8111987c88d
SHA1: 7eccd5cea88d4a117a12c6093608c9e1ef97dd19
SHA256: 3a1d993937d7fc05f65408e2e120e8c834d9600473005a6d43c1b67cadbd3fbd
Description-en: Secure Sockets Layer toolkit - cryptographic utility
This package is part of the OpenSSL project's implementation of the SSL
and TLS cryptographic protocols for secure communication over the
Internet.
.
It contains the general-purpose command line binary /usr/bin/openssl,
useful for cryptographic operations such as:
* creating RSA, DH, and DSA key parameters;
* creating X.509 certificates, CSRs, and CRLs;
* calculating message digests;
* encrypting and decrypting with ciphers;
* testing SSL/TLS clients and servers;
* handling S/MIME signed or encrypted mail.
Description-md5: 9b6de2bb6e1d9016aeb0f00bcf6617bd
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Origin: Ubuntu
Supported: 5y
Task: standard, ubuntu-core, ubuntu-core, mythbuntu-frontend, mythbuntu-backend-slave, mythbuntu-backend-master, ubuntu-touch-core, ubuntu-touch, ubuntu-sdk-libs-tools, ubuntu-sdk
Package: openssl
Priority: standard
Section: utils
Installed-Size: 934
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Debian OpenSSL Team <pkg-openssl-devel@lists.alioth.debian.org>
Architecture: amd64
Version: 1.0.2g-1ubuntu4.6
Depends: libc6 (>= 2.15), libssl1.0.0 (>= 1.0.2g)
Suggests: ca-certificates
Filename: pool/main/o/openssl/openssl_1.0.2g-1ubuntu4.6_amd64.deb
Size: 491668
MD5sum: 5a10ca1cfbd6ebad80264e2c2106c752
SHA1: 9ffe8c4291c09fd996a3afafdf85df8f5d9520ed
SHA256: 18f4d36f3e624be6ecfab385cd2e2fe4e5d7abc8030505dcacfa3ab243253cd1
Description-en: Secure Sockets Layer toolkit - cryptographic utility
This package is part of the OpenSSL project's implementation of the SSL
and TLS cryptographic protocols for secure communication over the
Internet.
.
It contains the general-purpose command line binary /usr/bin/openssl,
useful for cryptographic operations such as:
* creating RSA, DH, and DSA key parameters;
* creating X.509 certificates, CSRs, and CRLs;
* calculating message digests;
* encrypting and decrypting with ciphers;
* testing SSL/TLS clients and servers;
* handling S/MIME signed or encrypted mail.
Description-md5: 9b6de2bb6e1d9016aeb0f00bcf6617bd
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Origin: Ubuntu
Supported: 5y
Task: standard, ubuntu-core, ubuntu-core, mythbuntu-frontend, mythbuntu-backend-slave, mythbuntu-backend-master, ubuntu-touch-core, ubuntu-touch, ubuntu-sdk-libs-tools, ubuntu-sdk
Package: openssl
Priority: standard
Section: utils
Installed-Size: 934
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Debian OpenSSL Team <pkg-openssl-devel@lists.alioth.debian.org>
Architecture: amd64
Version: 1.0.2g-1ubuntu4
Depends: libc6 (>= 2.15), libssl1.0.0 (>= 1.0.2g)
Suggests: ca-certificates
Filename: pool/main/o/openssl/openssl_1.0.2g-1ubuntu4_amd64.deb
Size: 492190
MD5sum: 8280148dc2991da94be5810ad4d91552
SHA1: b5326f27aae83c303ff934121dede47d9fce7c76
SHA256: e897ffc8d84b0d436baca5dbd684a85146ffa78d3f2d15093779d3f5a8189690
Description-en: Secure Sockets Layer toolkit - cryptographic utility
This package is part of the OpenSSL project's implementation of the SSL
and TLS cryptographic protocols for secure communication over the
Internet.
.
It contains the general-purpose command line binary /usr/bin/openssl,
useful for cryptographic operations such as:
* creating RSA, DH, and DSA key parameters;
* creating X.509 certificates, CSRs, and CRLs;
* calculating message digests;
* encrypting and decrypting with ciphers;
* testing SSL/TLS clients and servers;
* handling S/MIME signed or encrypted mail.
Description-md5: 9b6de2bb6e1d9016aeb0f00bcf6617bd
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Origin: Ubuntu
Supported: 5y
Task: standard, ubuntu-core, ubuntu-core, mythbuntu-frontend, mythbuntu-backend-slave, mythbuntu-backend-master, ubuntu-touch-core, ubuntu-touch, ubuntu-sdk-libs-tools, ubuntu-sdk
accept@vm-jenkins2:~$
Ah yeah unfortunately I've never install a custom SSL certificate on a linux system so I'm not even sure what the "normal" mechanism for doing this is and if the behavior we're seeing diverges from that...
I'm running low on ideas :(
I don't exactly when it will happen yet, but there are plans to create a new build slave image from scratch and see if that will work. At this point for all we know it could be a borked Ubuntu image.
I'll post the results in this thread once I have them.
I'm going to close this issue due to a lack of activity... no news is good news, right???
Please reopen if this is still a problem! Thank you!!
We have a GitHub Enterprise server here at work, and a project setup that uses neon to bind Rust code to Node.JS.
In terms of dependencies the setup roughly looks like this:
top-level consumer app (sbr-viewer-ws)
←rust-backed npm module (sbr-validate-npm)
←multiple rust dependencies (regular rust crates e.g. sbr-xquery-interpreter-api)
. Manual cloning and pulling works fine on both GitHub.com and GHE, but this fails:I can't be sure but since manual cloning works fine, I'm inclined to believe that the certificates are properly installed, and Cargo somehow can't handle it. I've also heard on multiple occasions that Cargo uses
libgit2
and thatlibgit2
doesn't handle custom certificates properly. I have to wonder: is that the root cause of this issue?