Cargo: "The SSL certificate is invalid" on Debian 9 (latest stable)

[eguiraud]

Encountered a similar issue to #598 with the cargo binaries for linux 64 bit on Debian 9.2.

I unpacked the 64-bit binaries for linux found here, and use them to compire a rust project (i3status-rust, but I don't think it matters). The error was the following:

build --release --verbose
    Updating registry `https://github.com/rust-lang/crates.io-index`
error: failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  [16/-17] The SSL certificate is invalid

The workaround mentioned in that issue still works: executing

export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

before running cargo fixes the certificate issue.

[alexcrichton]

@bluehood mind running something like perf trace or strace over cargo and gisting the logs? I'm curious what directory Cargo is automatically picking up and how that may be wrong.

[eguiraud]

Here it is

[alexcrichton]

Ok it looks like Cargo is finding /usr/lib/ssl/certs/ca-certificates.crt first and using that. Do you know why that file exists and why it may not be valid when the /etc version is valid?

[eguiraud]

I don't know, but I guess you can check on packages.debian.org what package installs that file

On 6 Dec 2017 20:13, "Alex Crichton" notifications@github.com wrote:

Ok it looks like Cargo is finding /usr/lib/ssl/certs/ca-certificates.crt first and using that. Do you know why that file exists and why it may not be valid when the /etc version is valid?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rust-lang/cargo/issues/4785#issuecomment-349743838, or mute the thread https://github.com/notifications/unsubscribe-auth/AKfU-mDxTgvlxyEnwg_4F3cOqogdm6uLks5s9udcgaJpZM4Q4Ozm .

[alexcrichton]

Sorry I'm not very familiar with debian so I wouldn't really know where to start to investigate this :(

[eguiraud]

I quickly searched packages.debian.org and packages.ubuntu.com and could not find packages that contain /usr/lib/ssl/certs/ca-certificates.crt (note that I'm not an expert either). apt-file search/usr/lib/ssl/certs/ca-certificates.crt`, which should return the name of the package responsible for that file, prints nothing.

@WhoAteDaCake is on Ubuntu 16.04 and he is also affected. @WhoAteDaCake, do you also have a certificate like that in the same location? I'm wondering how it got there.

@alexcrichton, you might know: can these files be safely deleted?

[lukaslueg]

AFAIK /usr/lib/ssl/cert is thought to be a symlink to /etc/ssl/certs, having no right to be on it's own. Maybe you want to file a bug against Debian9 regarding the straying (and probably outdated) cert-files in /usr/lib.

Maybe check out update-ca-certificates ?

[eguiraud]

Ah, thank you @lukaslueg, that is certainly the case: /usr/lib/ssl/cert is a symlink to /etc/ssl/certs. I had only checked if ca-certificates.crt itself was a symlink to something else, but not its containing directory.

To vanquish all doubts: the inode of the two files is the same

~ ls -i /etc/ssl/certs/ca-certificates.crt 
146245 /etc/ssl/certs/ca-certificates.crt
~ ls -i /usr/lib/ssl/certs/ca-certificates.crt 
146245 /usr/lib/ssl/certs/ca-certificates.crt
~

So it seems /usr/lib/ssl/certs/ca-certificates.crt is not the one that is picked up by cargo (otherwise the workaround described in the first post would not change anything right?)

Finally, I ran update-ca-certificates --fresh as suggested, with no effect.

I don't know much about certificates. As far as I know I haven't changed them or installed custom ones. It would be interesting to know whether the problem is on my specific installation or the rust binaries are not working on vanilla Debian 9 installations.

[lukaslueg]

Could you post an strace-dump with SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

It's mildly suspicious that Cargo tries to access a hardcoded certificate path to the tune of /buildslave/rust-buildbot/slave/nightly... right before it finally bails.

[eguiraud]

Here you go. This one was much bigger, as one would expect (~50MB of text file), so I trucated it to the first 1000 lines (if I'm not mistaken it's already past the first github download), let me know if you need more.

[eguiraud]

As an experiment I pulled a docker image of debian stable and tried the cargo binaries on that (same task, compiling i3status-rust, same binaries, downloaded from here). Vanilla image + wget, rustc, git installed from package manager (seems that wget brings in the ca-certificates by the way).

cargo still chokes on the SSL certificates -- so at least we know it was not something I did to my debian installation :D

[eguiraud]

one more update: if I install stable cargo via https://static.rust-lang.org/rustup.sh script as described here there are no problems with certificates.

So stable cargo installed with rustup.sh is not affected but nightly cargo as per compiled binaries is.

[lukaslueg]

The juicy parts can be found by searching for "urandom" in both traces.

The working example picks up the cert-file right away, reads it and apparently uses the information later on without problems. The non-working example tries to read a cert-file from /buildslave/rust-buildbot/slave/nightly-dist-cargo-linux-64/build//target/openssl/x86_64-unknown-linux-gnu-install/ssl/cert.pem, which is a path from the time the compiler was built against local openssl on the build-bot. It fails to do that and later on tries to read files from that path again right before everything bails. It does successfully stat() the cert-files in /usr/lib, yet does not use this information later on.

It might be possible that the openssl-parts got some of their configuration hardwired into the final build. Maybe @alexcrichton can draw attention from someone more familiar with the subject.

[alexcrichton]

@bluehood er actually what version of cargo are you using? I think the ones you're using may be ancient by this point, or are you installing with rustup.rs?

[eguiraud]

As per links provided, I'm downloading the nightly binary builds from http://doc.crates.io/#install-nightly-cargo

Installing cargo via rustup.sh (available at the same link) provides a working version of cargo (but requires sudo so I can't really use that method for deployment)

[alexcrichton]

Yes can you get the version of that Cargo? I think those links are super out of date and need to be removed :(

[eguiraud]

Yes can you get the version of that Cargo?

Shouldn't you guys know? 😅 Downloading the tar archive at the usual page, and running cargo --version on the binary inside I get

cargo 0.16.0-nightly (3568be9 2016-11-26)

[alexcrichton]

Ah yes at over a year old that's a pretty out of date Cargo. Can you try using rustup and see if recent versions work for you?

[eguiraud]

As per my comments above, rustup works, but requires sudo privileges, which makes it a no-go for deployment on certain machines

[alexcrichton]

Ok sure, but I think that's the bug, not this. We can't really evaluate whether a year-old Cargo is buggy because of something still present or whether it's been fixed.

[eguiraud]

I agree. This issue can be closed as the problem is now clear. Nevertheless it's very misleading to leave one-year-old binaries under the title "install nightly cargo"...

[alexcrichton]

Ok, I'll close this in that case.

Mind opening an issue up on crates.io about removing the install links?

[eguiraud]

done

[carols10cents]

@alexcrichton the docs are in this repo though, i'm moving that issue back over here :P

[alexcrichton]

Oh oops, indeed!