Red Hat requires path to curl's cacert for https to work

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Try to use https with s3fs.  Access will fail with:
s3fs: ###problem with the SSL CA cert (path? access rights?)

What is the expected output? What do you see instead?
bucket access fails

What version of the product are you using? On what operating system?
s3fs-1.10
Red Hat Enterprise Linux Server release 5.5 (Tikanga)
CentOS release 5.5 (Final)

Please provide any additional information below.
To fix update file:
src/s3fs.cpp
@119
curl_easy_setopt(curl, CURLOPT_CAINFO, "/etc/pki/tls/certs/ca-bundle.crt");

recompile.

Original issue reported on code.google.com by jonas%va...@gtempaccount.com on 19 Nov 2010 at 11:03

[GoogleCodeExporter]

Just to clarify, this issue is seen on both the Red Hat and CentOS 
distributions?

Also please supply either your /etc/fstab entry or command line innvocation.

I did not prelim checking on this and admittedly, I'll not very familiar with 
the Curl option usage. We do need to be careful that by fixing this issue, we 
are not breaking s3fs on other systems.  

It appears that we may need to check for the ca-bundle.crt file's 
existance/readibility and be sure that https is being used.

A quick check on my machines finds that this file exists on my Ubuntu machine 
in several places:

139940  148 -rw-r--r--   1 root     root       149766 Aug 11 12:06 
./usr/share/ncat/ca-bundle.crt
1587062  172 -rw-r--r--   1 root     root       172729 Mar 10  2010 
./usr/share/kde4/apps/kssl/ca-bundle.crt
266277  160 -rw-r--r--   1 root     root       162145 May 14  2007 
./usr/share/apps/kssl/ca-bundle.crt

...but doesn't exist on my Debian (sid) machines.  ???

Original comment by dmoore4...@gmail.com on 20 Nov 2010 at 6:20

[GoogleCodeExporter]

Also, what version of curl are you using:

> curl --version
curl 7.21.0 (i686-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 
libidn/1.18
Protocols: dict file ftp ftps http https imap imaps ldap ldaps pop3 pop3s rtsp 
smtp smtps telnet tftp 
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz 

Original comment by dmoore4...@gmail.com on 20 Nov 2010 at 11:59

[GoogleCodeExporter]

Just to clarify, this issue is seen on both the Red Hat and CentOS 
distributions?
- YES

Also please supply either your /etc/fstab entry or command line innvocation.
- /etc/fstab:
/usr/local/bin/s3fs#test.server /mnt/s3/test.server fuse 
allow_other,url=https://s3.amazonaws.com

Also, what version of curl are you using:
- curl 7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b 
zlib/1.2.3 libidn/0.6.5

- I agree that the location of the bundle is distro sensitive and could break 
if incorrectly set.  The "curl_easy_setopt(curl, CURLOPT_CAINFO, 
"/some/path/ca-bundle.crt");" is distro-independent.  Can you the path to 
ca-bundle.crt be included in the ./configure process?  Checking to see if the 
file exists would be especially helpful.

This was tested behind a firewall that only allow tcp/443 out to verify the 
https port is being used.

Original comment by jonas%va...@gtempaccount.com on 23 Nov 2010 at 7:46

[GoogleCodeExporter]

So I spun up a CentOS 5.5 virtual machine and compiled/tested/installed s3fs 
successfully.  And was able to reproduce this issue:

/var/log/messages:Nov 24 13:33:40 centos s3fs: curlCode: 60  msg: problem with 
the SSL CA cert (path? access rights?)
/var/log/messages:Nov 24 13:33:40 centos s3fs: init $Rev: 245 $
/var/log/messages:Nov 24 13:33:48 centos s3fs: ###problem with the SSL CA cert 
(path? access rights?)
/var/log/messages:Nov 24 13:33:48 centos s3fs: ###retrying...
/var/log/messages:Nov 24 13:33:48 centos s3fs: ###problem with the SSL CA cert 
(path? access rights?)
/var/log/messages:Nov 24 13:33:48 centos s3fs: ###retrying...
/var/log/messages:Nov 24 13:33:48 centos s3fs: ###problem with the SSL CA cert 
(path? access rights?)
/var/log/messages:Nov 24 13:33:48 centos s3fs: ###retrying...
/var/log/messages:Nov 24 13:33:48 centos s3fs: ###giving up

So the good news is that I have a platform in which to reproduce this issue.

I have also confirmed that this is not an issue on my Debian installation using 
the same setup.

BUT, the ca-bundle.crt file does not exist on my Debian box (but it does exist 
on the Ubuntu install) ???

Admittedly, this is an area which I do not completely understand and my 
searching has probably given me enough information to be dangerous.

I suspect that the two different curl versions have something to do with it.  
7.15.5 on CentOS and 7.21.2 on Debian (sid)

I found a blurb on a change that occurred with curl 7.18 concerning certs which 
may or may not have anything to do with this: 
http://curl.haxx.se/docs/sslcerts.html

....

I've gotten a bit further since first composing this comment.  I am now 
trapping this curl error during the "check_service" routine with the latest 
subversion checkin.  This will prevent the nebulous I/O error and will prevent 
s3fs from starting giving a false sense of success.

Original comment by dmoore4...@gmail.com on 24 Nov 2010 at 10:47

• Changed state: Started

[GoogleCodeExporter]

This certainly is a strange one and definitely seems to be related to
how libcurl has been compiled for each distribution.  My test command
was this:

s3fs misc.suncup.org ~/misc.suncup.org -o url=https://s3.amazonaws.com

On Debian and Ubuntu, this works as expected. But on CentOS and Fedora,
this fails (but in different ways).

I used strace to try and figure out where libcurl was looking for the
certificate(s). On Debian/Ubuntu, it is looking in /etc/ssl/certs rather
than looking for a particular bundle file.

On CentOS, I can't tell where it is looking for the certificate, but
curl ends up returning an error: 

CURLE_SSL_CACERT (60)
Peer certificate cannot be authenticated with known CA certificates.

As jonas points out, this can be cured by setting the curl
CURLOPT_CAINFO option to point to the certificate file.  (I should also
point out that the issue can be resolved by setting the
CURLOPT_SSL_VERIFYPEER option to false, but we probably don't want to do
that).

Also we should note that this issue only shows itself when the https://
is used.

So I have found a solution to this issue that fixes it for CentOS 5.5
and doesn't break it for Debian/Ubuntu.  What the fix does is it "traps"
the CURLE_SSL_CACERT error.  The first time that this error is seen,
then the program tries to determine the setting for CURLOPT_CAINFO.
First, it checks to see if the CURL_CA_BUNDLE environment variable is
set. (This is the standard way to set this, see the curl man page).  If
it is set, it checks it for readility, sets the CURLOPT_CAINFO option
and tries again.  If the env var is not set, then it looks in a typical
location of the bundle file as outlined by the acinclude.m4 file that is
included in the curl source:  

acinclude.m4
========================================================
dnl CURL_CHECK_CA_BUNDLE
dnl -------------------------------------------------
dnl Check if a default ca-bundle should be used
dnl
dnl regarding the paths this will scan:
dnl /etc/ssl/certs/ca-certificates.crt Debian systems
dnl /etc/pki/tls/certs/ca-bundle.crt Redhat and Mandriva
dnl /usr/share/ssl/certs/ca-bundle.crt old(er) Redhat
dnl /usr/local/share/certs/ca-root.crt FreeBSD
dnl /etc/ssl/cert.pem OpenBSD
dnl /etc/ssl/certs/ (ca path) SUSE

Upon finding the file and it is readable, it sets the CURLOPT_CAINFO
option to point to it and tries again.

If upon trying again, if the CURLE_SSL_CACERT error is fired, then I
really don't know what to do and the program errors out.  It'll be that
way until we encounter this situation.

r269 just got checked in.  I think that this issue for CentOS is fixed.
However, Fedora has a different issue related to this. I'll open a new
issue for it.

If you're using subversion to get your code, then please give this a
try. Otherwise, let me know and I'll release another tarball.

Original comment by dmoore4...@gmail.com on 26 Nov 2010 at 3:16

[GoogleCodeExporter]

Original comment by dmoore4...@gmail.com on 26 Nov 2010 at 4:09

[GoogleCodeExporter]

jonas,

Have you been able to verify that the fix works or not?  Thanks. Dan

Original comment by dmoore4...@gmail.com on 2 Dec 2010 at 9:03

[GoogleCodeExporter]

No response from original reporter of issue. Assumed that this is fixed now.

Original comment by dmoore4...@gmail.com on 9 Dec 2010 at 9:23

• Changed state: Fixed