Investigate php5-curl issues with SNI and redirects

[PVince81]

Whenever an app in the the app store points to a Github URL (or some other servers), some curl on some environments will show:

{"reqId":"zM09Y0\/eIzFoer4ohglK","remoteAddr":"127.0.0.1","app":"core","message":"cURL error 60: SSL certificate problem: unable to get local issuer certificate","level":3,"time":"2015-07-07T13:06:33+00:00","method":"POST","url":"\/owncloud\/index.php\/settings\/ajax\/enableapp.php"}

It happened for me on the following setup:

• openSUSE Tumbleweed
• curl-7.42.1-1.1.x86_64
• php5-5.6.9-1.1.x86_64
• php5-curl-5.6.9-1.1.x86_64

The problem was with the calendar app which redirected to the following URL: https://github.com/owncloud/calendar/releases/download/v0.7.0/calendar.zip which itself seems to redirect to an amazon S3 link.

See https://github.com/owncloud/documentation/pull/1290 and https://github.com/owncloud/calendar/issues/826#issuecomment-119180578

@LukasReschke

[PolishedStaple]

So do I need to wait until 8.2 is released before my very basic installation of ownCloud with a shared calendar will work? Right now I'm getting the same error as mentioned by PVince81 when I try to enable the calendar app.

Thank you.

[DeepDiver1975]

@nestsman in case you have ssh access to the server you can also manually install apps.

• wget the app from the appstore
• extract the zip/tar into the apps folder
• enable the app in the apps management

[gamanakis]

I face the same problem with FreeBSD 10.1. Could this have to do with an older OpenSSL version? Mine is 1.0.1l with cURL 7.43.

[gamanakis]

I tried with OpenSSL 1.0.2c from ports, recompiled cURL and php5-curl (5.4.42), however this doesn't resolve it.

[ghost]

@DeepDiver1975 tried this with the Calendar app; still getting the cURL error 60 when trying to enable from the web interface.

[josh4trunks]

I thought this had to do with PHP not being able to find a certificate store but my hypothesis was wrong. Playing with php.ini's curl.cainfo, openssl.cafile, and openssl.capath didn't solve the issue.

@gamanakis seems it doens't have to do with old versions of openssl either, as what your testing with is very current.

[PVince81]

My distro where the problem appeared has openssl-1.0.2a-1.1.x86_64 My productive server where it worked properlty has openssl-1.0.1k-11.72.1.x86_64

@entyrion maybe it's trying to update the app. Did you get the correct version of the calendar app from Github ? It should be 0.7.0. Have a look in the file "apps/calendar/appinfo/version"

[gamanakis]

I did a git bisect. Known good commit: 8.1 RC1 d5ef1f28ce0d4014161155e2d0a2493bdba52913 Known bad commit: 8.1 RC2 e7a30e091d8ea79b14117ca47d6c2fd2dc61a48a So: ae3d402dbc3c9620cea5b32bedc7f00874b12dd0 is the first bad commit commit ae3d402dbc3c9620cea5b32bedc7f00874b12dd0 Author: Joas Schilling nickvergessen@owncloud.com Date: Mon Jun 29 11:15:56 2015 +0200

Send OC version to appstore so it can reply with the right links

:040000 040000 d3b7bb600bc8ec37ab5afb2f1c22905cc37e898f 2a78d56b5cdc508527d93638b4884ff1720046de M lib :040000 040000 aeb5c93adbac570d85c97bd11a32efb8ea43c8f1 b93b29d4137a1abdb0e5c034ac166919a9911c73 M settings :040000 040000 898d6d580c7917f818698d8154be41ec51a6679f 210fae7a1328a18fdb00ab3b8657564b63de14b6 M tests

Could someone verify this?

[PVince81]

I don't see anything related to SSL verification in this commit. But the parameters seem to have changed. Maybe without the parameters the link that the app store delivered was not from Github, which made it work ? And the new link (for 8.1) points to Github and causes the redirect issue ?

[gamanakis]

False alarm. On my system (FreeBSD 10.1, see above) reversing selectively the above on the official 8.1.0 doesn't produce the error anymore but also doesn't install the apps (contacts and calendar), meaning probably that no connection at all is made to the apps-server.

[ruspantini]

unsuccessful update 8.0.4 to 8.1 #17534 false alarm too. Reinstall worked fine without apps (contacts and calendar). They can't be reinstalled : cURL error 60

[ruspantini]

I just manually installed the calendar and it works !!! I downloaded here Calendar 0.6.5 for ownCloud 8.1 then Extracted the contents from the downloaded archive to the apps folder. Then navigate to "Apps" with an administrative user, and enabled the app again. This time it worked. I have to try with Contacts and will report ...

[jankkm]

Yes that works. Not really a solution to the problem though.

[ruspantini]

Well I'm not a developer ... kind of a newbie and I was quite proud of having found a solution and maybe helping you all ;-) still trying with contacts ...

[ruspantini]

It works with Contacts too. I forgot to mention ownCloud 8.1 asks for reinstallation each time. After a quick reinstallation contacts and calendar show up. Have a great evening everyone. I'll be back tomorrow

[PolishedStaple]

I prefer to wait for 8.1.2 to fix the problem. Apparently there is not a QA team in place to catch this kind of stuff before it's released?

[gamanakis]

I think I resolved it for FreeBSD 10.1. In addition to curl, php5-curl and php5-openssl also apache24 has to be compiled against OpenSSL 1.0.2c from ports.

There seems to be a problem with the OpenSSL 1.0.1l installed in the base system regarding SNI. For example: OpenSSL 1.0.1l base system /usr/bin/openssl s_client -servername chrismeller.com -connect chrismeller.com:443 returns "Verify return code: 20 (unable to get local issuer certificate)"

while OpenSSL 1.0.2c from ports /usr/local/bin/openssl s_client -servername chrismeller.com -connect chrismeller.com:443 returns "Verify return code: 0 (ok)"

[josh4trunks]

@gamanakis thanks, I'll try recompiling everything with "WITH_OPENSSL_PORT=yes" and see if that fixes my use cases.

[gamanakis]

@PVince81

My distro where the problem appeared has openssl-1.0.2a-1.1.x86_64

See: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=201446 "This is a bug with OpenSSL since 1998 and fixed in 1.0.1n..."

[josh4trunks]

I can confirm recompiling everything with openssl 1.0.2d worked for me. I'm not sure what package needs to be compiled against the latest version, but I assume php.

[gamanakis]

Compiling only apache24 against OpenSSL 1.0.2d in FreeBSD 10.1 resolves the issue. I think it has to do with the underlying OpenSSL installation.

[ruspantini]

Hail to all, I still can't get Calendar and Contacts to install on ownCloud 8.1. Even the manual install doesn't work. Please help me. I don-t know how to get around this (newbie). I have downloaded DAVdroid to get the mobile to synch eventually with ownCloud but I guess I have to get Calendar and Contacts installed first ? Thanks in advance if you can help Luca

[ghost]

Hi Luca, I posted my solution to get my broken apps to work here https://forums.freenas.org/index.php?threads/owncloud-plugin-now-available-in-plugins-menu.15046/page-8, perhaps a similar workflow may work for you. Note that for me, Contacts oddly installed correctly from the apps interface; my issues were specifically with Calendar, Documents, and Bookmarks (I'm still having issues with Bookmarks). I hope this helps!

[ruspantini]

Hi entyrion, please forgive my late comeback. Thanks for your interest in my post. I went to see your solution and don't understand how to correct my "cURL error 60" when I try to install either Calendar or Contacts manually. I shall try to erase both directories and install via ownCloud Apps folder. Being a Newbie I don't understand such words as "compile" "Debian" and where to get the "plugin" what plugin and where to install it ? Thanks again if you can help me ? Edit : automatique install via web-interface gives same "cURL error 60"

[LukasReschke]

I prefer to wait for 8.1.2 to fix the problem. Apparently there is not a QA team in place to catch this kind of stuff before it's released?

@nestsman As you can see not every platform is affected. Please defer from posting discouraging comments that do not add substantial information to the problem. Yes, it is something we need to have a look. No, there is no need to rant here. Thanks.

See also https://owncloud.org/faq/#upgradeyet:

All code entering ownCloud has been reviewed twice as well as tested by both automated means and through the efforts of volunteers. Releases are done only when all issues we are aware of are fixed or can be mitigated easily. In short, ownCloud releases are always as stable as we could make it without your help. If you want the guarantee that it will work in your specific situation, make sure you help test before a release is made so your issue can be fixed in time.

[LukasReschke]

As far I can see the bug in OpenSSL pointed out by Xin Li has been patched with https://github.com/openssl/openssl/commit/fe9b85c3cb79f1e29e61f01de105b34ce8177190 in the OpenSSL 1.0.1 branch which is also ported to 1.0.2 with https://github.com/openssl/openssl/commit/8338cf0bdb9a5b17cd46a0bf8f58c15f10960586.

When checking the tags that this is in I end up with OpenSSL_1_0_2b, which means that this is not in the 1.0.2a release that @PVince81 is using. @PVince81 please get your upstream distribution to ship newer things :speak_no_evil: – However as per http://download.opensuse.org/factory/repo/src-oss/suse/src/ I only can find 1.0.2d nowadays.

This again reminds me why backporting patches is one of the most thing that I really dislike about Linux distributions. We're now left with a situation where an upstream bug is fixed in 1.0.1d as well as 1.0.2b but we cannot detect this reliably as distributions may just backport this fix instead of bumping the version. Heuristics are also not very good here.

That heuristics are not very good here is also shown by the fact that @PVince81 uses openssl-1.0.1k-11.72.1.x86_64 which however has per upstream not this patch (k < n). When I downloaded the OpenSUSE RPM however I found that they apply some magic in openssl-1.0.1c-default-paths.patch which per openssl.changes is: "Fix from Fedora for openssl s_client not setting CApath by default". I have a strong suspicion that this is why it works there as the logic is a little bit different there. (and it patches the same bits) – Let me state though that I'm not a OpenSSL contributor and have barely an idea how the code behaves there and can't be 100% sure though.

So what I'd recommend is: If somebody experiences this error please get back to your distributor and ensure that they ship newer releases. The patch has been authored on 25 Februar 2015 and distributions really ought to ship the newest stuff. Or at least newer bug fixes…

I'm not sure if we want to add heuristics for old OpenSSL versions as this may anyway works though in case distributions just backport this patch. I will leave this up to @karlitschek.

As a workaround we could stop using SNI anywhere which is however not feasible for S2S and a lot of other stuff. We can maybe do this for the app store but then other stuff will break anyways as well…

So: Huge fun.

Thanks a lot for all the people that helped to investigate this. Especially @gamanakis, I really appreciate that you took the time to file an issue at FreeBSD. This is much appreciated.

[LukasReschke]

Closing as this is an upstream issue. Please complain upstream to distributions if you experience this error. They shall update to at least 1.0.1d or 1.0.2b.

[DeepDiver1975]

@LukasReschke thanks for digging into this - much appreciated!

Would it make sense to have an admin warning about outdated/broken openssl version?

[LukasReschke]

Would it make sense to have an admin warning about outdated/broken openssl version?

We can't detect the version reliably since this patch has got backported on some distributions (such as Fedora).

We only could use an heuristic which means that we need to do some requests to an affected host. Thinking about it: This works but will fail hard once the server is not reachable anymore (i.e. offline) since the heuristic will detect this as a bug while in fact the server is just not properly reachable. Will get us quite some additional bug reports.

What I can think about is doing a check for versions lower than 1.0.1d or 1.0.2b and mentioning that this "may" cause problems if the distribution did not backport recent patches. But then again a short check showed me:

Debian

:white_check_mark: Wheezy ships with 1.0.1e :white_check_mark: Jessie ships with 1.0.1k :white_check_mark: Stretch ships with 1.0.2d

Ubuntu

:question: 12.04 ships with an undefined version :white_check_mark: 14.04 ships with 1.0.1f :white_check_mark: 14.10 ships with 1.0.1f :white_check_mark: 15.04 ships with 1.0.1f

CentOS / RHEL

:white_check_mark: 6 ships with 1.0.1e :white_check_mark: 7 ships with 1.0.1e

From what I can tell all major distributions are up-to-date. If we're going to add checks for bugs in versions for each dependency we have this is going to be a major nightmare. From my PoV: Users shall update their stuff. It's that easy. :speak_no_evil:

[DeepDiver1975]

Thanks a lot!

[ruspantini]

Hi, I noticed this thread is closed. Forgive me for this last request. Please redirect me to where eventually the solution might be or come in the future ... I'll stay watching there if I can repair my ownCloud. Thanks in advance and apologies for posting here.

[ghost]

@ruspantini See https://github.com/owncloud/core/issues/17446#issuecomment-122875425 that you can't repair oC as this is an issue in openssl itself.

[b1ggjoe]

RealRancor,

Is it as simple as just updating or recompiling an old version of OpenSSL on our Distro that is hosting ownCloud to a new version?

J.

[ghost]

@b1ggjoe Which versions are shipping the bugfix are described above. Everything else is probably something you should clarify with your distro or a forums/community dedicated to it.

[b1ggjoe]

RealRancor,

Ok, I will check with the community for my Distro (CentOS). I'm just curious if anyone else has been able to fix this issue at all.

I was looking into the OpenSSL library on my Server, and it does look like it did receive a back-port fix:

root@xxxxx [/etc/yum.repos.d]# rpm -q --changelog openssl | grep CVE-2014-0224

• fix CVE-2014-0224 fix that broke EAP-FAST session resumption support

- fix CVE-2014-0224 - SSL/TLS MITM vulnerability

I will keep you all posted as to what I find out.

BJ

[b1ggjoe]

Gents,

I just updated and recompiled Apache and OpenSSL on my server. I'm now running the following:

• Apache Web Server latest build, version v2.4.16 located here:

http://httpd.apache.org/download.cgi#apache24

1. - OpenSSL library latest version: openssl-1.0.2d located here:

https://www.openssl.org/source/

I still cannot enable Calendar, Contacts and othe apps from the App Store.

Any other ideas?

J.

[josh4trunks]

@b1ggjoe just because it's apache's latest build doesn't mean it was compiled on openssl v1.02 right? I think there's a way to check what version of openssl it was compiled with.

[b1ggjoe]

Correct,

That is why we compiled it with the latest of both. If I enter 'opensslversion' I get this: OpenSSL 1.0.2d 9 Jul 2015.

BJ

[josh4trunks]

Hmm, not sure then. I know recompiling everything (apache / php / openssl) worked for me, but that probably doesn't help much =/

[ghost]

Yes, re-compiling only Apache and Openssl won't help here as the connection to the appstore is done from PHP by using php-curl. So at least PHP needs to be linked also but not quite sure about php-curl/curl.

[b1ggjoe]

Ok. So has anyone been able to correct this error that can provide a step-by-step as to how to properly do this?

[PVince81]

I updated my openSUSE Factory recently and now openssl is openssl-1.0.2d-1.1.x86_64 and the app store issues have disappeared.

[b1ggjoe]

Gents, I found the following article. I'm not sure if their method will fix our issues once and for all. I'm trying to do this on my server, but I'm running into issues.

Would anyone be willing to give this article a try?

Basically it's using the instructions on here:

http://flwebsites.biz/posts/how-fix-curl-error-60-ssl-issue

BJ

[b1ggjoe]

Since the ownCloud Server is trying to connect to: https://apps.owncloud.org/

I thought I would check it out, just to see if I notice anything.

Could the following errors be causing these strange issues at all by any chance?

Here's the screenshot: http://i.imgur.com/n3OLHR0.png

BJ

[karlitschek]

Are you sure you don't mismatch .org and .com? ownCloud should connect to apps.owncloud.com which should have a valid cert. Can you please double-check?

[b1ggjoe]

Karl,

What would be the most reliable way to double-check this from my ownCloud config or install?

BJ

[b1ggjoe]

I also just updated my cURL library to: 7.43.0.

But still, no dice...still the same error: cURL error 60: SSL certificate problem: unable to get local issuer certificate

Does anyone else have any ideas, recommendations or etc?

Has anyone tried this:

http://flwebsites.biz/posts/how-fix-curl-error-60-ssl-issue

BJ

[ghost]

@b1ggjoe You need to update OpenSSL and make sure PHP and curl is using that new version as already posted above.

[b1ggjoe]

Yep, I updated and recompiled Apache to 2.4, OpenSSL to OpenSSL 1.0.2d 9 Jul 2015 and cURL to curl-7.43.0-2.0.cf.rhel5.x86_64.rpm. The latest and greatest for all of them.

[ghost]

PHP is missing here as already written above. PHP and also the curl PHP module needs to be linked against the new OpenSSL version as also written above.