TLS connection certificate failure, certificate instructions?

[ehb54]

Trying to connect, we get a certificate issue Code:

<?php
echo "trying fsockopen to tls://api.scigap.org:9930\n";
$ret = fsockopen('tls://api.scigap.org', 9930 );
echo $ret ? "return true\n" : "return false\n";

Result (php 7.2.24 centos 8.3):

fsockopen to tls://api.scigap.org:9930
PHP Warning:  fsockopen(): SSL operation failed with code 1. OpenSSL Error messages:
error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed in /tmp/test3.php on line 4
PHP Warning:  fsockopen(): Failed to enable crypto in /tmp/test3.php on line 4
PHP Warning:  fsockopen(): unable to connect to tls://api.scigap.org:9930 (Unknown error) in /tmp/test3.php on line 4
return false

Result (php 5.4.16 centos 7.5)

trying fsockopen to tls://api.scigap.org:9930
return true

Are there instructions for certificate setup? Thanks, Emre

[ehb54]

Checking a bit further: php 7.2.24 centos8.3

php -i | grep ssl
Registered Stream Socket Transports => tcp, udp, unix, udg, ssl, tls, tlsv1.0, tlsv1.1, tlsv1.2
libSSH Version => libssh/0.9.4/openssl/zlib
openssl
Openssl default config => /etc/pki/tls/openssl.cnf
openssl.cafile => no value => no value
openssl.capath => no value => no value

php 5.4.16 centos7.5 (working with airavata)

php -i | grep ssl
Registered Stream Socket Transports => tcp, udp, unix, udg, ssl, sslv3, tls
http.request.datashare.ssl => 0 => 0
openssl

[ehb54]

php 7.2.24 centos8.3, running:

<?php
var_dump(openssl_get_cert_locations());

results in

array(8) {
  ["default_cert_file"]=>
  string(21) "/etc/pki/tls/cert.pem"
  ["default_cert_file_env"]=>
  string(13) "SSL_CERT_FILE"
  ["default_cert_dir"]=>
  string(18) "/etc/pki/tls/certs"
  ["default_cert_dir_env"]=>
  string(12) "SSL_CERT_DIR"
  ["default_private_dir"]=>
  string(20) "/etc/pki/tls/private"
  ["default_default_cert_area"]=>
  string(12) "/etc/pki/tls"
  ["ini_cafile"]=>
  string(0) ""
  ["ini_capath"]=>
  string(0) ""
}

which are supposedly used when php 7.2.24's openssl.cafile & .cpath are null

Those paths seem to have certificates:

ls -l /etc/pki/tls/certs
total 8
lrwxrwxrwx. 1 root root   49 Aug 11  2020 ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx. 1 root root   55 Aug 11  2020 ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
-rw-r--r--. 1 root root 3916 Jan 22 08:46 localhost.crt
-rw-r--r--. 1 root root 2216 Jan 22 09:18 postfix.pem
ls -l /etc/pki/tls/certs
total 8
lrwxrwxrwx. 1 root root   49 Aug 11  2020 ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx. 1 root root   55 Aug 11  2020 ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
-rw-r--r--. 1 root root 3916 Jan 22 08:46 localhost.crt
-rw-r--r--. 1 root root 2216 Jan 22 09:18 postfix.pem
[usadmin@demeler9 ~]$ ls -lR /etc/pki/ca-trust/
/etc/pki/ca-trust/:
total 8
-rw-r--r--. 1 root root 980 Aug 11  2020 ca-legacy.conf
drwxr-xr-x. 6 root root  70 Jan 22 05:56 extracted
-rw-r--r--. 1 root root 166 Aug 11  2020 README
drwxr-xr-x. 4 root root  80 Jan 22 06:00 source

/etc/pki/ca-trust/extracted:
total 4
drwxr-xr-x. 2 root root  39 Jan 22 06:00 edk2
drwxr-xr-x. 2 root root  35 Jan 22 06:00 java
drwxr-xr-x. 2 root root  47 Jan 22 06:00 openssl
drwxr-xr-x. 2 root root 101 Jan 22 06:00 pem
-rw-r--r--. 1 root root 560 Aug 11  2020 README

/etc/pki/ca-trust/extracted/edk2:
total 160
-r--r--r--. 1 root root 156842 Jan 22 06:00 cacerts.bin
-rw-r--r--. 1 root root    566 Aug 11  2020 README

/etc/pki/ca-trust/extracted/java:
total 160
-r--r--r--. 1 root root 157499 Jan 22 06:00 cacerts
-rw-r--r--. 1 root root    726 Aug 11  2020 README

/etc/pki/ca-trust/extracted/openssl:
total 248
-r--r--r--. 1 root root 249827 Jan 22 06:00 ca-bundle.trust.crt
-rw-r--r--. 1 root root    787 Aug 11  2020 README

/etc/pki/ca-trust/extracted/pem:
total 376
-r--r--r--. 1 root root 163655 Jan 22 06:00 email-ca-bundle.pem
-r--r--r--. 1 root root      0 Jan 22 06:00 objsign-ca-bundle.pem
-rw-r--r--. 1 root root    898 Aug 11  2020 README
-r--r--r--. 1 root root 216090 Jan 22 06:00 tls-ca-bundle.pem

/etc/pki/ca-trust/source:
total 4
drwxr-xr-x. 2 root root   6 Aug 11  2020 anchors
drwxr-xr-x. 2 root root   6 Aug 11  2020 blacklist
lrwxrwxrwx. 1 root root  59 Jan 22 06:00 ca-bundle.legacy.crt -> /usr/share/pki/ca-trust-legacy/ca-bundle.legacy.default.crt
-rw-r--r--. 1 root root 932 Aug 11  2020 README

/etc/pki/ca-trust/source/anchors:
total 0

/etc/pki/ca-trust/source/blacklist:
total 0

[machristie]

api.scigap.org.pem.txt

@ehb54 I've attached the certificate for api.scigap.org

[machristie]

Here's how I generated it, FYI:

openssl s_client -connect api.scigap.org:9930 | tee logfile
openssl x509 -inform PEM -in logfile -text > api.scigap.org.pem

[machristie]

I noticed though that our self-signed certificate is long since expired. So even with trusting it, I doubt it will successfully verify.

[ehb54]

I noticed though that our self-signed certificate is long since expired. So even with trusting it, I doubt it will successfully verify.

[fyi - the following tests were done on demeler9, not uslims4.aucsolutions]

Ah yes, I have been unable to get this to work with fsockopen() I put the pem in /usr/share/pki/ca-trust-source/anchors and ran update-ca-trust. This follows the steps described in /usr/share/pki/ca-trust-source/README:

This directory /usr/share/pki/ca-trust-source/ contains CA certificates and 
trust settings in the PEM file format. The trust settings found here will be
interpreted with a low priority - lower than the ones found in 
/etc/pki/ca-trust/source/ .

=============================================================================
QUICK HELP: To add a certificate in the simple PEM or DER file formats to the
            list of CAs trusted on the system:

            Copy it to the
                    /usr/share/pki/ca-trust-source/anchors/
            subdirectory, and run the
                    update-ca-trust
            command.

            If your certificate is in the extended BEGIN TRUSTED file format,
            then place it into the main source/ directory instead.
=============================================================================

Please refer to the update-ca-trust(8) manual page for additional information.

& verified there subsequently is a new entry in /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt & indeed there is:

# Apache Airavata
-----BEGIN TRUSTED CERTIFICATE-----
....

fsockopen() still failed, likely due to the expiration.

It turns out that fsockopen() does not allow turning off cert verify, but this alternative does:

<?php
echo "trying stream_socket_client to tls://api.scigap.org:9930\n";
$context = stream_context_create([
    'ssl' => [
        'verify_peer' => false,
        'verify_peer_name' => false
    ]
                                 ]);
$ret = stream_socket_client("tls://api.scigap.org:9930", $errno, $errstr, ini_get("default_socket_timeout"), STREAM_CLIENT_CONNECT, $context);
echo $ret ? "return true\n" : "return false\n";

And does return true. The return value description on the php manuals for fsockopen & stream_socket_client seem to match, so it should work as a direct replacement:

fsockopen() returns a file pointer which may be used together with the other file functions (such as fgets(), fgetss(), fwrite(), fclose(), and feof()). If the call fails, it will return false 

[stream_socket_client()]
On success a stream resource is returned which may be used together with the other file functions (such as fgets(), fgetss(), fwrite(), fclose(), and feof()), false on failure. 

Might be differences in exception handling, didn't check. So if we can not get certs to work, we could test this as an alternative. However, I suggest the better strategy would be to get certs working.

OTOH, there are only 2 places in the code where fsockopen seems to be used

./lib/Thrift/Transport/TSocket.php:223:      $this->handle_ = @pfsockopen($this->host_,
./lib/Thrift/Transport/TSocket.php:229:      $this->handle_ = @fsockopen($this->host_,

so it seems to be a simple change we could test.

[smarru]

Thank you Emre for these extra tests, yaa we need to stick to fsock, I will see if we can update our certs

[smarru]

@ehb54 I think we will try to update the server certs with letsencrypt following these instructions, that should mitigate any need of extra steps, we will update you what we find - https://ordina-jworks.github.io/security/2019/08/14/Using-Lets-Encrypt-Certificates-In-Java.html

[smarru]

@ehb54 following instructions at https://blog.keychest.net/lets-encrypt-certificate-into-java-jks I was able to use a letsencrypt certificate and fsocek now verifies correctly.

[ehb54]

@smarru Yes, I was able to connect from both demeler9 & uslims4 :) Thanks!

[ehb54]

@smarru will you setup an automated renewal of the certs?

[smarru]

@ehb54 thats my next worry on how to renew, we brainstormed some thoughts but did not converge. Letsencrypt renewal is trivial and so is java store creation. The challenge is to come up with a way so we don't have to restrat Airavata API server. So we have 90 days to put a automated renewal in place

[ehb54]

Certs seem expired again :( trying fsockopen to tls://api.scigap.org:9930 PHP Warning: fsockopen(): SSL operation failed with code 1. OpenSSL Error messages: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed in /tmp/test.php on line 4 PHP Warning: fsockopen(): Failed to enable crypto in /tmp/test.php on line 4 PHP Warning: fsockopen(): unable to connect to tls://api.scigap.org:9930 (Unknown error) in /tmp/test.php on line 4 return false

[smarru]

Emre, can you please try now