RHEL9 clients and dCache on java-17 compatibility

[kofemann]

Dear maintainers,

during dCache testing with java-17 and almalinux9 (probably all REHL9 clones) clients, we discovered that globus-url-copy fails to establish GSS context. The same command works without issues on CentOS7. In both cases, we use globus-url-copy 10.13. I would assume, that difference comes from openssl version (openssl-1.0.2k-26.el7_9.x86_64 vs. openssl-3.0.7-25.el9_3.x86_64) or globus+openssl integration.

Almalinux 9

# GLOBUS_GSSAPI_DEBUG_LEVEL=2 globus-url-copy -list gsiftp://store-door-svc.dcache-build-105787.svc.cluster.local:2811/data/
Disabling SSLv2 and SSLv3.
acquire_cred: MICV2 MECH OID
gsiftp://store-door-svc.dcache-build-105787.svc.cluster.local:2811/data/s2/
Creating context w/ GSS_C_NO_CREDENTIAL.
init_sec_context: no mech_type requested; using MICV2 MECH OID
Disabling SSLv2 and SSLv3.
acquire_cred: MICV2 MECH OID
Ciphers available:
init_sec_context:major_status:00000001:gss_state:0 req_flags=00001013:ret_flags=00000000
SSL handshake finished
Using TLSv1.3.
cred_usage=1
Cipher being used:
TLS_AES_256_GCM_SHA384         TLSv1.3 Kx=any      Au=any   Enc=AESGCM(256)            Mac=AEAD
X509 subject after proxy : /C=DE/ST=Hamburg/O=dCache.ORG/CN=store-door-svc.dcache-build-105787.svc.cluster.local
Comparing names:
Name 1 is of type GLOBUS_GSS_C_NT_X509:
Name 2 is of type GLOBUS_GSS_C_NT_HOST_IP:
Compared 1 
init_sec_context:major_status:00000001:gss_state:1 req_flags=00001013:ret_flags=000000ff
init_sec_context:major_status:00000001:gss_state:2 req_flags=00001013:ret_flags=000000ff
delete_sec_context: output_token->length=24
delete_sec_context: done

error: globus_ftp_control: gss_init_sec_context failed
GSS failure: 
GSS Major Status: General failure
GSS Minor Status Error Chain:
globus_gsi_gssapi: Error with GSI proxy
OpenSSL Error: crypto/asn1/tasn_dec.c:654: in library: asn1 encoding routines, function (null): nested asn1 error
OpenSSL Error: crypto/asn1/tasn_dec.c:349: in library: asn1 encoding routines, function (null): nested asn1 error Type=X509_EXTENSION
OpenSSL Error: crypto/asn1/tasn_dec.c:1188: in library: asn1 encoding routines, function (null): wrong tag

CentOS 7.9

# GLOBUS_GSSAPI_DEBUG_LEVEL=2 globus-url-copy -list gsiftp://store-door-svc.dcache-build-105787.svc.cluster.local:2811/data/   
Disabling SSLv2 and SSLv3.
acquire_cred: OLD MECH OID
gsiftp://store-door-svc.dcache-build-105787.svc.cluster.local:2811/data/
Creating context w/ GSS_C_NO_CREDENTIAL.
init_sec_context: no mech_type requested; using OLD MECH OID
Disabling SSLv2 and SSLv3.
acquire_cred: OLD MECH OID
Ciphers available:
init_sec_context:major_status:00000001:gss_state:0 req_flags=00001013:ret_flags=00000000
init_sec_context:major_status:00000001:gss_state:0 req_flags=00001013:ret_flags=00000000
SSL handshake finished
Using TLSv1.2.
cred_usage=1
Cipher being used:
ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(256) Mac=AEAD
X509 subject after proxy : /C=DE/ST=Hamburg/O=dCache.ORG/CN=store-door-svc.dcache-build-105787.svc.cluster.local
Comparing names:
Name 1 is of type GLOBUS_GSS_C_NT_X509:
Name 2 is of type GLOBUS_GSS_C_NT_HOST_IP:
Compared 1 
init_sec_context:major_status:00000001:gss_state:2 req_flags=00001013:ret_flags=000000ff
init_sec_context:major_status:00000000:gss_state:4 req_flags=00001013:ret_flags=000000ff
gss_wrap conf_req_flag=0 qop_req=0
gss_wrap conf_req_flag=0 qop_req=0
gss_wrap conf_req_flag=0 qop_req=0
gss_wrap conf_req_flag=0 qop_req=0
gss_wrap conf_req_flag=0 qop_req=0
gss_wrap conf_req_flag=0 qop_req=0
gss_wrap conf_req_flag=0 qop_req=0
gss_wrap conf_req_flag=0 qop_req=0
    g2/
    tmp/
    tree/
    testfile 

gss_wrap conf_req_flag=0 qop_req=0

[maarten-litmaath]

Hi, just to be sure, depending on the CAs you need to support, you may need to enable SHA-1 on your EL9 host, if it will have to support CentOS 7 peers:

update-crypto-policies --set DEFAULT:SHA1

Then reboot the host to ensure the change is picked up by everything that needs it.

[kofemann]

Thanks for the prompt reply!

Indeed, by changing the crypto-policies g-u-c works as expected. We obviously can't force that on all clients or expect CAs to use different hash functions. So, I guess we have to change something on dCache side (or java security policy) to keep clients happy...

[maarten-litmaath]

Hi again, for the record, the matter has been documented at least here:

https://twiki.cern.ch/twiki/bin/view/LCG/EL9vsSHA1CAs

Though CentOS 7 will have its EOL on June 30, there may still be jobs running in containers based on that OS for years to come...

You can do something about it on the server side indeed, enabling SHA-1 just for the dCache services, not the whole host.

[kofemann]

I am still confused. Our test server have certs with sha256:

# openssl x509 -in dCacheORG.pem -noout -text | grep Signature
        Signature Algorithm: sha256WithRSAEncryption
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:

# openssl x509 -in dCacheORG-CA.pem -noout -text | grep Signature
        Signature Algorithm: sha256WithRSAEncryption
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:

The proxy uses sha256:

# arcproxy -I   
Subject: /C=DE/ST=Hamburg/O=dCache.ORG/CN=Kermit the frog/CN=898777003
Issuer: /C=DE/ST=Hamburg/O=dCache.ORG/CN=Kermit the frog
Identity: /C=DE/ST=Hamburg/O=dCache.ORG/CN=Kermit the frog
Time left for proxy: 6 hours 28 minutes 1 second
Proxy path: /tmp/x509up_u0
Proxy type: X.509 Proxy Certificate Profile RFC compliant impersonation proxy - RFC inheritAll proxy
Proxy key length: 2048
Proxy signature: sha256
====== AC extension information for VO desy ======
VO        : desy
subject   : /C=DE/ST=Hamburg/O=dCache.ORG/CN=Kermit the frog
issuer    : /DC=org/DC=terena/DC=tcs/C=DE/ST=Hamburg/O=Deutsches Elektronen-Synchrotron DESY/CN=grid-voms.desy.de
uri       : grid-voms.desy.de:15104
attribute : /desy/Role=NULL/Capability=NULL
attribute : /desy/workshop/Role=NULL/Capability=NULL
Time left for AC: 6 hours 28 minutes 1 second

The TLS hanshake uses TLS1.3 with sha384:

Using TLSv1.3.
cred_usage=1
Cipher being used:
TLS_AES_256_GCM_SHA384         TLSv1.3 Kx=any      Au=any   Enc=AESGCM(256)            Mac=AEAD
X509 subject after proxy : /C=DE/ST=Hamburg/O=dCache.ORG/CN=store-door-svc.dcache-build-105787.svc.cluster.local

So, this is not diretly related to GSI, but something different, I guess... 🤷🏼‍♂️

[maarten-litmaath]

Interesting!

Might a side effect of enabling SHA-1 be that something else that is needed also gets enabled?

[msalle]

Could it be the signature on the self-signed root CA or one of the intermediates (latter one is less likely) ? That would still give this error I'd say and more likely than any of the other tools.

[kofemann]

@msalle the dcache test CA signed with sha256WithRSAEncryption. Is there a way to debug at which point g-u-c tries to use sha1?

[msalle]

Are both server and client certs signed by the same test CA? Is it a rootCA and/or are there intermediates? Also, are you using frontend/backend nodes for dCache or is it all a single node? In any case I would probably run an strace and look for (succesfully) opened files in /etc/grid-security/certificates. It probably might also help to add the -dbg flag which should give control channel debug info. But it's also possible it's actually not one of the CAs, although I'm not sure how.

[kofemann]

ok, looks like this is related to proxy delegation:

globus_i_gsi_gss_put_token exiting: major_status=0
globus_gsi_proxy_inquire_req entering
****** START X509_REQ ******
Certificate Request:
    Data:
        Version: 1 (0x0)
        Subject: C=DE, ST=Hamburg, O=dCache.ORG, CN=Kermit the frog, CN=2982459500, CN=1261921953
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:ca:90:83:56:6a:11:76:06:32:90:49:5a:a2:25:
                    c0:bb:d4:2d:c3:69:e7:f0:91:81:90:16:28:d4:e7:
                    34:8e:b8:7f:71:6d:8e:62:27:56:a5:b9:2a:62:ba:
                    6e:1a:c8:23:a9:3b:f5:fa:44:ec:b3:c8:4a:f4:6b:
                    c6:d4:a7:12:10:20:92:6e:9b:b0:f7:be:b5:f7:6c:
                    8c:fe:1f:33:45:2d:ee:03:db:4a:37:56:89:6a:46:
                    00:41:14:c0:08:b8:67:e9:e5:90:0d:67:25:3b:48:
                    3a:27:d2:cc:fa:90:f0:53:c2:72:fd:c1:a9:8e:b8:
                    96:dd:12:3d:68:31:2c:1a:7f:41:a9:41:26:68:01:
                    14:b9:8f:50:e8:bd:e6:18:77:88:6c:08:41:8d:50:
                    c0:40:57:39:0b:d5:5e:df:17:78:d5:82:8f:29:be:
                    d1:09:d9:60:39:47:3a:70:00:1e:39:7c:9e:d6:32:
                    d6:88:7c:46:10:1a:ad:95:f6:22:25:04:8a:6d:80:
                    d9:96:94:59:17:08:25:4c:5b:5a:1e:27:3d:87:a8:
                    b3:3f:71:64:98:8a:22:6d:18:77:b7:05:c2:20:4f:
                    48:e4:fb:fb:88:9f:af:57:b2:b7:cd:d0:e2:14:64:
                    83:72:2c:db:d7:b2:e5:72:89:81:c9:89:7b:60:a2:
                    8e:19
                Exponent: 65537 (0x10001)
        Attributes:
    Signature Algorithm: sha1WithRSA
    Signature Value:
        24:d1:c7:c2:e5:f4:22:d5:46:3c:f0:db:80:48:25:4f:6e:6d:
        f2:06:6d:ad:15:63:30:d1:d4:3b:b9:1d:2b:3d:d0:19:9a:68:
        2f:bb:d0:3d:e2:41:01:a2:6f:59:28:4e:39:5b:30:6f:31:78:
        60:4a:32:fe:d4:71:ca:66:b3:b2:45:8c:f4:a7:3a:f2:0e:2c:
        6b:61:d6:09:ee:0d:b5:32:a7:1e:5c:1e:17:f7:d9:58:43:32:
        34:71:ca:4b:83:d6:0d:af:91:65:44:0a:53:6c:97:82:45:cd:
        93:90:f4:4b:ff:46:04:e1:a0:24:d2:34:e2:73:6a:23:41:be:
        3b:f6:32:8e:b0:0c:e3:a1:37:86:36:43:ba:d2:43:b2:7a:c2:
        b2:e6:0a:23:e6:13:55:7a:c0:09:d0:f6:bb:c7:14:fd:01:0c:
        f1:c4:f3:20:dc:e4:40:0e:a0:60:42:9d:d6:72:e1:c4:94:bd:
        d2:32:d7:f1:3a:d0:00:e3:88:e1:a1:cd:7c:b1:74:31:51:67:
        8a:b7:2c:a0:c4:59:13:e5:1a:98:5b:db:9b:36:e4:68:41:dc:
        ea:56:3c:5c:eb:2d:30:30:5e:8d:e3:b1:97:7a:b8:46:5c:17:
        ff:47:6c:47:6e:93:31:19:34:ce:19:b1:be:c6:c4:d5:a4:43:
        78:a1:6f:9d
******  END X509_REQ  ******

[kofemann]

is it related to https://github.com/gridcf/gct/issues/153 ? And how to control which signature is used by delegation?

[kofemann]

Eureka!

https://github.com/eu-emi/canl-java/blob/master/src/main/java/eu/emi/security/authn/x509/proxy/ProxyCSRGenerator.java#L147

canl-java have hard-coded sha1.

[kofemann]

magic all debug options by globus (just in case someone will hit similar debug issue):

for i in * /lib64/libglobus_*; do strings $i | grep DEBUG_LEVEL; done | sort -u | awk '{ printf("%s=99 \\\n", $1); }'
GLOBUS_CALLOUT_DEBUG_LEVEL=99 \
GLOBUS_FTP_CLIENT_DEBUG_LEVEL=99 \
GLOBUS_FTP_CONTROL_DEBUG_LEVEL=99 \
GLOBUS_GSI_CALLBACK_DEBUG_LEVEL=99 \
GLOBUS_GSI_CERT_UTILS_DEBUG_LEVEL=99 \
GLOBUS_GSI_CRED_DEBUG_LEVEL=99 \
GLOBUS_GSI_GSS_ASSIST_DEBUG_LEVEL=99 \
GLOBUS_GSI_OPENSSL_ERROR_DEBUG_LEVEL=99 \
GLOBUS_GSI_PROXY_DEBUG_LEVEL=99 \
GLOBUS_GSI_SYSCONFIG_DEBUG_LEVEL=99 \
GLOBUS_GSSAPI_DEBUG_LEVEL=99 \

[msalle]

Eureka!

https://github.com/eu-emi/canl-java/blob/master/src/main/java/eu/emi/security/authn/x509/proxy/ProxyCSRGenerator.java#L147

canl-java have hard-coded sha1.

Nice find! I.e. in the end it's server side (dCache) and not a client side/GCT issue.

[kofemann]

Disabling delegation by default still makes sense.

[msalle]

you probably have a point there. Probably only for 3rd party copies right? I'll reopen and see what the others have to say. Personally I think we should focus on really breaking issues or potential security issues for the GCT. @maarten-litmaath @fscheiner what do you think?

[fscheiner]

Well, we never found a real conclusion for https://github.com/gridcf/gct/issues/153 back then, other then that we might not have the manpower to fix non-breaking issues.

But I think we wouldn't categorically say no to patches that don't break defaults in this regard, right @msalle, @maarten-litmaath, @ellert ?

Another thought: For gsissh delegation can be disabled AFAIR. So maybe there is a hidden switch for guc to do that, too? But I started to wonder if delegation is not already needed for split-process GridFTP server configurations. But still if it can be disabled with an additional switch it shouldn't hurt.

[kofemann]

Sure, there is always a cost/benefit factor that should be taken into account. From dCache point of view, it's less than 1% of transfers that use gridftp. This issue has popped up in our internal testing, where we use all kinds of clients to avoid all kinds of regressions.

Unfortunately, GSI will stay longer with us. Thus, not requesting delegation by default will speed up the handshake and utilize less CPU resources on both the client and server sides.

[msalle]

Well, we never found a real conclusion for #153 back then, other then that we might not have the manpower to fix non-breaking issues.

ah great. Means we should probably close this one again...

But I think we wouldn't categorically say no to patches that don't break defaults in this regard, right @msalle, @maarten-litmaath, @ellert ?

no, I think that's fine. But actively patching it ourselves is probably not what we should be doing.

Another thought: For gsissh delegation can be disabled AFAIR. So maybe there is a hidden switch for guc to do that, too? But I started to wonder if delegation is not already needed for split-process GridFTP server configurations. But still if it can be disabled with an additional switch it shouldn't hurt.

For split process gridftp isn't it all between client and a single server? Not between server and a second server?

[fscheiner]

For split process gridftp isn't it all between client and a single server? Not between server and a second server?

Only if IP and DTP(s) are on the same host. If DTPs are on separate hosts, they need to have the means to authenticate the client connection somehow, I believe.

[msalle]

yes but that goes back to the original client which already has a proxy?

[fscheiner]

True. I think I need to do some more research here. :-)

[msalle]

In any case I'm reclosing this one, suggest we continue in #153 We can discuss if there are other scenarios than 3rd party copies that need delegated creds.