search

dCache / dcache

dCache - a system for storing and retrieving huge amounts of data, distributed among a large number of heterogenous server nodes, under a single virtual filesystem tree with a variety of standard access methods
https://dcache.org
282 stars 135 forks source link

NullPointerException with Dutchgrid certificates since CA update 1.106 #5427

Open onnozweers opened 4 years ago

onnozweers commented 4 years ago

Dear dCache devs,

Since yesterday, we're seeing NullPointerExceptions, even after restarting gPlazma, but (as far as we can tell) only for Dutchgrid user certificates (we tested three of those). Example:

+ uberftp -mkdir gsiftp://gridftp.grid.sara.nl:2811/pnfs/grid.sara.nl/data/projects.nl/tropomi/s5p//output_2/ch4/14
500 Operation failed due to internal error: java.lang.NullPointerException

The normal Digicert/Terena certificates work as expected.

This may be related to https://github.com/dCache/dcache/issues/5426, but I'm not quite sure, so I'm creating a new issue.

13 May 2020 16:39:16 (GFTP-shark1-AAWliItj3BA) [door:GFTP-shark1-AAWliItj3BA@ftp-shark1Domain] FTP command 'ADAT ...' got exception
com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2052)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3943)
        at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4865)
        at org.dcache.util.CachingCertificateValidator.validate(CachingCertificateValidator.java:83)
        at eu.emi.security.authn.x509.helpers.ssl.SSLTrustManager.checkIfTrusted(SSLTrustManager.java:66)
        at eu.emi.security.authn.x509.helpers.ssl.SSLTrustManager.checkClientTrusted(SSLTrustManager.java:51)
        at java.base/sun.security.ssl.AbstractTrustManagerWrapper.checkClientTrusted(SSLContextImpl.java:1517)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkClientCerts(CertificateMessage.java:673)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:402)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:366)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1048)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:995)
        at org.dcache.dss.SslEngineDssContext.handshake(SslEngineDssContext.java:102)
        at org.dcache.dss.SslEngineDssContext.accept(SslEngineDssContext.java:221)
        at org.dcache.ftp.door.GssFtpDoorV1.ftp_adat(GssFtpDoorV1.java:175)
        at jdk.internal.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.dcache.ftp.door.AbstractFtpDoorV1$CommandRequest.runCommand(AbstractFtpDoorV1.java:612)
        at org.dcache.ftp.door.AbstractFtpDoorV1$CommandRequest.run(AbstractFtpDoorV1.java:597)
        at org.dcache.ftp.door.AbstractFtpDoorV1.ftpcommand(AbstractFtpDoorV1.java:1723)
        at org.dcache.ftp.door.AbstractFtpDoorV1.execute(AbstractFtpDoorV1.java:1794)
        at diskCacheV111.doors.NettyLineBasedDoor$Command.run(NettyLineBasedDoor.java:435)
        at org.dcache.util.BoundedExecutor$Worker.run(BoundedExecutor.java:251)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.NullPointerException: null

We have a client debug log with the GridFTP ADAT info, but I'm not posting that here because I don't know if that contains sensitive data. Let me know if you need that.

Cheers, Onno

onnozweers commented 4 years ago

It appears a restart of a GridFTP door fixes this. We will restart all doors.