open-eid / digidoc4j

DigiDoc for Java. Javadoc:
http://open-eid.github.io/digidoc4j
GNU Lesser General Public License v2.1
73 stars 39 forks source link

TslDownloadException during DataToSign.finalize() with version 5.1.0 in test configuration #121

Closed mrts closed 1 year ago

mrts commented 1 year ago

When upgrading DigiDoc4j from version 4.3.0 to 5.1.0, tests started to fail during DataToSign.finalize() with the following exception:

org.opentest4j.AssertionFailedError: Unexpected exception thrown: org.digidoc4j.exceptions.TslDownloadException: Failed to download LoTL: https://open-eid.github.io/test-TL/tl-mp-test-EE.xml
    at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:152)
    at org.junit.jupiter.api.AssertDoesNotThrow.createAssertionFailedError(AssertDoesNotThrow.java:84)
    at org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:53)
    at org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:36)
    at org.junit.jupiter.api.Assertions.assertDoesNotThrow(Assertions.java:3164)
    at org.example.DigiDocSignerTest.testSignContainer(DigiDocSignerTest.java:14)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
Caused by: Failed to download LoTL: https://open-eid.github.io/test-TL/tl-mp-test-EE.xml
    at org.digidoc4j.impl.asic.tsl.DefaultTSLRefreshCallback.lambda$validateDownloadState$2(DefaultTSLRefreshCallback.java:206)
    at java.base/java.util.Optional.map(Optional.java:265)
    at org.digidoc4j.impl.asic.tsl.DefaultTSLRefreshCallback.validateDownloadState(DefaultTSLRefreshCallback.java:206)
    at org.digidoc4j.impl.asic.tsl.DefaultTSLRefreshCallback.validateState(DefaultTSLRefreshCallback.java:176)
    at org.digidoc4j.impl.asic.tsl.DefaultTSLRefreshCallback.ensureLOTLState(DefaultTSLRefreshCallback.java:98)
    at org.digidoc4j.impl.asic.tsl.DefaultTSLRefreshCallback.ensureTSLState(DefaultTSLRefreshCallback.java:91)
    at org.digidoc4j.impl.asic.tsl.LazyTslCertificateSource.refreshTsl(LazyTslCertificateSource.java:157)
    at org.digidoc4j.impl.asic.tsl.LazyTslCertificateSource.initTsl(LazyTslCertificateSource.java:147)
    at org.digidoc4j.impl.asic.tsl.LazyTslCertificateSource.refreshIfCacheExpired(LazyTslCertificateSource.java:133)
    at org.digidoc4j.impl.asic.tsl.LazyTslCertificateSource.getCertificateSource(LazyTslCertificateSource.java:139)
    at org.digidoc4j.impl.asic.tsl.LazyTslCertificateSource.getByPublicKey(LazyTslCertificateSource.java:103)
    at eu.europa.esig.dss.spi.x509.ListCertificateSource.getByPublicKey(ListCertificateSource.java:238)
    at eu.europa.esig.dss.validation.SignatureValidationContext.addCertificateSource(SignatureValidationContext.java:257)
    at eu.europa.esig.dss.validation.SignatureValidationContext.addDocumentCertificateSource(SignatureValidationContext.java:235)
    at eu.europa.esig.dss.validation.SignatureValidationContext.addSignatureForVerification(SignatureValidationContext.java:204)
    at eu.europa.esig.dss.validation.SignedDocumentValidator.prepareSignatureForVerification(SignedDocumentValidator.java:628)
    at eu.europa.esig.dss.validation.SignedDocumentValidator.prepareSignatureValidationContext(SignedDocumentValidator.java:610)
    at eu.europa.esig.dss.validation.SignedDocumentValidator.prepareValidationContext(SignedDocumentValidator.java:495)
    at eu.europa.esig.dss.validation.SignedDocumentValidator.getValidationData(SignedDocumentValidator.java:522)
    at eu.europa.esig.dss.validation.SignedDocumentValidator.getValidationData(SignedDocumentValidator.java:512)
    at eu.europa.esig.dss.xades.signature.XAdESLevelBaselineLT.extendSignatures(XAdESLevelBaselineLT.java:94)
    at eu.europa.esig.dss.xades.signature.XAdESLevelBaselineT.extendSignatures(XAdESLevelBaselineT.java:148)
    at eu.europa.esig.dss.xades.signature.XAdESLevelBaselineT.extendSignatures(XAdESLevelBaselineT.java:83)
    at eu.europa.esig.dss.xades.signature.XAdESService.signDocument(XAdESService.java:146)
    at eu.europa.esig.dss.xades.signature.XAdESService.signDocument(XAdESService.java:167)
    at org.digidoc4j.impl.asic.xades.XadesSigningDssFacade.signDocument(XadesSigningDssFacade.java:123)
    at org.digidoc4j.impl.asic.AsicSignatureFinalizer.finalizeSignature(AsicSignatureFinalizer.java:85)
    at org.digidoc4j.DataToSign.finalize(DataToSign.java:93)
    at org.example.DigiDocSigner.signContainer(DigiDocSigner.java:72)
    at org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:49)
    ... 6 more
    Suppressed: (Re)parsing needed for LoTL: https://open-eid.github.io/test-TL/tl-mp-test-EE.xml
        at org.digidoc4j.impl.asic.tsl.DefaultTSLRefreshCallback.validateParsingState(DefaultTSLRefreshCallback.java:229)
        at org.digidoc4j.impl.asic.tsl.DefaultTSLRefreshCallback.validateState(DefaultTSLRefreshCallback.java:177)
        ... 32 more
    Suppressed: (Re)validation needed for LoTL: https://open-eid.github.io/test-TL/tl-mp-test-EE.xml
        at org.digidoc4j.impl.asic.tsl.DefaultTSLRefreshCallback.validateValidationState(DefaultTSLRefreshCallback.java:253)
        at org.digidoc4j.impl.asic.tsl.DefaultTSLRefreshCallback.validateState(DefaultTSLRefreshCallback.java:178)
        ... 32 more
Caused by: Unable to process GET call for url [https://open-eid.github.io/test-TL/tl-mp-test-EE.xml]. Reason : [PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed]
    at java.base/java.util.Optional.map(Optional.java:265)
    at org.digidoc4j.impl.asic.tsl.DefaultTSLRefreshCallback.createCauseIfExceptionMessageExists(DefaultTSLRefreshCallback.java:270)
    at org.digidoc4j.impl.asic.tsl.DefaultTSLRefreshCallback.validateDownloadState(DefaultTSLRefreshCallback.java:205)
    ... 33 more

The DigiDoc4j configuration mode in tests is Configuration.of(Configuration.Mode.TEST).

There were no test errors with version 4.3.0.

Production configuration (Configuration.of(Configuration.Mode.PROD)) works without errors.

The Caused by: Unable to process GET call for url [https://open-eid.github.io/test-TL/tl-mp-test-EE.xml]. Reason : [PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed] error seems to indicate that there is an issue with the open-eid.github.io TLS certificate validation process which may be caused by the TLS handling changes in v5.1.0.

Here's a minimal self-contained example to reproduce the problem: https://gitlab.com/mrts/digidoc4j-test-tsl-download-failure

And here's the failing test: https://gitlab.com/mrts/digidoc4j-test-tsl-download-failure/-/jobs/4944833443

What do you think, can the above test be added to the DigiDoc4j test suite to catch such issues in the future?

rsarendus commented 1 year ago

This is a curious case, but it seems that mocking the System with jmockit somehow interferes with loading TLS truststores and/or validating TLS certificate chains. When removing the following lines from Dates::setMockedDate in your self-contained example, then the TSL loads successfully:

new MockUp<System>() {
    @Mock
    public long currentTimeMillis() {
        return mockedDateTime.toInstant().toEpochMilli();
    }
};

Although in 4.X.X versions, DigiDoc4j PROD mode used custom TLS truststore (which was removed in 5.0.0) for loading TSL over secure connections, DigiDoc4j TEST mode has always used the default Java truststore which contains the root certificate of the GitHub TLS certificate (where the test LOTL is hosted). I also tried to configure a custom TLS truststore in DigiDoc4j, but that functionality seems to be affected by jmockit as well. It didn't work with your initial example either.

Another change that I could think of, was the upgrade from org.apache.httpcomponents:httpclient to org.apache.httpcomponents.client5:httpclient5 since version 5.1.0. But that doesn't seem to be the case either, because I couldn't get your example to load the TSL with DigiDoc4j 5.0.0 either.

Unfortunately I couldn't think of any other changes that could have caused this. It could be anything that changed either in DigiDoc4j itself between versions 4.X.X and 5.0.0, in DSS between versions 5.8 and 5.9, or in any other dependency.

mrts commented 1 year ago

Good catch, thank you! Makes perfect sense in hindsight: mocking System.currentTimeMillis() to return April 14, 2020 affected the code that checks the validity of the open-eid.github.io TLS certificate, as the certificate is not valid before February 21, 2023.

Mocking BLevelParameters.getSigningDate() instead of System.currentTimeMillis() fixed the TslDownloadException, see the relevant commit here. Using System.currentTimeMillis() for mocking the signing date was way too intrusive for the specific issue anyway.

While it is curious that the test was successful as recently as on August 15 with DigiDoc 4.3.0, I believe we can put this behind us now.