search

lovele0107 / signatures-conformance-checker

7 stars 0 forks source link

XAdES : AllDataObjectsTimestamp fails with an XPath reference transform #1

Open bsanchezb opened 4 years ago

bsanchezb commented 4 years ago

Hello,

I have a signature with AllDataObjectsTimestamp, validation of which fails in ETSI Conformance Checker.

According to ETSI EN 319 132-1 (5.2.8.1 The AllDataObjectsTimeStamp qualifying property):

The input to the computation of the message imprint shall be the result of processing all the ds:Reference elements within the ds:SignedInfo except the one referencing the SignedProperties element, in their order of appearance, as follows: 1) process the retrieved ds:Reference element according to the reference-processing model of XMLDSIG [1] clause 4.4.3.2; 2) if the result is a XML node set, canonicalize it as specified in clause 4.5; and 3) concatenate the resulting octets to those resulting from previously processed ds:Reference elements in ds:SignedInfo.

After extracting a reference output and applying a canonicalization algorithm I expect to get the following result:

<doc xmlns="http://example.org" Id="dss1">
    <e1 xmlns:a="http://www.w3.org" attr="Hello" a:attr="World">
        <e2>
            <e3 xmlns="http://www.ietf.org"></e3>
        </e2>
    </e1>
</doc>

While, based on my investigations, the ETSI Checker computes the following message-imprint:

<ds:Object xmlns:ds="http://www.w3.org/2000/09/xmldsig#" Id="document"><doc xmlns="http://example.org" Id="dss1">
    <e1 xmlns:a="http://www.w3.org" attr="Hello" a:attr="World">
        <e2>
            <e3 xmlns="http://www.ietf.org"></e3>
        </e2>
    </e1>
</doc></ds:Object>

What seems to be wrong, because the same reference output is not used for a normal reference validation (ds:Object shall not be included based on the XPath 2 Filter value).

Could you please investigate this case?

Regards, Aleksandr.

signature.zip

jccruellas commented 4 years ago

Dear BSanchezB,

I would say that this is the same issue as the issue#2, and consequently, my reaction is similar to that issue. The problem, to me, is not on the XAdESCC, but on the fact that the ds:Reference that does not reference the SignedProperties, has an empty URI. According to the XML Signature 1.1 retrieving the object pointed by this URI results in the whole document that includes the URI attribute...which in this case also includes the AllDataObjectsTimeStamp itself. As I said, you should include in the first ds:Reference an URI="document". This would actually make that the retrieval process would only retrieve this document in validation.

Regards Juan Carlos.

bsanchezb commented 4 years ago

Dear Juan Carlos,

I'm sorry, but the signature in the archive does not contain empty URI references. Moreover, the ds:Reference that does not refer a SignedProperties has URI="#document" as you suggested. A snippet from the attached file to the first message:

<ds:Reference Id="Canonicalization-Ref-Test" Type="http://www.w3.org/2000/09/xmldsig#Object" URI="#document">
     <ds:Transforms>
          <ds:Transform Algorithm="http://www.w3.org/2002/06/xmldsig-filter2">
               <dsig-filter2:XPath xmlns:dsig-filter2="http://www.w3.org/2002/06/xmldsig-filter2" Filter="intersect">//*[@Id='dss1']</dsig-filter2:XPath>
          </ds:Transform>
     </ds:Transforms>
     <ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/>
     <ds:DigestValue>yu/FuEuF21xiJFus9ek5PQUp5dGZ9MGMbt6MWiWIzZg=</ds:DigestValue>
</ds:Reference>
<ds:Reference Type="http://uri.etsi.org/01903#SignedProperties" URI="#xades-id-327ba84ef942ccd690eac7ca9d3be95f">
     <ds:Transforms>
          <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
     </ds:Transforms>
     <ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/>
     <ds:DigestValue>ZhG4pqprmPLZjDbT63Yb46LoXakZ4fZJKkmS6/s/ChE=</ds:DigestValue>
</ds:Reference>

I can re-upload the document if you have a problem with it.

Best regards, Aleksandr.

jccruellas commented 4 years ago

Dear BSanchez, This is extrange.... I copy below the contents of the firstSignature.xml file that you uploaded...and please note that the first ds:Reference, has an empty URI attribute URI=""......this is ordering to the relying party to retrieve the whole document as part of the process of generating the input to the message imprint for AllDataObjectsTimeStamp.... it seems that we are managing two different xml documents here....

And something similar occurs with your secondSignature.xml file.....the first ds:Reference has an empty URI attribute....

?xml version="1.0" encoding="UTF-8"?>

not(ancestor-or-self::ds:Signature) KfEefDNjJivsIJuqaoxXVitltss0cWMhjALexGuFUfM= 8N8Jtz5iTOLjNBBv/gYvQG1DcjsErrnGyjcioamlFxQ= kj14bjmWkv24YHkPMYRx2RpbZsJDqhHVMo+P/AjLsLlmnUvopNYY9hLwD4hl03z7CKQvYmIqBMPRch/0ema+sjvIes8eGGei3xa0ij7j/395vRBjWnjQHVXPdZCLvZnM3JAVFpuXa+VdlpzKGQTkwUn/EREv8rxFzNvgEXM30OSp1dUUzvZlClGDLjrbnwJ6P7lj9XmkufmoU6ilvQAUDP6QUV+NMcGbdCgFPW2ouCPPJhEjbJtYhHD+Qr+/mOIB0V5gVhurHs6vXjQ4pUaykKO6JGNn1p34I7YbVuW0+yuiJAG4Pl+hsRDOFKzYJAjRffskTJKbLRc2YzT7lpiAog== MIID1DCCArygAwIBAgIBCjANBgkqhkiG9w0BAQsFADBNMRAwDgYDVQQDDAdnb29kLWNhMRkwFwYDVQQKDBBOb3dpbmEgU29sdXRpb25zMREwDwYDVQQLDAhQS0ktVEVTVDELMAkGA1UEBhMCTFUwHhcNMTkwOTAzMDYzMjUwWhcNMjEwNzAzMDYzMjUwWjBPMRIwEAYDVQQDDAlnb29kLXVzZXIxGTAXBgNVBAoMEE5vd2luYSBTb2x1dGlvbnMxETAPBgNVBAsMCFBLSS1URVNUMQswCQYDVQQGEwJMVTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKsgmdAGbGYrruKaZACWLXEkJ4nN4agqvjVNkx8lbZMwBzwKnwWermwhUf1FBv7Z/Sazin8yXMAxawEd1P5Mfq4SKisbNQsc74EywgdPYohUhukko/REefQ6X+yFIVAADkD/5gyxumNpEXNRIzY3zdm7ap6uoWqFExY48S4yqRO/njt4Elcivt5umWUL4l8cDX1S0pXm51siqt2wWC08U1+7kV7mFiYB8EoQo86xmzcGmLLvyB9aY28+LliSS+FNj1L0EHohAwfEixAT+3StvRgGYf3KLic9ib4NWeV/0upiWK3FGS1miX2TOW/tfCyJIjVaX3sNT4GHXE7mW/9i7RECAwEAAaOBvDCBuTAOBgNVHQ8BAf8EBAMCBkAwgYcGCCsGAQUFBwEBBHsweTA5BggrBgEFBQcwAYYtaHR0cDovL2Rzcy5ub3dpbmEubHUvcGtpLWZhY3Rvcnkvb2NzcC9nb29kLWNhMDwGCCsGAQUFBzAChjBodHRwOi8vZHNzLm5vd2luYS5sdS9wa2ktZmFjdG9yeS9jcnQvZ29vZC1jYS5jcnQwHQYDVR0OBBYEFLhUuT1MCy99LE9Q4jf9bykSiCaHMA0GCSqGSIb3DQEBCwUAA4IBAQA3nuxMH+9amzeaVNJXB9tU421qhFxBwbFY5Zlb7/Em8ONqNFAnrT/B8nS1n5YSbnoYzTpMT4I2MjnPo8CYw5Ffpfz5Q732WgP4v8zs4yfrYBpj4szdmcFhrQWwZKaMZCiXTRXH/RTSBmtmrA7AhAsfBQV7n58mJa8BcpHQvUnmhxZ4bd5LGjIgbJ+UzbYWM1ACplAZoOl04i5mDG7rkvt17eO0jyjrb146hcPM8NUNNlsXgqYmgL5K6AAqZrSNjYsmuxfZhiH+vUTx/nAxiontmGnZSdYmNrY5E9WmOC47ioQlir6QXsbblFOSEvj7QvqlkFThecqooMRiKw8pxvpU MIID6jCCAtKgAwIBAgIBBDANBgkqhkiG9w0BAQsFADBNMRAwDgYDVQQDDAdyb290LWNhMRkwFwYDVQQKDBBOb3dpbmEgU29sdXRpb25zMREwDwYDVQQLDAhQS0ktVEVTVDELMAkGA1UEBhMCTFUwHhcNMTkwOTAzMDYzMjQ4WhcNMjEwNzAzMDYzMjQ4WjBNMRAwDgYDVQQDDAdnb29kLWNhMRkwFwYDVQQKDBBOb3dpbmEgU29sdXRpb25zMREwDwYDVQQLDAhQS0ktVEVTVDELMAkGA1UEBhMCTFUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC/4d44S5WEPN1Qx6SQ69yOjERUIalj8clurj9Rq4nhAxoNmepEk77bUrYT2W5ZkQEsTKPM5/tliMkQcPra4dxoZ9+6+r9iz5eeKaoIgeswzcC6i8xlwdExkJp2Anf+gTx37zQnSvwCxn4KGVM6KD66W/0X9igAPMrGpTNN4oDsRMXOmu4lNZlx6dHLL5c2QzecJD7JlpQlJBIPgTS0F/i9oGioCXXuYhcNbVe4ttMaM1qBuMwoLBvNm1zQjxDrpgdd5zArZonjpB89uPpOcMKGzVBgjNZ8HB+sEVlRbFQ8ovvtmORTA86IsfbvlNlAEs6CG5PxovBo2T90ZuER8bLnAgMBAAGjgdQwgdEwDgYDVR0PAQH/BAQDAgEGMEEGA1UdHwQ6MDgwNqA0oDKGMGh0dHA6Ly9kc3Mubm93aW5hLmx1L3BraS1mYWN0b3J5L2NybC9yb290LWNhLmNybDBMBggrBgEFBQcBAQRAMD4wPAYIKwYBBQUHMAKGMGh0dHA6Ly9kc3Mubm93aW5hLmx1L3BraS1mYWN0b3J5L2NydC9yb290LWNhLmNydDAdBgNVHQ4EFgQUWfZGEPqCJUzYFenTHkGdETxGquUwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAvBrcf3yVyoXeOjupoZlJH8tA0YyId79pya4P2Es0ttS6d05KchrqfOzv6l8z5xVxHqHJemhBe5lntcb0fMS5krRvnafEhFF+nlLLDVSjsTB1eBuey/OnZ2b8an7MFIhfZf4oDiI3NtTs8ZMW3NOyhfXG7xqohJltN1pzjv3NFI5Wf4b0m8lcP7HLEuVCxNUGwNdFbc+6ybMSiiRsd3zA6tRQYn2l2AVZ/Ok9HJ9WrIMKgoq1MrWDvWHFHnrBZUr+9MfI+Xmg3Xk/MUmgrHDcmfhogfnTBdaCRoQz1cVlh19NPgAWnTRALT6yzrbpnca/x6IHAvFNLuPRz+WjOPxDJA== 2020-08-14T14:01:18Z nBU3gXP4VuxzkeKT5WeApx7Mux1qcgl7NiBFjhq508JA6CTO1Wf3vIgNs4xV/w+LCVjdvto8gW1z8ZB4GGXTQQ== MFYwUaRPME0xEDAOBgNVBAMMB2dvb2QtY2ExGTAXBgNVBAoMEE5vd2luYSBTb2x1dGlvbnMxETAPBgNVBAsMCFBLSS1URVNUMQswCQYDVQQGEwJMVQIBCg== text/xml MIIKSAYJKoZIhvcNAQcCoIIKOTCCCjUCAQMxDzANBglghkgBZQMEAgEFADBxBgsqhkiG9w0BCRABBKBiBGAwXgIBAQYDKgMEMDEwDQYJYIZIAWUDBAIBBQAEIDfInfQbzGuGiE7TIxFNtMd+oc2TMRQ+OOMwk85Z29GAAhAcZ5klE8eaeqTr+hIq4wVgGA8yMDIwMDgwMTEyMDAwMFqgggdSMIIDVzCCAj+gAwIBAgIBATANBgkqhkiG9w0BAQ0FADBNMRAwDgYDVQQDDAdyb290LWNhMRkwFwYDVQQKDBBOb3dpbmEgU29sdXRpb25zMREwDwYDVQQLDAhQS0ktVEVTVDELMAkGA1UEBhMCTFUwHhcNMTkwODAzMDYzMjQ3WhcNMjEwODAzMDYzMjQ3WjBNMRAwDgYDVQQDDAdyb290LWNhMRkwFwYDVQQKDBBOb3dpbmEgU29sdXRpb25zMREwDwYDVQQLDAhQS0ktVEVTVDELMAkGA1UEBhMCTFUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDZ7JX2q8QypdrznoMbaEUw6ihqNcX/Nl+sBOdydkKNH3Nt869KujIO9xpRBKpKICwQvw1lHdF0/I/OUXXki845fPp8WhgZzU5rEpySGLBMCAMe908vIVgoIHWlsXGs5FhHjpFrdHaW5+YdtcrlUb8llQVEAnX50LN1LB7T6mjg1UzEDd3TuHa1CcIn9OI0cLEeTL3ebH0MgDJr3IGJAwDicjYMIpLMBNax947WLEvMs8uzzr/+1m5ycwgnq+CceeGqvjWSfK+w7hu7wdqKtXwpQcmE2qRe7dj4r7szuHFagqyD2TtHAVKeUo6g0glxweVnyTtngk/SzpLDx3WnDkvlAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAdBgNVHQ4EFgQU9S5pd1uILQPiX0wzF2r7VAOtRf4wDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQ0FAAOCAQEAUGayYIni5o7a/O9YZl92ZNOFQ2aHKWZ0BVVGnOsrSuNVX1l7p32CwvvfJNcdtKcbgp7Nh++u+zTWyR7fT+VwEt2fFTZ8ed1Gq6c/IiPCqR4YQ3eaydqWpEcP1WhJ6rHUAxxlqQcFPm5cqMcIxz6aD1PFOTiLVPd6+9+Ei39Lorx0ww0xw1IQ74LGIvUC/0kmjMTooEiO/bpaFWXiwR8fejxlklR/6fhh2SIi04hciPty/Bs47ameno29DhBDvMxWvIpRF2TBNhezWyivbjPYFVFJbQkVwW44mLkRyI7PyCyw9ejN9R8+ve/l3HFCpd14gWWIsXUr1PXaC7iXd5uCpTCCA/MwggLboAMCAQICAgH0MA0GCSqGSIb3DQEBCwUAME0xEDAOBgNVBAMMB3Jvb3QtY2ExGTAXBgNVBAoMEE5vd2luYSBTb2x1dGlvbnMxETAPBgNVBAsMCFBLSS1URVNUMQswCQYDVQQGEwJMVTAeFw0xOTA5MDMwNjMyNTZaFw0yMTA3MDMwNjMyNTZaME4xETAPBgNVBAMMCGdvb2QtdHNhMRkwFwYDVQQKDBBOb3dpbmEgU29sdXRpb25zMREwDwYDVQQLDAhQS0ktVEVTVDELMAkGA1UEBhMCTFUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDgx9LFFa+JWe9KjI9eM/I5o06+AksIoBV7+L3ODjfKZunC8TAtRYX8v4V1LfM49ele0jpJkQe3m8kExE3fCo8F5P0hBI2f8lHDWkFVhZwmicfHQJifQ6uyVtE+SkiVx+s4lbcJ4TjTMnFG1aXMx5aCZLKfwtSBozCvzXfBanbQLhujFucvOH23bvA9ql0nF/HjuyDChqb1NM0oA/lHu8jbqXNPBiMO3N8buP6E1FBA6BfO5WeK5N6bNnuwK/UBUFA4Qhk7LBaiUb0OUBoebaeLTshUgHU5FtRfQ8tfKVcQCgH43tkiyxhVOvNV4BVM2NHnBtob74QXgi6d8dpqi327AgMBAAGjgdswgdgwDgYDVR0PAQH/BAQDAgeAMBYGA1UdJQEB/wQMMAoGCCsGAQUFBwMIMEEGA1UdHwQ6MDgwNqA0oDKGMGh0dHA6Ly9kc3Mubm93aW5hLmx1L3BraS1mYWN0b3J5L2NybC9yb290LWNhLmNybDBMBggrBgEFBQcBAQRAMD4wPAYIKwYBBQUHMAKGMGh0dHA6Ly9kc3Mubm93aW5hLmx1L3BraS1mYWN0b3J5L2NydC9yb290LWNhLmNydDAdBgNVHQ4EFgQU43ueZoAOPm+3ic5YZzj5JDlDN6MwDQYJKoZIhvcNAQELBQADggEBAA+86ZkyZ063ia1Wxogwr98EO1AqvjvE5vQApDGUK1txGKrSSrXHkTHRUsjn7EKf3FcuEZ2AtbZprvyRRXZ5Y21SxXc8s0QoK8vD75SSXwgKYk4qZvOo1W4IzkYs3h59pMNMYQq9FV/vFFDwS1/ms9X0lPileWMXEAM8Ke+b0TZW1+iIYrdCEH8jXLHm+oNKMLP3v6iAsaE8ouyUjkHl6PGAfDjoJVkje3l7Xct4cOu8253sQrGrS1QQOteqjMH/vB+vOn0WhFDpjx6pNxsaqjUaKHQZO66y6cI0rwltQqQqf2FY9r/wvbzhHFrm2Ewf9VS4HGwZU7Z+oLdewkETBukxggJUMIICUAIBATBTME0xEDAOBgNVBAMMB3Jvb3QtY2ExGTAXBgNVBAoMEE5vd2luYSBTb2x1dGlvbnMxETAPBgNVBAsMCFBLSS1URVNUMQswCQYDVQQGEwJMVQICAfQwDQYJYIZIAWUDBAIBBQCggdMwGgYJKoZIhvcNAQkDMQ0GCyqGSIb3DQEJEAEEMBwGCSqGSIb3DQEJBTEPFw0yMDA4MTQxNDAxMjBaMC0GCSqGSIb3DQEJNDEgMB4wDQYJYIZIAWUDBAIBBQChDQYJKoZIhvcNAQELBQAwLwYJKoZIhvcNAQkEMSIEIIhodz/ohZvUK6Jxpu741d0+LqZI1lw6qVL/21r59z+nMDcGCyqGSIb3DQEJEAIvMSgwJjAkMCIEIAgf4RRRMYbCYBv9SRxUtzWh2F0TuUTs771QrrWk7w29MA0GCSqGSIb3DQEBCwUABIIBADS/ZeVrtCBFV1TkwJGO7JKSGL6o2+EsWkGS9RQG/Zp13a0s3KyDR9M/3oOyKyQ5sPTp9dMRFYvFWzi2LH7D+xvc7jGzW0ZUBzDfsAIgHEorff235lbWTa91oTK0r2qZwLvf9jErnGVIiWfqQ5h7a3xkubvjEDu4Ey34KOMNgR9PiXgtez/Vd4pS6NleIwdaKLDEw9JI1IPVdKduJxyuRWkm90tW7EL/NWECjQqmJ8nxiGLlf8YilDt1dqJGGemCamFlKmyCwLfBdfhUzDIecLcSzsA0c2MhWg88yLM9+1kOBfOaTK//FnA6Jd74XGz248xJqXpTj8s94qK9McxMTkk=
bsanchezb commented 4 years ago

This is the signature from the issue https://github.com/lovele0107/signatures-conformance-checker/issues/2 .

In this issue there is only one signature inside an archive. I have re-uploaded one to the message (the same from the first post).

The file containing a signature from the archive is called "signature.xml".

signature.zip

jccruellas commented 4 years ago

Ah.... I see...sorry for the missunderstanding.....

Will check what is going on and let you know.

Juan Carlos.

jccruellas commented 4 years ago

Hi Alexander,

As already mentioned I am using Apache XML sec for dealing with all the stuff relating to retrieval, canonicalization and cryptographic computations related to XML Signatures.

I have been doing some additional work on the code. It seems that the method of XMLsec Apache in Reference class that I used for getting the output of its transforms (called getContentsAfterTransformation() ) does not return the proper node set as it includes the ds:Object....after some indagations I have replaced it by another method (getReferencedBytes()), which seems to return the right bytes. I copy them below:

As you can see, the root node is the doc element, as you suggest, but now if you compare with your output:

You may appreciate certain differences, namely the presence of two namespaces attributes in my result, which are not present in yours.

The actual truth is that still the XAdESCC complains: the message imprint computed and the message imprint in the time-stamp token are not the same.

Now, if your message imprint is computed on your doc element (without the two additional namespaces attributes) I would say that the XAdESCC is now working OK. Note that the XAdESCC cryptographically verifies your signature, meaning that it suceeds in getting the referenced bytes by this ds:Reference element and compute its digest....and the bytes obtained during the verification of the signature are precisely the ones that I have shown above....this means that for the generation of the digest value in teh ds:Reference element we both use the same input: the one that I have shown above with the two extra namespaces attributes.....and because the specification of AllDataObjectsTimeStamp, these are the bytes that contribute to the computation of the message imprint. Therefore, if you are computing the message imprint on the doc element that you mentioned, without the two additional namespace attributes, then this should be the reason for the discrepancy.

If this is not the case and you really take as input to the mssage imprint computation the same bytes as the ones got by the XAdESCC, then the only explanation that I can imagine is that we are using different digest algorithms....in my case is the algorithm identified in the time-stamp token, namely, the one whose identifier is 2.16.840.1.101.3.4.2.1 (SHA256).

Will wait your comments....although as I am on holidays, maybe I will be slow in reacting....

Juan Carlos.

jccruellas commented 4 years ago

Oops.... I do not see the XML pieces in my comment...do not know what happened. Here you have:

MY XML (the one obtained by the XAdESCC after processing the ds:Reference:

Your doc element in your previous comment lacks the two namespace attributes

jccruellas commented 4 years ago

I see that the tool does not get the XML ....guess that some markup issue is going there...therefore I have put the result in a text file, which I attach ReferencedBytes.txt

bsanchezb commented 4 years ago

The octets you receive after performing an XMLDSIG Reference Processing Model are correct. However, according to ETSI EN 319 132-1 it is not complete for a message-imprint computation. "5.2.8.1 The AllDataObjectsTimeStamp qualifying property" :

The input to the computation of the message imprint shall be the result of processing all the ds:Reference elements within the ds:SignedInfo except the one referencing the SignedProperties element, in their order of appearance, as follows: 1) process the retrieved ds:Reference element according to the reference-processing model of XMLDSIG [1] clause 4.4.3.2; 2) if the result is a XML node set, canonicalize it as specified in clause 4.5; and 3) concatenate the resulting octets to those resulting from previously processed ds:Reference elements in ds:SignedInfo.

In other words, the octets in the file you have attached is the result of the step 1).

From XMLDSIG:

4.4.3.2 The Reference Processing Model The data-type of the result of URI dereferencing or subsequent Transforms is either an octet stream or an XPath node-set.

In my reference, the last performing transform is XPath Filter 2.0:

<ds:Reference Id="Canonicalization-Ref-Test" Type="http://www.w3.org/2000/09/xmldsig#Object" URI="#document">
     <ds:Transforms>
          <ds:Transform Algorithm="http://www.w3.org/2002/06/xmldsig-filter2">
               <dsig-filter2:XPath xmlns:dsig-filter2="http://www.w3.org/2002/06/xmldsig-filter2" Filter="intersect">//*[@Id='dss1']</dsig-filter2:XPath>
          </ds:Transform>
     </ds:Transforms>
     <ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/>
     <ds:DigestValue>yu/FuEuF21xiJFus9ek5PQUp5dGZ9MGMbt6MWiWIzZg=</ds:DigestValue>
</ds:Reference>

Returning to XMLDSIG, we can find:

6.6.3 XPath Filtering Input: octet-stream, node-set Output: node-set

Therefore, the result of retrieving of the mentioned ds:Reference is an XML Node set, hence the step 2) from ETSI EN 319 132-1 shall be processed, i.e. a canonicalization algorithm shall be applied to the result of step 1). In our case, the node set from you file shall be canonicalized with a method defined in an xades:AllDataObjectsTimeStamp (which is http://www.w3.org/2001/10/xml-exc-c14n#).

NOTE: the canonicalization here is an additional step defined explicitly for a timestamp message-imprint computation, and it is not executed by default with Apache Santuario Signature.

After performing the canonicalization you should obtain the same result I have indicated in the first message.

Please, correct me if my understanding of the ETSI standard is misleading in this part.

Best regards, Aleksandr.

P.s. Just to be as clear as possible, the output of XPath Filter 2.0 transform is a node-set as well : https://www.w3.org/TR/2002/REC-xmldsig-filter2-20021108/#sec-ProcModel

jccruellas commented 4 years ago

Oooopss.....you are totally right: when I modified the code, just blindly deleted the post-retrieval processing, i.e. the check whether the retrieved bytes are a node set or not and the subsequent canonicalization if yes (don't know why :-), guess that this kind of things happens when one tries to go faster than one should)...After bringing the code again, the exclusive canonicalization is applied, and the message imprint check succeeds.

Thank you very much for your comments. Apologies for the initial misunderstanding and the last deletion of the post-retrieval processing. I attach a zip file with the reports. You should open the index.html for accessing the different parts. You should be able to see the mssg imprint contribution as well as the contents of the time-stamp token.

Regards Juan Carlos.

PSD: if you agree I would close the issue. sSignature.zip

bsanchezb commented 4 years ago

Thank you very much for the investigations!

Could you please also verify if the change fixes the issue #2 as well?

This ticket can be closed.

P.S. Please also verify a processing of ArchiveTimeStamp type. I believe it has the same issue. If you need an example, I can create a such a file.

jccruellas commented 4 years ago

Will check again the signatures in issue#2.... and recheck ArchiveTimeStamp....as the message imprint input also includes the processing of ds:Reference elements.

Juan Carlos.

jccruellas commented 4 years ago

Good morning Alexander,

I have been re-reading both clause 4.4.3.2 of XML Signature 1.1 and clause 5.5.2.2 of ETSI EN 319 132-1 and I would say that we have an issue in there. The thing would go like this:

Clause 4.4.3.2 of XML Signature 1.1 (The Reference Processing Model) specifies how to retrieve the data object referenced by the URI of the reference, how to process the different transforms, and HOW TO GET THE BYTES TO BE DIGESTED. This last thing is important: this clause actually says that the processing always ends with an octet string, either because the last transform in the chain is a canonicalization or because if this is not the case and the result of the last transform is a node-set, then by default the application must canonicalize it.

Now, what happens is that ETSI TS 319 132-1 is not completely right in its wording in clause 5.5.2.2. Look the text, when it is specified how to incorporate the contribution of the ds:Reference:

Take all the ds:Reference elements in their order of appearance within ds:SignedInfo referencing whatever the signer wants to sign including the SignedProperties element. Process each one as indicated below:

The problem is in the first bullet: "Process the retrieved ds:Reference element according to the reference processing model of XMLDSIG [1], clause 4.4.3.2.". If one interprets "Process" as the complete processing, i.e. the one whose final result is an octet stream because it incorporates a canonicalization (either explicit in the chain of transforms or implicit, as indicated in XML Signature), then the two next bullets do not make sense!.... However so far, my reading was that "process" was restricted to two steps: 1) Retrieve the data referenced by the URI and 2) Process the. chain of transforms. If this is the "process", then the two last bullets replace the last step in clause 4.4.3.2 of XMLSig, and then if the final transformation is an explicit canonicalization, the canonicalization in the ArchiveTimeStamp does not play any role, as the result of that canonicalization is an octet stream, but if the result of the last transform is a node-set (and this can be seen from the transformation itself -at least the registered transforms in XML Sig), then this node-set is canonicalized using the algorithm indicated in ArchiveTimeStamp instead of the regular canonical algorithm.

After all these disgressions, I propose the following:

  1. Agree in that the right reading of this clause is the last one, ie, that "processing" refers only to the retrieval of the data object referenced by the URI and the computation of all the transforms.

  2. Propose to ESI to ammend clause 5.5.2.2 of ETSI 319 132-1 (we are in a phase of reviewing of this series of ENs) so that the wording makes it clear that only retrieval of data object referenced by the URI and the computation of the transforms constitute the "processing" mentioned in the first bullet.

  3. Rework the AdESCC to align to this interpretation. This would require to take your signatures and see how the AdESCC should behave.

Hope this helps.

Juan Carlos.

bsanchezb commented 4 years ago

Dear Juan Carlos,

Thank you for the investigations. I completely agree with your position and the conclusion.

Just a few comments about XMLDSIG algorithm 4.4.3.2. From my understanding the algorithm clearly allows a node-set as an output:

The data-type of the result of URI dereferencing or subsequent Transforms is either an octet stream or an XPath node-set.

The Transforms specified in this document are defined with respect to the input they require. The following is the default signature application behavior:

  • If the data object is an octet stream and the next transform requires a node-set, the signature application must attempt to parse the octets yielding the required node-set via [XML10] well-formed processing.
  • If the data object is a node-set and the next transform requires octets, the signature application must attempt to convert the node-set to an octet stream using Canonical XML [XML-C14N].

Please note that the implicit canonicalization is applied only when the next step requires the octets. In our case it is not required, because of step 2) in clause 5.5.2.2.

Therefore depending on the last transform output we decide if the step 2) of clause 5.5.2.2 shall be applied or not. The transforms output types can be obtained from XMLDSIG as you said. Or did I miss something (related to your first interpretation of the "process")?

And about the following points:

  1. Agree in that the right reading of this clause is the last one, ie, that "processing" refers only to the retrieval of the data object referenced by the URI and the computation of all the transforms.

As I have described, I agree with this interpretation of the 4.4.3.2 process.

  1. Propose to ESI to ammend clause 5.5.2.2 of ETSI 319 132-1 (we are in a phase of reviewing of this series of ENs) so that the wording makes it clear that only retrieval of data object referenced by the URI and the computation of the transforms constitute the "processing" mentioned in the first bullet.

By my opinion, the current 5.5.2.2 is still correct. Maybe some clarifications are required, in a form of "NOTE" for example. Additionally I would insist on the clarification of the "XML Node set" term. Currently the wording is ambiguous (one can understand it like a possibility to convert the output of the step 1) to an XML Node set, and not based on the last transform output). I think it must be clearly defined that a canonicalization on the step 2) must be applied only when the last transform returns a node-set according to its specification in XMLDSIG.

  1. Rework the AdESCC to align to this interpretation. This would require to take your signatures and see how the AdESCC should behave.

I will share with you some signatures (with content and archive tsts), once I will start to work on the changes in DSS (in the current version the algorithm synchronized with XAdES CC is used).

Best regards, Aleksandr.

jccruellas commented 4 years ago

Hi Aleksandr, the reason why I said that the processing as per clause 4.4.3.2 of XML Sig always results in an octet stream regardless the chains of transforms if because after the two bullets with the Ifs, this clause contains the following paragraph:

Users may specify alternative transforms that override these defaults in transitions between transforms that expect different inputs. The final octet stream contains the data octets being secured. The digest algorithm specified by DigestMethod is then applied to these data octets, resulting in the DigestValue.

Look at the second sentence: "the final octet stream"...the final octet stream of this processing, I understand. Therefore it is this sentence the one that mandates to XML Signature applications to apply the C1N4 regular algorithm in the case that the last transform results in a node-set.

In any case I share with you your views on the fact that also the clarification of the Node-set in reference to the output of the last transform is required.

Juan Carlos.