Signing multiple references and non-XML content in detached mode

[asbjornoskal]

Hi

I am trying to port some C code to python that is signing an XML referencing another document, a so called detached signature. I was told that it is possible to specify "detached" as signature method when creating XMLSigner. But then I expected that I also had to supply an uri-resolver method to the sign method just like with the verify method to supply the data for the uri specified. Something like the following:

def my_uri_resolver(self, uri):
    if uri == 'uri1':
        return 'data for uri1'
    if uri == 'uri2':
        return 'data for uri2'
...
signer = XMLSigner()
signed = signer.sign(
    unsigned_xml, 
    key=my_key,
    passphrase=my_passphrase,
    cert=my_cert,
    reference_uri=['uri1', 'uri2'],
    uri_resolver=my_uri_resolver)
...

Am I missing something here?

Regards Asbjørn Oskal

[kislyuk]

Quoting the README file:

signed_root = XMLSigner(method=signxml.methods.detached).sign(root, key=key, cert=cert)

For detached signatures, the code above will use the Id or ID attribute of root to generate a relative URI (<Reference URI="#value"). You can also override the value of URI by passing a reference_uri argument to sign(). To verify a detached signature that refers to an external entity, pass a callable resolver in XMLVerifier().verify(data, uri_resolver=...).

You pass the data to be referenced by the signature directly as the first argument.

[asbjornoskal]

Hi again and thanks for the response.

I am sorry to bug you but the scenario I want to solve is this: I have two files. One envelope-xml and one payload (It can be any kind of file really. For example a gif-file) I sign the envelope-xml with a reference to the gif-file so that in the envelope xml there will be two signature elements. One for the envelope itself and one for the external file The DigestValue for "cid:ref1" will be the hash of the external gif-file.

The envelope-xml should then end up with SignedInfo content like this:

...
<SignedInfo>
    <CanonicalizationMethod Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315"/>
    <SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/>
    <Reference URI="">
        <Transforms>
            <Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
            <Transform Algorithm="http://www.w3.org/TR/1999/REC-xpath-19991116">
                <XPath xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">not(ancestor-or-self::node()[@SOAP-ENV:actor="urn:oasis:names:tc:ebxml-msg:actor:nextMSH"] | ancestor-or-self::node()[@SOAP-ENV:actor="http://schemas.xmlsoap.org/soap/actor/next"])</XPath>
            </Transform>
            <Transform Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315"/>
        </Transforms>
        <DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/>
        <DigestValue>wIZFJZjZ7LjKHT6cRmFvlQrxcXA=</DigestValue>
    </Reference>
    <Reference URI="cid:ref1">
        <DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/>
        <DigestValue>tzvUabyaqBNxxhQ+NfkgpwqTFvk=</DigestValue>
    </Reference>
</SignedInfo>
...

The xml over is generated signing with xmlsec library (https://www.aleksey.com/xmlsec/). It can also be accomplished with Microsoft's SignedXml class by supplying a resolver to supply the data (Even if Microsoft default has disabled detached signatures). In this way the signed xml is signing both the envelope and the external payload. (The external reference could even be an uri like "http://somehost/somefolder/somefile.png")

I cannot see how

signed_root = XMLSigner(method=signxml.methods.detached).sign(root, key=key, cert=cert)

is able to solve this scenario as long as there is no way for the sign method to resolve the uris passed in with the reference_uri parameter. I guess you can reference another node in the xml, but there is currently no way for the signing process to get hold of the external data like in my example.

Regards Asbjørn Oskal

[kislyuk]

OK, so to clarify, you're looking to be able to generate a detached signature with multiple references, right? That use case is not currently supported.

[kislyuk]

I must confess I don't understand what <Reference URI=""> means either; can you point me to any docs that describe this construction?

[asbjornoskal]

Hi again and thnk you for the quick answer.

Yes you are correct. I need to sign with two references. The first reference which is to an element in the xml itself (ref="") and and the other reference to some external data.

From the specification (https://www.w3.org/TR/xmldsig-core/#sec-Reference): URI="" Identifies the node-set (minus any comment nodes) of the XML resource containing the signature

Or in other words it refers to the element in which the signature element is placed.

Regards Asbjørn Oskal

[kislyuk]

OK, this should not be very hard to implement, but I'd like to have a complete test example to work with. Can you please supply or find one?

[asbjornoskal]

I will try to come up with some suitable example. I guess you mean some code and example xml and payload.

Regards Asbjørn Oskal

[asbjornoskal]

XMLDSIGTest.zip

[kislyuk]

Thanks, I'm taking a look at implementing this.

[kislyuk]

See https://www.w3.org/TR/xmldsig-core2/#sec-o-SignatureProperty

[asbjornoskal]

Hi

I am not sure in what way the link you posted is relevant to my case other than that in xmldsig 2.0 there are other ways of referencing the objects signed.

The case I have described is what is described in the chapter "Compatibility mode" https://www.w3.org/TR/xmldsig-core2/#sec-Compatibility-Mode

Here the reference uri like in my case points out of the xml to some content. In the example it points to "http://www.w3.org/TR/2000/REC-xhtml1-20000126/" so it in effect signs the HTML content of that link.

Regards Asbjørn Oskal

[kislyuk]

I posted the link for my own (implementer's) reference as it demonstrates the use of multiple references (one internal and one external). SignXML works fine with external references, it just doesn't yet support multiple references when generating an external reference. That's what this issue is about.

[asbjornoskal]

Hi again

You wrote that "SignXML works fine with external references". Can you give me an example of this using SignXML? Maybe we mean different things by "external references".

Regards Asbjørn

[kislyuk]

I gave an example in my first comment: https://github.com/XML-Security/signxml/issues/149#issuecomment-640325908

[asbjornoskal]

Hi again

I am sorry but unless you can show me how to reference an external file I do not agree that SignXML does support signing external files like in the c# example code I sent you.

In the SignXml code, all references are run through the _resolve_reference method. But the only references resolved are empty uris and uris starting with "#". (And # is added by default to all URIs not starting with # in _get_c14n_inputs_from_reference_uris before calling _resolve_reference) All calls to _resolve_reference when signing, is done with an uri_resolver method parameter set to None.

So in my case sending in an custom URI like "cid:ref1" would, as far as I can see, first end up like "#cid:ref1". Then this would then be tried resolved by an xpath query which is not what I would want is this case. What I would want is that a uri_resolver method implemented in my code could instead be called on this "cid:ref1" uri to return the data my code relates to "cif:ref1". This is what other implementations I have used does.

I still cannot see that SignXML is able to do this.

Regards Asbjørn

[kislyuk]

Here is an example of SignXML performing a detached signature specifying an arbitrary external reference string. The thing missing to support your use case is signing multiple references, including a mix of internal and external ones.

from lxml import etree
from signxml import XMLSigner, XMLVerifier, methods

data_to_sign = "<Test/>"
cert = open("example.pem").read()
key = open("example.key").read()
root = etree.fromstring(data_to_sign)
signed_root = XMLSigner(method=methods.detached).sign(root, key=key, cert=cert, reference_uri="cid:ref1")
print(etree.tostring(signed_root, pretty_print=True).decode())

[asbjornoskal]

Hi

I am sorry, but I have been wrong and called my example a detached signature when it in fact is enveloped signatures referencing both an internal and an external source. I guess I was confused by something in the standard and I apologize for that.

Anyway, I ran the example code and I can see that it is possible to generate a detached signature signing an arbitrary xml. Again I'm sorry that we have been talking passed each other.

Then you wrote: "The thing missing to support your use case is signing multiple references, including a mix of internal and external ones"

I would rewrite this slightly: "The thing missing to support your use case is signing multiple references enveloped, including a mix of internal and external ones, where the external ones could point to any binary source".

The standard alos says that URIs without a #-prefix must point to octet streams and is not to be parsed only digested. ("Unless the URI-Reference is such a 'same-document' reference , the result of dereferencing the URI-Reference must be an octet stream")

Then how to refer to external sources? The standard only mentions http URIs explicitly: "We RECOMMEND XML Signature applications be able to dereference URIs in the HTTP scheme"

However the three implementations I have used (xmlsec, microsoft and java) all supports other custom protocols (like cid: in my case) by supporting custom resolvers.

Regards Asbjørn Oskal

[kislyuk]

I may be rusty on my spec but I don't see a fundamental difference between an enveloped signature with external references outside the enveloping document and a detached signature with external references. You can generate a detached signature and then embed it in a document, making it enveloped.

But you raise a good point - SignXML needs to be able to handle string (octet-stream) references for detached signing, not requiring them to be XML nodes. I've updated the title of the issue to reflect that.

[asbjornoskal]

I have never tried it, but I guess you are right about embedding a detached signature in a document to make it enveloped. Sorry again for a lengthy discussion. It improved my spec knowledge which also may have been a bit corroded.

Regards Asbjørn

[isaacyip621]

Do we have any update on signing / handling data other than XML nodes in the external references? I am currently trying to work this out as well since we need to secure the soap attachment during the request conforming to the SwA standard (http://docs.oasis-open.org/wss-m/wss/v1.1.1/os/wss-SwAProfile-v1.1.1-os.html).

[isaacyip621]

Also, I have worked some prototypes to build a list of reference: [{id = cid: foo, data = data}, {id = bar, data = data}, etc.]

and append this to reference_uri, and then in the back then when processing the reference_uri, detect whether the iterator is a list and hence process the id and data accordingly.

One question I have been trying to figure out is how to pass the gzip.file, or any other mime type really into the c14input and what should be the c14 output as the digest input. Since the data is not a xml node then c14n is not applicable per the standard. Would be great to hear your thoughts.

Best regards Isaac

[kislyuk]

The outcome of the discussion above was that signxml currently lacks the capability to sign multiple external references, and also lacks the capability to sign external references that are not XML nodes.

In all likelihood, the only type of data that will be accepted for signing instead of XML nodes will be an arbitrary bytestring or bytestream. I don't expect there will be special handling for any other data types.

I maintain this project in my spare time; there is no ETA for when this feature will become available. PRs and sponsorships are welcome.

[isaacyip621]

Thank you for your reply. I have been trying out this new signing for the attachment over the weekend and the server has been responding 'signatures verifications error'. The file I have sent through is a gzip file and contain arbitrary binary data and hence no transform has been done. The data is inputted directly as pre-digest value and hence go through the digest value accordingly and successfully obtain the reference node and hash values and etc. the binary data is attachment in the MIME content without any transfer encoding with the email.message.MIMEApplication. Would you be able to share some thoughts on how to troubleshoot? I would reckon once this is settled, my code will be able to assist the project to develope this features. Much appreciated for your time on this. Regards, Isaac

[isaacyip621]

Hi there,

Hope you have been well. This is a great project indeed! I cannot get enough of it but the non-XML content signing part gives me no choice to use other coding languages in the back end. I still prefer python for the backend.

Just wondering would you be able to support the signing of non-XML content? If there anything I can help to fast track this feature? Specifically, the standard is http://docs.oasis-open.org/wss-m/wss/v1.1.1/os/wss-SwAProfile-v1.1.1-os.html.

thanks & regards Isaac

[kislyuk]

Thanks. There are two ways you can help: by donating your time to write a quality PR for that feature, or donating money to support this project using the "Sponsor" button at the top of the page.