WSSE X509 signature fails with Zeep, works with SoapUI and PHP's SoapClient

[gboor]

1. The version of zeep (or if you are running master the commit hash/date) 4.0.0

2. The WSDL you are using Local WSDL file, can be downloaded with all XSDs here; mmchub.zip

3. And most importantly, a runnable example script which exposes the problem. test.zip

I have a service that requires both transport auth and WSSE signing. Transport auth works fine, but the remote end keeps saying the signature is wrong.

I set up the same endpoint in SoapUI and created a test project in PHP - both of which work fine with the exact same certificates, so this is clearly an issue in Zeep, or xmlsec or something.

I've followed the steps in #1084, but that did not help. I've also added the timestamps, again it didn't help. I am not sure what to do next here.

For security reasons I cannot provide the actual certificates and passwords to use for the test script, but I can provide a dump of the generated output; request.zip. I have also included the certificate and public key that are used on the other side for verification.

For reference, here is a request generated by PHP that DOES work; working.zip - I have meticulously compared the 2, but cannot find any real differences in structure. I've also tried other signature methods, like BinarySignature, exporting the p12 to PEMs and loading those, etc. - all to no avail.

I hope that someone can make sense of this, or at least point me in the right direction. I've spent the last 2 days hitting my head against various walls on this issue and I'm out of ideas.

[gboor]

A further update on this; I managed to get the whole end-to-end cycle working with PHP for the time being and all signatures work and are accepted by the remote party.

The library I use for signing in PHP is https://github.com/robrichards/wse-php. I will spend some time over the next couple of weeks to try and translate the logic there into Python to see if that works.

[AnibalRGC]

Hi gboor,

I am currently working with exactly the same wsdl using the same library version. I keep getting "TEN-500021 - Incorrect message signing" error. I guess it is the same as yours.

It works totally fine with SoapUI.

I can not find another Python library. Did you manage to make progress on the issue ?

[gboor]

Nope, never made it work. I tried GO, but there are no soap libs that also support WSSE signing... I eventually built just this part in PHP - which strangely worked out of the box. I prefer not having PHP in a production environment, but it's the only thing that worked on short notice.

I also tried some other Python libraries just for the signing part, but I couldn't get any of them to work.

[ba1dr]

@gboor I am having the same issue with TenneT's API. Calls to "tqf" environment do not require signing and it works somehow, but for "acc" it returns an error. Could you please share your PHP working code? Perhaps I can adapt it to Python myself or end up with PHP too.

@AnibalRGC I cannot make it working with SoapUI either. Can you share your solution too (without keys of course)?

[gboor]

@ba1dr I am using the built-in SOAP libraries from PHP for the normal connection and https://github.com/robrichards/wse-php for the WSSE. I tried porting it to Python, but at some point it goes into the actual underlying C libs and that's where the results are different. I was never able to figure it out, so I'm still running the PHP version of the code.

I can't share the whole thing, but here is the class I use for communicating to TenneT MMC-Hub (which requires both transport certs and signing certs);

<?php

use RobRichards\WsePhp\WSSESoap;
use RobRichards\XMLSecLibs\XMLSecurityKey;

class TennetSoap extends SoapClient
{
    private $signingCert;
    private $signingPk;

    public function __construct($wsdl, $transportCert, $transportPk, $signingCert, $signingPk,
                                array $options = null)
    {
        $context = stream_context_create([
            'ssl' => [
                'local_cert' => $transportCert,
                'local_pk'   => $transportPk
            ]]);

        if(is_null($options))
        {
            $options = [];
        }

        $options['stream_context'] = $context;

        parent::__construct($wsdl, $options);

        $this->signingCert = $signingCert;
        $this->signingPk = $signingPk;
    }

    public function __doRequest($request, $location, $saction, $version, $one_way = NULL)
    {
        $doc = new DOMDocument('1.0');
        $doc->loadXML($request);

        $objWSSE = new WSSESoap($doc);

        // create new XMLSec Key using AES256_CBC and type is private key
        $objKey = new XMLSecurityKey(XMLSecurityKey::RSA_SHA1, array('type' => 'private'));

        // load the private key from file - last arg is bool if key in file (true) or is string (false)
        $objKey->loadKey($this->signingPk, true);

        // Sign the message - also signs appropriate WS-Security items
        $options = array("insertBefore" => false);
        $objWSSE->signSoapDoc($objKey, $options);

        // Add certificate (BinarySecurityToken) to the message
        $token = $objWSSE->addBinaryToken(file_get_contents($this->signingCert));

        // Attach pointer to Signature
        $objWSSE->attachTokentoSig($token);

        return parent::__doRequest($objWSSE->saveXML(), $location, $saction, $version, $one_way);
    }
}

This extends the built-in SoapClient and adds the signing step before sending it off. It can be used to call remote SOAP functions as you would with the normal SoapClient.

The various keys given to the constructor are all pointers to PEM files.

I have another abstract class that I use to wrap this client into an job that I then use to extend jobs from;

<?php

abstract class SoapJob extends DatabaseJob
{
    /**
     * @var TennetSoap The SOAP client to use for SOAP calls
     */
    protected $client;

    /**
     * SoapJob constructor.
     * @throws SoapFault
     */
    public function __construct()
    {
        parent::__construct();

        // Create SOAP client
        $this->client = new TennetSoap($this->getWsdl(), $_SERVER['TRANSPORT_CERT'], $_SERVER['TRANSPORT_PK'],
            $_SERVER['SIGNING_CERT'], $_SERVER['SIGNING_PK']);
        $this->client->__setLocation($this->getLocation($_SERVER['IS_TEST'] == '1'));
    }

    /**
     * Sets the addressing SOAP header with the given content type
     *
     * @param string $technicalMessageId string The technical message ID
     * @param $contentType string The content type of the addressing; ACTIVATED_FCR or ACK_ACTIVATED_FCR
     * @param $correlationId string|null (Optional) the correlation ID for this message
     */
    protected function setAddressing(string $technicalMessageId, string $contentType, string $correlationId = null)
    {
        $headers = [
            'technicalMessageId' => $technicalMessageId,
            'senderId' => Settings::SENDER_EAN,
            'receiverId' => Settings::RECEIVER_EAN,
            'carrierId' => Settings::CARRIER_EAN,
            'contentType' => $contentType
        ];

        if(isset($correlationId))
        {
            $headers['correlationId'] = $correlationId;
        }

        $header = new SoapHeader('http://sys.svc.tennet.nl/MMCHub/Header/v1', 'MessageAddressing', $headers);
        $this->client->__setSoapHeaders($header);
    }

    /**
     * Get the WSDL path for this job
     *
     * @return string Path to the WSDL file to use
     */
    abstract protected function getWsdl();

    /**
     * Get the SOAP URL for this job
     *
     * @param bool $isTest True if running in test mode, False if running in production
     * @return string The URL to use for this operation
     */
    abstract protected function getLocation(bool $isTest);
}

The getLocation is needed as the WSDL provided by TenneT does not point to any existing endpoint. The WSDL is also included locally and loaded from the local filesystem.

Hope that helps!

[MushyMiddle]

I've run into this issue as well - short version, Zeep (4.2.1 from PyPI), request is sent and processed OK using WSSE signing on the request, but the response, like most I've seen in other SOAP-based services, is unsigned, so SignatureVerificationFailed, even though the response was returned with no errors.

I was able to work around this for the moment by using Settings(raw_response=True), but then I guess I'm on my own in terms of parsing the response vs. using the (nice) built-in WSDL object factory stuff.

Seems like we need some way to tell Zeep to expect unsigned responses.

[kwetag]

@MushyMiddle Could you please share how you used WSSE signing in Zeep 4.2.1 to send correctly signed XML request?

[MushyMiddle]

@kwetag Relevant code:

from zeep import Client, Settings, xsd
from zeep.cache import SqliteCache
from zeep.transports import Transport
from zeep.wsse.signature import Signature, MemorySignature
from pathlib import Path
from cryptography.hazmat.primitives.serialization import Encoding, PrivateFormat, NoEncryption
from cryptography.hazmat.primitives.serialization.pkcs12 import load_key_and_certificates
from cryptography.hazmat.backends import default_backend
import logging.config
import argparse, configparser, sys, os, requests, io, base64, gzip
import xml.etree.ElementTree as ET
...
# Builds the Signature based on the configuration, either PFX or PEM (key, cert)
def build_signature(defaults):

    signature = None`
    pfx_file = defaults['SigningCertPFX']

    # Handle PKCS#12 if specified
    if pfx_file != None and pfx_file != '':

        pfx_data = Path(pfx_file).expanduser().read_bytes()
        key, cert, add_certs = load_key_and_certificates(pfx_data, defaults['SigningPassword'].encode('utf-8'), default_backend())
        key_bytes = key.private_bytes(Encoding.PEM, PrivateFormat.PKCS8, NoEncryption())
        cert_bytes = cert.public_bytes(Encoding.PEM)
        signature = MemorySignature(key_bytes, cert_bytes, None)

    # Otherwise, use cert/key PEM files
    else:

        cert_file = Path(defaults['SigningCert']).expanduser()
        key_file = Path(defaults['SigningCertKey']).expanduser()
        signature = Signature(certfile=cert_file, key_file=key_file, password=defaults['SigningPassword'])

    return signature

# Builds the SOAP client
def build_client(args, defaults):

    # Work around https://github.com/mvantellingen/python-zeep/issues/1158
    settings = Settings(raw_response=True, force_https=False)

    # Use the ID cert registered
    signature = build_signature(defaults)

    # Cache WSDL
    # TODO: May need to be persistent to be effective
    session = build_session(args)

    transport = Transport(session=session, cache=SqliteCache(), timeout=args['timeout'])

    # Create the client
    try:
        client = Client(defaults['SOAPEndpoint'] + "?wsdl", transport=transport, wsse=signature, settings=settings)

        return client

    except HTTPError as e:

        print(f"Exception loading WSDL: {e} ")
        sys.exit(os.EX_NOHOST)

# Later, in Main:
    # Build the connection/request
    client = build_client(args, defaults)
    service = build_service(client, defaults)
    request = build_request(client, args, defaults)
...
            response = service.doMethod(None, request)

[kwetag]

@MushyMiddle Thanks for your answer. I was already at a similar code and the MMCHUB isAlive service is working fine with it. On the other hand when the service requires a header, like getting the messages, there is always the signing error. It cannot be a problem with the key+certificate because it is working with SOAP UI.

I had to downgrade the version of lxml to 4.9.3 because the higher version (>=5.0.0) was giving a segmentation fault with xmlsec. Furthermore as the default wsdl binding is not pointing toward TQF, the service is created as follows:

service = client.create_service( binding_name="{default url}MMCHubBinding", address=f"default url with tqf in the middle", )

but since the isAlive is working it cannot be the source of the problem.

In build_request above, there is probably the creation of the SOAP Headers (like a dict) and the SOAP Body (also dict) but then do you do something special when Zeep signs the header?

[MushyMiddle]

@kwetag The only thing I do prior to using the factory in build_request is:

    factory = client.type_factory('ns0')
    request = factory.MyRequest(...pass in a bunch of args...)

[kwetag]

@MushyMiddle OK, strange that for me Zeep signing is not working out of box like for you. There is probably something different between our setup.

[airguru]

@kwetag I have a same problem, but when I inspected generated SOAP message, I have noticed that default signature and digest methods are not up to MMC hub specs.

The specs say: Only the payload of the message is signed (everything within the SOAP body), using the following specs: • SignatureMethod – RSAwithSHA256 • CanonicalizationMethod – xml-exc-c14n# • DigestMethod – xmlenc#sha256 • KeyInfo/X509Data – X509SubjectKeyIdentifier (X509IssuerSerial has been deprecated)

Canonicalization method seems to be ok. I believe, that signature and digest methods may be altered like this:

signature = Signature('privkey.pem', 'certificate.crt', signature_method=xmlsec.constants.TransformRsaSha256, digest_method=xmlsec.constants.TransformSha256)

However I have also noticed, that request uses X509IssuerSerial which is deprecated in spec. I don't know how to adjust that, it seems to be hardcoded in BinarySignature._signature_prepare() method.

Another contributing factor is, that I am currently not supplying the full certificate chain.

@MushyMiddle do you have anything relevant to this in your code?

[slavanyx]

Hi, have you managed to solve this problem? If yes, would you be able to post the solution? I am looking for help with Tennet API onboarding - if you can help, please reach out to slava@slavany.com Potentially I am looking for paid consulting/help with Tennet onboarding @MushyMiddle @kwetag @airguru TIA