Add support for SOAP

[madhukaw]

Description:

Need to improve the soap module to facilitate new requirements.

[shafreenAnfar]

Following meeting notes from release planning meeting.

Support for consuming SOAP services in Ballerina

• Is there a WSDL? If yes
  • We can write bal wsdl –mode=client foo.wsdl
  • WSDL can contain refs to use of WS-Addressing, SOAPAction, WS-Sec basic auth and some basic token profile
• We don’t need to support other WS-* specs
• If not, we can write a simple SOAPClient remote client similar to HTTPClient for SOAP/HTTP
  • WSDL generated client one can use this
  • We can do SOAP/JMS the same way
• Support WSDL 1.1, SOAP 1.1 and 1.2, WS-Addressing xx, WS-Security yy

[shafreenAnfar]

This is what we need to target to support.

[shafreenAnfar]

Sample SOAP message.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://www.example.com/webservice">
    <soapenv:Header>
        <!-- Optional SOAP header elements go here -->
    </soapenv:Header>
    <soapenv:Body>
        <web:HelloRequest>
            <web:Name>John Doe</web:Name>
        </web:HelloRequest>
    </soapenv:Body>
</soapenv:Envelope>

[shafreenAnfar]

Sample MTOM message.

POST /webservice HTTP/1.1
Host: www.example.com
Content-Type: multipart/related; type="application/xop+xml"; boundary="uuid:1ab9c480-8f78-482a-92f8-7df4f1e8e2c1"; start="<soap@envelope>"; start-info="text/xml"

--uuid:1ab9c480-8f78-482a-92f8-7df4f1e8e2c1
Content-Type: application/xop+xml; charset=UTF-8; type="text/xml"
Content-ID: <soap@envelope>

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
                  xmlns:web="http://www.example.com/webservice"
                  xmlns:xop="http://www.w3.org/2004/08/xop/include">
    <soapenv:Header>
        <!-- Optional SOAP header elements go here -->
    </soapenv:Header>
    <soapenv:Body>
        <web:UploadAttachments>
            <!-- Image Attachment -->
            <web:Attachment>
                <xop:Include href="cid:image1"/>
            </web:Attachment>
            <!-- Additional Attachment -->
            <web:Attachment>
                <xop:Include href="cid:file1"/>
            </web:Attachment>
        </web:UploadAttachments>
    </soapenv:Body>
</soapenv:Envelope>

--uuid:1ab9c480-8f78-482a-92f8-7df4f1e8e2c1
Content-Type: image/jpeg
Content-Transfer-Encoding: binary
Content-ID: <image1>

... Binary Image Data ...

--uuid:1ab9c480-8f78-482a-92f8-7df4f1e8e2c1
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
Content-ID: <file1>

... Binary File Data ...

--uuid:1ab9c480-8f78-482a-92f8-7df4f1e8e2c1--

[shafreenAnfar]

Sample WSDL.

<?xml version="1.0" encoding="UTF-8"?>
<wsdl:definitions xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/"
                  xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
                  targetNamespace="http://www.example.com/weather">
    <wsdl:types>
        <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" targetNamespace="http://www.example.com/weather">
            <xs:element name="City" type="xs:string"/>
            <xs:element name="WeatherForecast" type="xs:string"/>
        </xs:schema>
    </wsdl:types>
    <wsdl:message name="GetWeatherRequest">
        <wsdl:part name="City" element="tns:City"/>
    </wsdl:message>
    <wsdl:message name="GetWeatherResponse">
        <wsdl:part name="WeatherForecast" element="tns:WeatherForecast"/>
    </wsdl:message>
    <wsdl:portType name="WeatherServicePortType">
        <wsdl:operation name="GetWeather">
            <wsdl:input message="tns:GetWeatherRequest"/>
            <wsdl:output message="tns:GetWeatherResponse"/>
        </wsdl:operation>
    </wsdl:portType>
    <wsdl:binding name="WeatherServiceBinding" type="tns:WeatherServicePortType">
        <soap:binding style="document" transport="http://schemas.xmlsoap.org/soap/http"/>
        <wsdl:operation name="GetWeather">
            <soap:operation soapAction="http://www.example.com/weather/GetWeather"/>
            <wsdl:input>
                <soap:body use="literal"/>
            </wsdl:input>
            <wsdl:output>
                <soap:body use="literal"/>
            </wsdl:output>
        </wsdl:operation>
    </wsdl:binding>
    <wsdl:service name="WeatherService">
        <wsdl:port name="WeatherServicePort" binding="tns:WeatherServiceBinding">
            <soap:address location="http://www.example.com/weather"/>
        </wsdl:port>
    </wsdl:service>
</wsdl:definitions>

[shafreenAnfar]

Above SOAP payload converted to JSON.

{
   "soapenv:Envelope":{
      "soapenv:Header":{

      },
      "soapenv:Body":{
         "web:HelloRequest":{
            "web:Name":"John Doe",
            "@xmlns:web":"http://www.example.com/webservice"
         }
      },
      "@xmlns:soapenv":"http://schemas.xmlsoap.org/soap/envelope/",
      "@xmlns:web":"http://www.example.com/webservice"
   }
}

[shafreenAnfar]

A possible client.

public client class SoapClient {

    public function init(string url, http:ClientConfiguration config) {}

    remote function sendReceive(xml|mime:Entity[] soap) returns xml|mime:Entity[]|error {}

    remote function sendOnly(xml|mime:Entity[] soap) returns error? {}

    remote function sendReceiveEnve(SoapEnv soapenv, boolean simpleJsonMapping = true) returns SoapEnv|error {}

    remote function sendOnlyEnve(SoapEnv soapenv, boolean simpleJsonMapping = true) returns error? {}
}

public type SoapEnv record {|
    xml|record {} header?;
    xml|record {} body;
|};

// Is for the MTOM suff
public type Attachment record {|
    string name;
    byte[]|stream<byte[]>|string content;
|};

[shafreenAnfar]

Just like we have VSCode command to go from JSON to record, we need one to go from XML to record.

[manuranga]

We can use xml|[xml, mime:Entity...] soap instead xml|mime:Entity[] soap as a better type.

[manuranga]

s/simpleJsonMapping/ingoreNamespace/g

[manuranga]

We are providing two levels of functionality here. The low-level api (sendReceive, sendOnly) and high-level opinionated api (sendReceiveEnve, sendOnlyEnve). Better to give these as separate abstractions. Users for former should not concern themselves with the latter.

[shafreenAnfar]

I think we can start with the below two clients.

public client class BasicClient {

    public function init(string url, http:ClientConfiguration config) {}

    remote function sendReceive(xml|mime:Entity[] soap) returns xml|mime:Entity[]|error {}

    remote function sendOnly(xml|mime:Entity[] soap) returns error? {}
}

public client class AdvancedClient {

    public function init(string url, http:ClientConfiguration config) {}

    remote function sendReceive(SoapEnv soapenv, boolean simpleJsonMapping = true) returns SoapEnv|error {}

    remote function sendOnly(SoapEnv soapenv, boolean simpleJsonMapping = true) returns error? {}
}

public type SoapEnv record {|
    xml|record {} header?;
    xml|record {} body;
|};

// Is for the MTOM suff
public type Attachment record {|
    string name;
    byte[]|stream<byte[]>|string content;
|};

[sameerajayasoma]

Had a quick chat with @shafreenAnfar on this API. We agreed to go ahead with the BasicClient first. We can review the AdvancedClient in the meantime.

Here are some of the things that we discussed:

• Can we say soap:Client instead of soap:BasicClient?
• [Optional] I prefer fireAndForgot instead of sendOnly as it is commonly used in this context. See: https://axis.apache.org/axis2/java/core/apidocs/org/apache/axis2/client/ServiceClient.html#fireAndForget(org.apache.axiom.om.OMElement)
• I would use soap:Envelop instead of soap:SoapEnv

[shafreenAnfar]

Given the behaviour of the SOAPAction, I think we are better off having two clients which user common code underneath as soap11:Client and soap12:Client.

[Nuvindu]

SOAP 1.1 & 1.2 clients offer the capability to configure and apply one or multiple security policies during the initiation of a SOAP client connection. These security policies can be applied as follows:

SOAP 1.1 Client: UsernameToken and TranportBinding Policy

import ballerina/crypto;
import ballerina/mime;
import ballerina/soap:soap11;
import ballerina/soap:wssec;

public function main() returns error? {
    soap11:Client soapClient = check new ("http://www.dneonline.com/calculator.asmx?WSDL", 
        {
            soapSecurity: [
            {
                username: "username",
                password: "password",
                passwordType: wssec:TEXT
            },
            TRANSPORT_BINDING
            ]
        });

    xml envelope = xml `<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
                            <soap:Body>
                            <quer:Add xmlns:quer="http://tempuri.org/">
                                <quer:intA>2</quer:intA>
                                <quer:intB>3</quer:intB>
                            </quer:Add>
                            </soap:Body>
                        </soap:Envelope>`;
    xml|mime:Entity[] response = check soapClient->sendReceive(envelope, "http://tempuri.org/Add");
}

SOAP 1.2 Client: Symmetric Binding Policy

import ballerina/crypto;
import ballerina/mime;
import ballerina/soap:soap12;
import ballerina/soap:wssec;

public function main() returns error? {
    crypto:KeyStore keyStore = {
        path: KEY_STORE_PATH,
        password: KEY_PASSWORD
    };

    crypto:PrivateKey privateKey = check crypto:decodeRsaPrivateKeyFromKeyStore(keyStore, KEY_ALIAS, KEY_PASSWORD);
    crypto:PublicKey publicKey = check crypto:decodeRsaPublicKeyFromTrustStore(keyStore, KEY_ALIAS);

    soap12:Client soapClient = check new ("http://www.dneonline.com/calculator.asmx?WSDL", 
        {
            soapSecurity: {
                signatureAlgorithm: wssec:RSA_SHA256,
                encryptionAlgorithm: wssec:RSA_ECB,
                bindingKey: privateKey,
                verificationKey: publicKey,
                x509Token: X509_PUBLIC_CERT_PATH_2
            };
        });

    xml envelope = xml `<soap:Envelope
                        xmlns:soap="http://www.w3.org/2003/05/soap-envelope"
                        soap:encodingStyle="http://www.w3.org/2003/05/soap-encoding">
                            <soap:Body>
                            <quer:Add xmlns:quer="http://tempuri.org/">
                                <quer:intA>2</quer:intA>
                                <quer:intB>3</quer:intB>
                            </quer:Add>
                            </soap:Body>
                        </soap:Envelope>`;
    xml|mime:Entity[] response = check soapClient->sendReceive(envelope, "http://tempuri.org/Add");
}

[Nuvindu]

We decided to use two security configurations for the SOAP client:

• inboundSecurity: This configuration is applied to the SOAP envelope when a request is made. It includes various ws security policies such as Username Token, Timestamp Token, X509 Token, Symmetric Binding, Asymmetric Binding, and Transport Binding, either individually or in combination with each other.

• outboundSecurity: This configuration is applied to the SOAP envelope when a response is received. Its purpose is to decrypt the data within the envelope and verify the digital signature for security validation.

import ballerina/crypto;
import ballerina/mime;
import ballerina/soap:soap12;
import ballerina/soap:common;

public function main() returns error? {
    crypto:KeyStore clientKeyStore = {
            path: X509_KEY_STORE_PATH_2,
            password: KEY_PASSWORD
    };
    crypto:PrivateKey clientPrivateKey = check crypto:decodeRsaPrivateKeyFromKeyStore(clientKeyStore, KEY_ALIAS, KEY_PASSWORD);

    crypto:PublicKey clientPublicKey = check crypto:decodeRsaPublicKeyFromTrustStore(clientKeyStore, KEY_ALIAS);

    ​​crypto:PublicKey serverPublicKey = ...//

    soap12:Client soapClient = check new ("http://www.dneonline.com/calculator.asmx?WSDL",
    {
            inboundSecurity: {
                signatureKey: clientPrivateKey,
                signatureAlgorithm: wssec:RSA_SHA256,
                encryptionKey: serverPublicKey,
                encryptionAlgorithm: wssec:RSA_ECB
            },
            outboundSecurity: {
                verificationKey: serverPublicKey,
                signatureAlgorithm: wssec:RSA_SHA256,
                decryptionKey: clientPrivateKey,
                decryptionAlgorithm: wssec:RSA_ECB
            }
    });

    xml envelope = xml `<soap:Envelope
                        xmlns:soap="http://www.w3.org/2003/05/soap-envelope"
                        soap:encodingStyle="http://www.w3.org/2003/05/soap-encoding">
                            <soap:Body>
                            <quer:Add xmlns:quer="http://tempuri.org/">
                                <quer:intA>2</quer:intA>
                                <quer:intB>3</quer:intB>
                            </quer:Add>
                            </soap:Body>
                        </soap:Envelope>`;

    xml|mime:Entity[] response = check soapClient->sendReceive(envelope, "http://tempuri.org/Add");
}

[Nuvindu]

Following our discussion, we've decided to remove the soap.common sub-module and merge into the main soap module. As a result, we will now expose only three modules: soap, soap.soap11, and soap.soap12.

import ballerina/crypto;
import ballerina/mime;
import ballerina/soap;
import ballerina/soap:soap12;

public function main() returns error? {
    crypto:KeyStore clientKeyStore = {
            path: KEY_STORE_PATH,
            password: KEY_PASSWORD
    };
    crypto:PrivateKey clientPrivateKey = check crypto:decodeRsaPrivateKeyFromKeyStore(clientKeyStore, KEY_ALIAS, KEY_PASSWORD);
    crypto:PublicKey clientPublicKey = check crypto:decodeRsaPublicKeyFromTrustStore(clientKeyStore, KEY_ALIAS);
    ​​crypto:PublicKey serverPublicKey = ...//

    soap12:Client soapClient = check new ("http://www.dneonline.com/calculator.asmx?WSDL",
    {
            inboundSecurity: {
                    signatureAlgorithm: soap:RSA_SHA256,
                    encryptionAlgorithm: soap:RSA_ECB,
                    signatureKey: clientPrivateKey,
                    encryptionKey: serverPublicKey,
            },
            outboundSecurity: {
                    verificationKey: serverPublicKey,
                    signatureAlgorithm: soap:RSA_SHA256,
                    decryptionKey: clientPrivateKey,
                    decryptionAlgorithm: soap:RSA_ECB
            }
    });

    xml body = xml `<soap:Envelope
                    xmlns:soap="http://www.w3.org/2003/05/soap-envelope"
                    soap:encodingStyle="http://www.w3.org/2003/05/soap-encoding">
                    <soap:Body>
                    <quer:Add xmlns:quer="http://tempuri.org/">
                        <quer:intA>2</quer:intA>
                        <quer:intB>3</quer:intB>
                    </quer:Add>
                    </soap:Body>
                </soap:Envelope>`;

    xml|mime:Entity[] response = check soapClient->sendReceive(envelope, "http://tempuri.org/Add");
}

[Nuvindu]

In the previous SOAP APIs, users had to work with a union type of xml | mime:Entity[] for the responses. In most cases, this required users to add conditional statements to extract the specific type from the union.

xml|mime:Entity[] response = check soapClient->sendReceive(envelope, "http://tempuri.org/Add");
if response is xml {
    //…
} else {
    //…
}

However, with the new update, this additional condition is no longer necessary. Users can now directly infer the response type, whether it's xml or mime:Entity[].

xml response = check soapClient->sendReceive(envelope, "http://tempuri.org/Add");

OR

mime:Entity[] response = check soapClient->sendReceive(envelope, "http://tempuri.org/Add");