search

adessoSE / kong-plugin-soap2rest

A plugin for the Kong Microservice API Gateway to redirect a SOAP request to a REST API and convert the JSON Response to SOAP response.
Apache License 2.0
14 stars 3 forks source link

Installation steps, plugin setup in Kong and usage #3

Open adivardhansingh opened 2 years ago

adivardhansingh commented 2 years ago

This is a great plugin and can be used for legacy systems that are not yet migrated to RESTful services. However, could you please add the steps to install the plugin for a local environment?

The README file has two sections: Installation & Docker

Do we have to clone the project/download the project & execute either Makefile or Dockerfile? Or should we execute both?

Also, do we have to update the environment variable for KONG_PLUGIN or does the config files take care of the same?

And the Development section is for setup of Kong?

Sorry, these questions may sound naive, but I am not at all familiar with Kong and currently in explore state.

DanielKraft commented 2 years ago

The easiest way to test the plugin is to clone the repository and run the Dockerfile. These steps will automatically create a Docker image with soap2rest enabled (so KONG_PLUGIN does not need to be changed).

After starting, the plugin can be configured as described in Enable Plugin. In #2 the configuration and the required parameters are described again.

adivardhansingh commented 2 years ago

@DanielKraft Once we create the image, will Kong be able to fetch it from any location? I am not familiar with Docker, so I am assuming I just have to use the command:docker build Dockerfile from the cloned location?

DanielKraft commented 2 years ago

The easiest way to start a Kong instance with the plugin via Docker is to use the docker-compose.yml file.

Docker-compose setup

To pass the required OpenAPI and WSDL files to Kong, the following line must be modified: https://github.com/adessoSE/kong-plugin-soap2rest/blob/938d5519c34ad8f0dda8d70f43703a5e35746a7a/docker-compose.yml#L75

For example, if you have your OpenAPI and WSDL in a folder like this:

kong-plugin-soap2rest/
├── api/
|   ├── config/
│   │   ├── openAPI.yaml
│   │   └── soap.wsdl

Then change the line to ./api/config:/api/config. This makes the files available inside the Docker container at /kong-plugin/api/config/.

After the file has been customized, Kong can be started with docker compose up.

Kong configuration

After you have started Kong, you can configure Kong via the Kong Admin API.

# Create service
curl --request POST \
  --url http://localhost:8001/services/ \
  --header 'Content-Type: multipart/form-data; boundary=---011000010111000001101001' \
  --form 'name=<service_name>' \
  --form 'url=<rest_api_path>'

# Create rest api route
curl --request POST \
  --url http://localhost:8001/services/<service_name>/routes \
  --header 'Content-Type: multipart/form-data; boundary=---011000010111000001101001' \
  --form 'name=<soap_route_name>' \
  --form 'paths[]=/' \
  --form 'strip_path=false'

# Create soap api route
curl --request POST \
  --url http://localhost:8001/services/<service_name>/routes \
  --header 'Content-Type: multipart/form-data; boundary=---011000010111000001101001' \
  --form 'name=<soap_route_name>' \
  --form 'paths[]=/soap' \
  --form 'strip_path=false'

# Enable plugin for soap api route
curl --request POST \
  --url http://localhost:8001/routes/<soap_route_name>/plugins \
  --header 'Content-Type: multipart/form-data; boundary=---011000010111000001101001' \
  --form 'name=soap2rest' \
  --form 'config.rest_base_path=/' \
  --form 'config.openapi_yaml_path=/kong-plugin/api/config/<openapi yaml path>' \
  --form 'config.wsdl_path=/kong-plugin/api/config/<wsdl path>' \
  --form 'config.operation_mapping.<operation id>=<operation path>'
adivardhansingh commented 2 years ago

@DanielKraft Would we not need to create a SOAP service as well? Which service would the soap route belong to?

adivardhansingh commented 2 years ago

@DanielKraft My Soap API on directly calling (without Kong): curl --header "content-type: text/xml" -d @request.xml http://localhost:8080/ws. This works fine and gives the response as expected.

My Rest API on directly calling (without Kong): curl -i -X GET "http://localhost:8089/products/country?name=Spain". This works fine and gives the response as expected.

Steps Followed: `# Create service curl -i -X POST http://localhost:8001/services \ --data name=country_service \ --data url='http://localhost:8089'

Create rest api route

curl -i -X POST http://localhost:8001/services/country_service/routes \ --data 'name=country_route_rest' \ --data 'paths[]=/products/country' \ --data 'strip_path=false'

Create soap api route

curl -i -X POST http://localhost:8001/services/country_service/routes \ --data 'name=country_route_soap' \ --data 'paths[]=/ws' \ --data 'strip_path=false'

Enable plugin for soap api route

curl -i -X POST http://localhost:8001/routes/country_route_soap/plugins \ --data 'name=soap2rest' \ --data 'config.rest_base_path=/' \ --data 'config.openapi_yaml_path=/api/config/country.yaml' \ --data 'config.wsdl_path=/api/config/countries.wsdl' `

When following the above steps, as you mentioned, and executing the API on Kong: curl --header "content-type: text/xml" -d @request.xml http://localhost:8000/ws

kong | 2022/04/05 09:18:18 [error] 25#0: *41558 [kong] handler.lua:64 [soap2rest] ...al/share/lua/5.1/kong/plugins/soap2rest/wsdl_handler.lua:228: attempt to index field 'definitions' (a nil value), client: 172.20.0.1, server: kong, request: "POST /ws HTTP/1.1", host: "localhost:8000" kong | 2022/04/05 09:18:18 [error] 25#0: *41558 [kong] handler.lua:72 [soap2rest] ...share/lua/5.1/kong/plugins/soap2rest/openapi_handler.lua:116: bad argument #1 to 'pairs' (table expected, got nil), client: 172.20.0.1, server: kong, request: "POST /ws HTTP/1.1", host: "localhost:8000" kong | 2022/04/05 09:18:18 [error] 25#0: *41558 [kong] handler.lua:82 [soap2rest] ...share/lua/5.1/kong/plugins/soap2rest/request_handler.lua:379: attempt to index field 'operations' (a nil value), client: 172.20.0.1, server: kong, request: "POST /ws HTTP/1.1", host: "localhost:8000" kong | 2022/04/05 09:18:18 [error] 25#0: *41558 connect() failed (111: Connection refused) while connecting to upstream, client: 172.20.0.1, server: kong, request: "POST /ws HTTP/1.1", upstream: "http://127.0.0.1:8080/ws", host: "localhost:8000" kong | 2022/04/05 09:18:18 [error] 25#0: *41558 connect() failed (111: Connection refused) while connecting to upstream, client: 172.20.0.1, server: kong, request: "POST /ws HTTP/1.1", upstream: "http://127.0.0.1:8080/ws", host: "localhost:8000" kong | 2022/04/05 09:18:18 [error] 25#0: *41558 connect() failed (111: Connection refused) while connecting to upstream, client: 172.20.0.1, server: kong, request: "POST /ws HTTP/1.1", upstream: "http://127.0.0.1:8080/ws", host: "localhost:8000" kong | 2022/04/05 09:18:18 [error] 25#0: *41558 connect() failed (111: Connection refused) while connecting to upstream, client: 172.20.0.1, server: kong, request: "POST /ws HTTP/1.1", upstream: "http://127.0.0.1:8080/ws", host: "localhost:8000" kong | 2022/04/05 09:18:18 [error] 25#0: *41558 connect() failed (111: Connection refused) while connecting to upstream, client: 172.20.0.1, server: kong, request: "POST /ws HTTP/1.1", upstream: "http://127.0.0.1:8080/ws", host: "localhost:8000" kong | 2022/04/05 09:18:18 [error] 25#0: *41558 connect() failed (111: Connection refused) while connecting to upstream, client: 172.20.0.1, server: kong, request: "POST /ws HTTP/1.1", upstream: "http://127.0.0.1:8080/ws", host: "localhost:8000" kong | 2022/04/05 09:18:18 [error] 25#0: *41558 attempt to set status 502 via ngx.exit after sending out the response status 500, client: 172.20.0.1, server: kong, request: "POST /ws HTTP/1.1", upstream: "http://127.0.0.1:8080/ws", host: "localhost:8000" kong | 2022/04/05 09:18:18 [error] 25#0: *41558 failed to run body_filter_by_lua*: /usr/local/share/lua/5.1/kong/plugins/soap2rest/handler.lua:176: attempt to concatenate local 'RequestAction' (a nil value) kong | stack traceback: kong | /usr/local/share/lua/5.1/kong/plugins/soap2rest/handler.lua:176: in function </usr/local/share/lua/5.1/kong/plugins/soap2rest/handler.lua:135> kong | /usr/local/share/lua/5.1/kong/init.lua:264: in function 'execute_plugins_iterator' kong | /usr/local/share/lua/5.1/kong/init.lua:1167: in function 'body_filter' kong | body_filter_by_lua:2: in main chunk, client: 172.20.0.1, server: kong, request: "POST /ws HTTP/1.1", upstream: "http://127.0.0.1:8080/ws", host: "localhost:8000" kong | 172.20.0.1 - - [05/Apr/2022:09:18:18 +0000] "POST /ws HTTP/1.1" 500 0 "-" "curl/7.64.1"

DanielKraft commented 2 years ago

The plugin is designed to simulate a SOAP API by converting a SOAP request into a REST request. The plugin is designed to make new REST APIs accessible to lagacy systems that can only communicate via SOAP. If you have already developed both APIs you do not need the plugin.

adivardhansingh commented 2 years ago

@DanielKraft The plugin says: Redirect a SOAP request to a REST API.

Does it not mean that the plugin will accept a SOAP request, and redirect to the corresponding REST API, and return the response (converting rest response into soap)?

adivardhansingh commented 2 years ago

@DanielKraft If you have already developed both APIs you do not need the plugin.

I want to STOP using the SOAP API.

The legacy system uses a SOAP API, let us assume there are new REST endpoints (similar to SOAP) that should be used instead. Without changing the legacy system's client code (which makes the API call), can't we use this plugin to do the transformation?

adivardhansingh commented 2 years ago

In short, I want to stop using the legacy SOAP API and use the REST API instead.

But the client wants to continue to make the SOAP call.

DanielKraft commented 2 years ago

Sorry, I have misunderstood you.

As far as I understand the logs, the failure is because your WSDL could not be parsed. The plugin could not find a definition tag in the WSDL. Since the parsing of the WSDL already fails (first line of your logs), the parsing of the OpenApi also fails (second line of your logs) and thus no SOAP requests can be processed by the plugin (from the third line of your logs).

Here is an example of a WSDL as expected by the parser: https://github.com/adessoSE/kong-plugin-soap2rest/blob/master/spec/soap2rest/resources/test.wsdl

We developed the plugin at that time to work as well as possible with our WSDLs, so it can happen that especially the parsing of WSDLs or OpenAPIs fails.

adivardhansingh commented 2 years ago

@DanielKraft There is a definition tag in the WSDL, which I am using (generated by SOAP service)

<wsdl:definitions xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:sch="http://localhost/springsoap/gen"
    xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:tns="http://localhost/springsoap/gen" targetNamespace="http://localhost/springsoap/gen">
    <wsdl:types>

Would this not work with the plugin?

DanielKraft commented 2 years ago

Is the definitions tag inside another tag or at the top level?

As you can see in the following code snippet, the definitions tag must not be encapsulated in another tag. https://github.com/adessoSE/kong-plugin-soap2rest/blob/938d5519c34ad8f0dda8d70f43703a5e35746a7a/kong/plugins/soap2rest/wsdl_handler.lua#L228

If this is not the case, I'm afraid I can't think of anything else it could be?

adivardhansingh commented 2 years ago

@DanielKraft No, the definitions tag is not inside any other tag.

Below is my wsdl file:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<wsdl:definitions xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:sch="http://localhost/springsoap/gen"
    xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:tns="http://localhost/springsoap/gen" targetNamespace="http://localhost/springsoap/gen">
    <wsdl:types>
        <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified" targetNamespace="http://localhost/springsoap/gen">

            <xs:element name="getCountryRequest">
                <xs:complexType>
                    <xs:sequence>
                        <xs:element name="name" type="xs:string" />
                    </xs:sequence>
                </xs:complexType>
            </xs:element>

            <xs:element name="getCountryResponse">
                <xs:complexType>
                    <xs:sequence>
                        <xs:element name="country" type="tns:country" />
                    </xs:sequence>
                </xs:complexType>
            </xs:element>

            <xs:complexType name="country">
                <xs:sequence>
                    <xs:element name="name" type="xs:string" />
                    <xs:element name="population" type="xs:int" />
                    <xs:element name="capital" type="xs:string" />
                    <xs:element name="currency" type="tns:currency" />
                </xs:sequence>
            </xs:complexType>

            <xs:simpleType name="currency">
                <xs:restriction base="xs:string">
                    <xs:enumeration value="GBP" />
                    <xs:enumeration value="EUR" />
                    <xs:enumeration value="PLN" />
                </xs:restriction>
            </xs:simpleType>
        </xs:schema>
    </wsdl:types>
    <wsdl:message name="getCountryResponse">
        <wsdl:part element="tns:getCountryResponse" name="getCountryResponse">
        </wsdl:part>
    </wsdl:message>
    <wsdl:message name="getCountryRequest">
        <wsdl:part element="tns:getCountryRequest" name="getCountryRequest">
        </wsdl:part>
    </wsdl:message>
    <wsdl:portType name="CountriesPort">
        <wsdl:operation name="getCountry">
            <wsdl:input message="tns:getCountryRequest" name="getCountryRequest">
            </wsdl:input>
            <wsdl:output message="tns:getCountryResponse" name="getCountryResponse">
            </wsdl:output>
        </wsdl:operation>
    </wsdl:portType>
    <wsdl:binding name="CountriesPortSoap11" type="tns:CountriesPort">
        <soap:binding style="document" transport="http://schemas.xmlsoap.org/soap/http" />
        <wsdl:operation name="getCountry">
            <soap:operation soapAction="" />
            <wsdl:input name="getCountryRequest">
                <soap:body use="literal" />
            </wsdl:input>
            <wsdl:output name="getCountryResponse">
                <soap:body use="literal" />
            </wsdl:output>
        </wsdl:operation>
    </wsdl:binding>
    <wsdl:service name="CountriesPortService">
        <wsdl:port binding="tns:CountriesPortSoap11" name="CountriesPortSoap11">
            <soap:address location="http://localhost:8080/ws" />
        </wsdl:port>
    </wsdl:service>
</wsdl:definitions>
DanielKraft commented 2 years ago

I implemented a possible solution to your problem in branch #3/Fix_failed_parsing. I hope this helps with your problem.

adivardhansingh commented 2 years ago

@DanielKraft Let me take a pull and use the fixed branch.

But could you also confirm on this comment

Should we not register a SOAP service in Kong as well? How will the SOAP route use the REST service?

DanielKraft commented 2 years ago

Should we not register a SOAP service in Kong as well?

No, you do not need a soap service in Kong.

How will the SOAP route use the REST service?

You have to set config.rest_base_path to the path of the REST route. In your example it should be /products/country.

adivardhansingh commented 2 years ago

@DanielKraft The soap route path[] will remain as /ws, right?

Also, do I need to build the code (make install) after switching to the new branch? I did docker compose up on the new branch, but it had the same error on the same line. I think it did not take the latest fixed changes.

adivardhansingh commented 2 years ago

You have to set config.rest_base_path to the path of the REST route. In your example it should be /products/country

This gives the below error: {"message":"schema violation (config.rest_base_path: must starts and ends with '/')","name":"schema violation","fields":{"config":{"rest_base_path":"must starts and ends with '/'"}},"code":2}

Is it a bug? I had to make it config.rest_base_path=/products/country/

DanielKraft commented 2 years ago

The soap route path[] will remain as /ws, right?

Yes.

Also, do I need to build the code (make install) after switching to the new branch?

You need to remove the old image and rebuild a new one:

docker compose down
docker compose up --build

I had to make it config.rest_base_path=/products/country/

Yes, this is correct.

adivardhansingh commented 2 years ago

@DanielKraft

I have used the new fix as mention on this comment

This still gives me error as below: kong | 2022/04/06 07:38:55 [error] 25#0: *1298 [kong] handler.lua:82 [soap2rest] ...share/lua/5.1/kong/plugins/soap2rest/request_handler.lua:380: attempt to index local 'operation' (a nil value), client: 172.21.0.1, server: kong, request: "POST /ws/ HTTP/1.1", host: "localhost:8000" kong | 2022/04/06 07:38:55 [error] 25#0: *1298 connect() failed (111: Connection refused) while connecting to upstream, client: 172.21.0.1, server: kong, request: "POST /ws/ HTTP/1.1", upstream: "http://127.0.0.1:8089/ws/", host: "localhost:8000" kong | 2022/04/06 07:38:55 [error] 25#0: *1298 connect() failed (111: Connection refused) while connecting to upstream, client: 172.21.0.1, server: kong, request: "POST /ws/ HTTP/1.1", upstream: "http://127.0.0.1:8089/ws/", host: "localhost:8000" kong | 2022/04/06 07:38:55 [error] 25#0: *1298 connect() failed (111: Connection refused) while connecting to upstream, client: 172.21.0.1, server: kong, request: "POST /ws/ HTTP/1.1", upstream: "http://127.0.0.1:8089/ws/", host: "localhost:8000" kong | 2022/04/06 07:38:55 [error] 25#0: *1298 connect() failed (111: Connection refused) while connecting to upstream, client: 172.21.0.1, server: kong, request: "POST /ws/ HTTP/1.1", upstream: "http://127.0.0.1:8089/ws/", host: "localhost:8000" kong | 2022/04/06 07:38:55 [error] 25#0: *1298 connect() failed (111: Connection refused) while connecting to upstream, client: 172.21.0.1, server: kong, request: "POST /ws/ HTTP/1.1", upstream: "http://127.0.0.1:8089/ws/", host: "localhost:8000" kong | 2022/04/06 07:38:55 [error] 25#0: *1298 connect() failed (111: Connection refused) while connecting to upstream, client: 172.21.0.1, server: kong, request: "POST /ws/ HTTP/1.1", upstream: "http://127.0.0.1:8089/ws/", host: "localhost:8000" kong | 2022/04/06 07:38:55 [error] 25#0: *1298 attempt to set status 502 via ngx.exit after sending out the response status 500, client: 172.21.0.1, server: kong, request: "POST /ws/ HTTP/1.1", upstream: "http://127.0.0.1:8089/ws/", host: "localhost:8000" kong | 2022/04/06 07:38:55 [error] 25#0: *1298 failed to run body_filter_by_lua*: /usr/local/share/lua/5.1/kong/plugins/soap2rest/handler.lua:176: attempt to concatenate local 'RequestAction' (a nil value) kong | stack traceback: kong | /usr/local/share/lua/5.1/kong/plugins/soap2rest/handler.lua:176: in function </usr/local/share/lua/5.1/kong/plugins/soap2rest/handler.lua:135> kong | /usr/local/share/lua/5.1/kong/init.lua:264: in function 'execute_plugins_iterator' kong | /usr/local/share/lua/5.1/kong/init.lua:1167: in function 'body_filter' kong | body_filter_by_lua:2: in main chunk, client: 172.21.0.1, server: kong, request: "POST /ws/ HTTP/1.1", upstream: "http://127.0.0.1:8089/ws/", host: "localhost:8000" kong | 172.21.0.1 - - [06/Apr/2022:07:38:55 +0000] "POST /ws/ HTTP/1.1" 500 0 "-" "curl/7.64.1"

adivardhansingh commented 2 years ago

@DanielKraft

Here's my request ` curl -i -X POST --url http://localhost:8000/ws/ \ --header "Content-Type: text/xml" \ --data '<?xml version="1.0" encoding="utf-8"?> <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:gs="http://localhost/springsoap/gen">

          <soapenv:Body>
              <gs:getCountryRequest>
                  <gs:name>Spain</gs:name>
              </gs:getCountryRequest>
          </soapenv:Body>
        </soapenv:Envelope>'

`

Do I need to add operation_mapping=??

DanielKraft commented 2 years ago

I have tried to find a solution to your problem, however it keeps failing because the plugin can't handle your WSDL. This is mainly because our plugin is tailored to a different naming convention.

We generated all our WSDLs from OpenAPI .yaml files using https://github.com/OpenAPITools/openapi-generator. Therefore, we could also assume in the plugin that the WSDLs were read correctly.

So I would recommend you to fork this plugin and customize it to your WSDL.

Again, sorry I can't help you with anything else.

adivardhansingh commented 2 years ago

@DanielKraft

I do not understand Lua and may not be able to fork and update, with limited time in hand. If I generate my WSDL with the above-mentioned tool, and if it fails then, would you be able to support me in that case?

adivardhansingh commented 2 years ago

@DanielKraft

Can you provide an example of config.operation_mapping? config.operation_mapping: rest path must never begin with '/'

You mentioned in the example as "/..." but it seems it is not possible to have it start with '/' So what should exactly be updated here (base path or the endpoint)?

DanielKraft commented 2 years ago

I do not understand Lua and may not be able to fork and update, with limited time in hand. If I generate my WSDL with the above-mentioned tool, and if it fails then, would you be able to support me in that case?

With generated WSDLs the plugin should work.

Can you provide an example of config.operation_mapping? config.operation_mapping: rest path must never begin with '/' You mentioned in the example as "/..." but it seems it is not possible to have it start with '/' So what should exactly be updated here (base path or the endpoint)?

Just remove the starting / then the plugin should accept the input.

adivardhansingh commented 2 years ago

@DanielKraft

With generated WSDLs the plugin should work.

Do you mean generated from the above tool? Because the WSDL I used was also generated, and not manually created. It was generated from the maven build of the SOAP service.

DanielKraft commented 2 years ago

Do you mean generated from the above tool?

Yes

bbekdas commented 2 years ago

@DanielKraft

Thank you for sharing your work. I have a question about a slightly different scenario. In this case, my application expects SOAP requests. I want to use API Manager to receive REST requests from a client (outside world), convert them to SOAP requests with Kong API Gateway, send SOAP message to my application, get a SOAP response back from the application, convert it to REST with Kong API Gateway, and send it back to the client. Do you think I can reuse some of what you built? If not, would you share the general outline I should follow? Thank you in advance.