search

wso2 / api-manager

All issues, tasks, improvements and new features of WSO2 API Manager
Apache License 2.0
34 stars 8 forks source link

Add Custom Backend for APIs #3007

Open BLasan opened 2 months ago

BLasan commented 2 months ago

Problem

Some API developers want to use connectors within the api manager to implement mediation logic as they do not want to use an additional MI deployment if their use cases are simple. Within the API Manager, currently, we do not have a construct that will allow us to facilitate this requirement.

Solution

Introduce a way to define a Custom Backend for an API which gets added to the api.xml file of the API.

Affected Component

APIM

Version

No response

Implementation

No response

Related Issues

No response

Suggested Labels

No response

BLasan commented 1 week ago

Please find the below attachment related to the issue. https://docs.google.com/document/d/1DxjI4f-XIwvr414PkT0Mo7C4vuJT_BCGzCidEKwSdF0/edit

BLasan commented 1 week ago

[Update 1]

As per the initial discussion, following was the proposed solution.

  1. Introduce a new tab under Endpoints called Custom Sequence to upload a sequence.
  2. Implement a new REST API to update the API with Custom Sequence provided. Following is the REST API format. 
      requestBody:
    content:
      multipart/form-data:
        schema:
          properties:
            sequence:
              type: string
              description: The sequence that needs to be uploaded.
              format: binary
            type:
              type: string
              description: Type of the Endpoint
            apiData:
              type: string
              description: API Data to be updated
  3. Update velocity template of the API to override the existing configurations with Custom Backend provided.
  4. This will update the api.xml provided with the custom backend without storing the provided sequence in sequences directory of the APIM pack. This might affect the reeadabiility of the api.xml.
  5. On update, the endpointConfiguration object will look as follows. 
    endpointConfig:
    endpoint_type: custom_sequence
    sequence: FILE_NAME
BLasan commented 1 week ago

[Update 2]

As per the second suggestion, changed the velocity template logic to refer the sequence file provided. Following is the approached followed.

  1. Upload the file via the REST API and store under sequences directory which is used to load all sequences by synapse.
  2. Velocity template refers to the Sequence key generated by the code which looks as below. 
                #if( $endpoint_config.get("endpoint_type") == 'custom_sequence' )
                #set( $sequenceKey = $endpoint_config.get("sequence_key") )
                <sequence key="$sequenceKey"/>
  3. This approach does not contain any tables to store uploaded custom backends. Only the API Artifact contains all necessary configurations of the endpoint.
BLasan commented 1 week ago

[Update 3]

Changed the REST API to support add, delete, get operations of the custom sequence as the UI requirement. Followings are the changes introduced.

  1. Changed the REST API to only support Custom Backend upload. Following is the REST API format. 
      requestBody:
    content:
      multipart/form-data:
        schema:
          properties:
            sequence:
              type: string
              description: The sequence that needs to be uploaded.
              format: binary
            type:
              type: string
              description: Type of the Endpoint
  2. Introduced a new table AM_API_CUSTOM_BACKEND to store sequence related data. Following is the proposed schema for the table. 
    CREATE TABLE IF NOT EXISTS AM_API_CUSTOM_BACKEND (
ID INTEGER NOT NULL AUTO_INCREMENT,
API_UUID VARCHAR(256) NOT NULL,
REVISION_UUID VARCHAR(256) DEFAULT NULL,
SEQUENCE MEDIUMBLOB,
NAME VARCHAR(256) NOT NULL,
TYPE VARCHAR(120) NOT NULL,
PRIMARY KEY (ID),
FOREIGN KEY (API_UUID) REFERENCES AM_API(API_UUID) ON DELETE CASCADE
)ENGINE INNODB;
  3. New REST API supports the following operations related to the Custom Backend.
  1. When retrieving the API Artifact, we need to retrieve the corresponding sequence from the AM_API_CUSTOM_BACKEND table using API_UUID and REVISION_ID.
BLasan commented 1 week ago

[Update 4]

  1. Changed the schema of the new table as follows.

    CREATE TABLE IF NOT EXISTS AM_API_CUSTOM_BACKEND (
ID VARCHAR(60) NOT NULL,
API_UUID VARCHAR(256) NOT NULL,
REVISION_UUID VARCHAR(256) DEFAULT NULL,
SEQUENCE MEDIUMBLOB,
NAME VARCHAR(256) NOT NULL,
TYPE VARCHAR(120) NOT NULL,
PRIMARY KEY (ID),
FOREIGN KEY (API_UUID) REFERENCES AM_API(API_UUID) ON DELETE CASCADE
)ENGINE INNODB;

  2. Name of the Custom Backend should be in the following format. <API_UUID>-<ENDPOINT_TYPE>

  3. Issue with the current table schema.

    • Cannot define a PRIMARY KEY as same policy ID will be replicated in the table on a revision deployment.
    • Cannot define a composite primary key as REVISION_UUID can be a null

  4. Solutions for the schema issue.

    • Make REVISION_UUID not null and make REVISION_UUID = "1" for the undeployed version of the API and have a composite primary key (REVISION_UUID, ID, API_UUID)
    • Introduce two tables. One to store sequence data (this will make a Custom Backend globally available) another one to map Custom Backend to API & Revision.
BLasan commented 1 week ago

[Update 5]

  1. Changed the endpointConfig object structure to the following as Custom Backends can be added to both Production and Sandbox endpoints. 
    endpointConfig: {
"endpoint_type": "custom_backend",
"production": {
     "sequence_id": xxxxx,
     "sequence_name": xxxxxx,
     "type": "PRODUCTION"
},
"sandbox": {
     "sequence_id": xxxxx,
     "sequence_name": xxxxxx,
     "type": "SANDBOX"
}
}

2 Implemented get, delete REST APIs which fetches and deletes the Custom Backends from the API_UUID, BACKEND_UUID, TYPE and REVISION_UUID

  1. Made default value as "0" for the REVISION_UUID. Also, new composite primary key is (REVISION_UUID, API_UUID, TYPE and ID)
BLasan commented 1 week ago

[Update 6]:

  1. Tested Custom Backend upload, Get operations
  2. Working on revision deployment. Fixing minor database issues.

Draft PR - https://github.com/wso2/carbon-apimgt/pull/12524 Updated REST API - https://github.com/wso2/carbon-apimgt/pull/12524/files#diff-d8623bfc39bc03ceb819b7dff3e7cb16254f9eb97238c795f40d3b025a4a3747R491

BLasan commented 1 week ago

[Update 7]:

  1. Tested the following flows against the new feature and working without an issue. (Tested for MySQL 8)

    • API creation
    • API deletion
    • Revision Deployment
    • Revision Restore
    • Revision Deletion
    • Changing Endpoint Configurations
    • API Invocation
    • API Import via APICTL

  2. New REST API changes can be found via the following links

  3. Tasks completed

    • REST APIs for Custom Backend
    • Revision related operations
    • API Velocity Template support for Custom Backend
    • API related operations (API Creation, Deletion)
    • APICTL support for Custom Backend (Import & Export)

  4. Tasks remaining

    • UI for Custom Backend upload and Endpoint Configuration change
    • Test APICTL export API
    • Analytics related changes
BLasan commented 1 week ago

All scenarios to be tested

  1. API Creation
  2. Changing API Endpoint
  3. Add Custom Backend, Delete Custom Backend, Change Endpoint Config, Download Sequence
  4. API Revision Deployment, Undeploy revision, Delete revision, Restore revision
  5. API Delete
  6. API Invocation
  7. APICTL Import, Export, Update
  8. APIProduct creation, deploy, invoke, import & export