[Proposal]: Integration Specification Document Generation

[AzeemMuzammil]

Summary

This proposal aims to introduce an automated AI-driven tool that generates comprehensive integration specification documents for Ballerina integrations. This tool will enhance the documentation process by providing a structured, accurate, and detailed overview of integrations, including design diagrams, API details, security protocols, data mapping, and environment setup guides, all triggered via a simple VSCode command or code action.

Goals

• Enable automated generation of detailed integration specification documents from Ballerina code bases.
• Incorporate visual aids like architecture, sequence, and type diagrams automatically in the documentation.
• Simplify the documentation process, making it accessible with the VSCode command or code-action in VSCode.
• Support continuous documentation updates aligned with code changes to ensure real-time accuracy.
• Facilitate better knowledge transfer and onboarding for new team members through standardized documents.
• Reduce the manual effort developers and technical writers require in creating and maintaining integration documentation.

Non-Goals

• The tool will not generate documentation for external services that the integration might consume.
• It will not automate the generation of business logic descriptions.

Motivation

In modern software development, maintaining up-to-date and comprehensive documentation is crucial but often neglected due to the time-intensive nature of writing and updating these documents. For Ballerina integrations, which can be complex and multifaceted, the lack of precise documentation can lead to miscommunication, increased onboarding time for new developers, and potential security risks from misunderstood or improperly implemented integrations. Automating the generation of integration specification documents can drastically reduce these risks and improve efficiency and understanding across development teams.

Description

Definition

The Integration Specification Document Generation tool will use Large Language Models (LLMs) to interpret and document Ballerina integrations. The tool will analyze the source code and associated diagrams generated within VSCode, like sequence and type diagrams, to produce a Markdown (MD) formatted document. This document will serve as a single source of truth for developers, providing a clear and detailed description of the integration's purpose, structure, and operational details.

Proposed Design

The tool will integrate directly into the Ballerina VSCode plugin, allowing developers to invoke the documentation generation process through either a specific VSCode command or a code action within the IDE. The process will leverage LLMs for prompt engineering, where the model will be fed with a pre-defined document structure and the integration's source code and diagrams.

Document Format

The output document will follow a structured Markdown format, ensuring that it is both human-readable and compatible with version control systems. Here's an overview of the expected document sections:

• Overview: General description, repository information, and API base paths.
• Design Diagrams: Automatically generated and embedded diagrams including architecture, sequence, and type diagrams.
• API Overview: Detailed API descriptions, including endpoints, methods, parameters, and sample requests and responses.
• Security and Authentication: Detailed authentication methods and security protocols used within the integration.
• Environment and Deployment: Configuration details for different deployment environments (Dev, Staging, Production).

Ballerina Integration Example: Simple User Management Service

Ballerina Code:

import ballerina/http;

public type User record {
    string userId;
    string userName;
};

service /userManagement on new http:Listener(8080) {
    resource function get user(int userId) returns json {
        if (userId == 1) {
            return { "userId": 1, "name": "John Doe", "email": "john.doe@example.com" };
        } else {
            return { "message": "User not found" };
        }
    }

    resource function post addUser(User user) returns json {
        return { "message": "User added successfully", "userId": user.userId };
    }
}

Integration Specification Document

Simple User Management Service - Organization XYZ

• Authors: John Doe
• Created Date: July 17, 2024
• Updated Date: July 17, 2024
• Version: 1.0.0

Table of Contents

1. Overview
   • Description of the Integration (AI-generated based on the Integration code and the diagrams)
   • Project Repository Info (Extracted from the project source)
   • API Base Path Info (Extracted from the project source)
2. Design Diagrams
   • Architecture Diagram
     • Resource Diagram (From VSCode/langserver APIs)
     • Type Diagram (From VSCode/langserver APIs)
   • Sequence Diagram (From VSCode/langserver APIs)
3. API Overview (Ai-generated using the OpenAPI spec generated for the Integration, integration code, and the diagrams & if possible with the knowledge base of used external services(ie: GitHub))
   • API 1
     • Description of the API
     • Key Functions
     • API Details
       • Path
       • Method
       • Caller EPs
       • Source EPs
       • Headers (With Example)
       • Request Parameters (With Example)
       • Request Body (With Example)
       • Response Status Codes and Bodies (With Examples)
     • Data Mapping
       • Source to Destination Field Mappings
       • Transformations
4. Security and Authentication (Extracted from the codebase and OpenAPI spec)
   • Authentication Method
5. Environment and Deployment (Extracted information from the codebase(Config.toml) and OpenAPI spec)
   • Dev
     • Configs
   • Staging
     • Configs
   • Production
     • Configs

Overview

Description of the Integration

This integration provides a straightforward user management system designed to handle basic operations such as retrieving and adding user data. Implemented in Ballerina, it is built to offer high performance and easy scalability, integrating seamlessly with existing infrastructure.

Project Repository Info

The codebase is maintained on GitHub

API Base Path Info

The base path for all API calls is /userManagement, serving as the entry point for interaction with the user management functionalities.

Design Diagrams

Architecture Diagram

Resource Diagram

Type Diagram

Sequence Diagram

API Overview

Get User

Description of the API

Retrieves details of a user by user ID.

Key Functions

Fetch user data from the system.

API Details

• Path: /userManagement/user
• Method: GET
• Caller EPs: /user/{userId}
• Source EPs: Not applicable
• Headers: None
• Request Parameters: userId (integer)
• Request Body: Not applicable
• Response Status Codes and Bodies:
  • 200 OK: { "userId": 1, "name": "John Doe", "email": "john.doe@example.com" }
  • 404 Not Found: { "message": "User not found" }
• Data Mapping: Direct mapping from request to user data structure.

  Add User

  Description of the API

  Adds a new user to the system.

  Key Functions

  Insert new user data into the system.

  API Details

• Path: /userManagement/addUser
• Method: POST
• Caller EPs: /addUser
• Source EPs: Not applicable
• Headers: None
• Request Parameters: None
• Request Body: { "userId": int, "name": string, "email": string }
• Response Status Codes and Bodies:
  • 200 OK: { "message": "User added successfully", "userId": 123 }
• Data Mapping: Direct mapping from request body to user data structure.

Security and Authentication

Authentication Method

None (for demonstration purposes, real service should include OAuth2 or similar)

Environment and Deployment

• Dev
  • Configs: Localhost server on port 8080.

[AzeemMuzammil]

Closing this as this is transferred to the repo: https://github.com/wso2-enterprise/ballerina-copilot

The issue: https://github.com/wso2-enterprise/ballerina-copilot/issues/201