[OpenAPI CLI] Mock Endpoint and Test Generation

[niveathika]

Description: This proposal introduces two new features to the OpenAPI to Ballerina CLI tool:

1. Generation of mock endpoints.
2. Automated test case creation.

Currently, the tool's primary function is to generate Ballerina code from OpenAPI specifications.

The proposed features aim to enhance the developer experience by:

• Facilitating the generation of mock endpoints for preliminary testing.
• Automating the creation of initial test cases based on the OpenAPI contract.

Additional Benefits

1. Scenario Testing: This allows for comprehensive scenario testing, ensuring the application's robustness and resilience.

2. Reduced Dependencies: These features enable developers to work independently on their components, minimizing dependencies and eliminating bottlenecks in the development process.

3. Enhanced Test Coverage: Automated test generation based on the OpenAPI contract ensures extensive test coverage, reducing the likelihood of unnoticed bugs.

4. Efficient API Contract Validation: Developers can validate the API contract early and continuously during development, ensuring that the documentation and implementation are aligned.

[niveathika]

Explored Swagger Codegen, OpenAPI Generator, Dredd, OpenAPI Testers, and the fancy ones like EvoMaster, Karate, and Tcases.

What I Found:

1. Code Generators: Swagger and OpenAPI Generator are good for code but a bit lackluster on test cases. They throw in basic service calls and leave assertions to developers.

2. Testing Tools: Dredd and OpenAPI Testers check if your API follows rules but don't guide much on creating meaningful test cases.

3. Specialized Solutions: EvoMaster shines with AI algorithms.

Please refer OpenAPI specification validation for more research into existing solutions.

Generating test cases from OpenAPI specs is tricky. Tools are cool but fall short. AI is an option, but it comes with a hefty price tag.

My suggestion? Hold off on the test case generation feature for now. Wait until the need outweighs the cost. Keep it practical.

[niveathika]

I will be looking into mock services next.

[niveathika]

I've explored two methods for mocking external endpoints to test our client:

1. Ballerina Mock Functionality:

   • Utilizes Ballerina's built-in mock functionality.
   • No need to start a separate server, handled by Ballerina.
   • Requires modifying the generated client by moving the initialized http:Client to the global level.
   • Limitations:
     • Incompatible with testing if a connector wraps the generated client in another client.
     • Cumbersome for APIs with a large number of paths as we need to used switch statement to return values for each path.
   • Example: MockHttpClient.

2. Bal OpenAPI CLI for Mock Server:

   • Uses the bal openapi CLI to generate a mock server, added as a submodule.
   • The submodule, when imported in tests, starts the server.
   • Higher readability, as the server is essentially a Ballerina service.
   • Ability to return default values and introduce a "try-to-fill" feature for randomizing and returning values.
   • Quick Implementation, Reuses 80% of the CLI code to generate a server for an OpenAPI definition.
   • No restrictions on modules, suitable for testing code with wrapped clients.
   • Drawback: For authenticated servers, additional effort is needed to generate auth servers.
   • Example: OpenAPI Mock Service.

In summary, the second approach leveraging the Bal OpenAPI CLI for generating a mock server appears more flexible, readable, and suitable for diverse testing scenarios. While it requires additional effort for authenticated servers, its advantages in code readability and versatility make it a promising choice.

Note: The progress has been delayed as I've encountered issues with Testerina. #41825 #41821

[niveathika]

After evaluating, we have decided generate the service skeleton from the bal openapi tool and fill it with dummy values for the time being with the other priorities