[Proposal] Map OpenAPI Specification (OAS) to service object type

Summary

OpenAPI Specification (OAS) is a way to describe the service interface of REST APIs. Counterpart of this in Ballerina is service object type. This proposal is to map those two together.

Goals

Map OAS to service object type

Motivation

At the moment we directly map the OAS to the service implementation. Ideally, we should map the OAS to service object type and then associate the service object type to the service implementation. That way it cleans up the mapping between three entities.

OAS (interface) -> Service object type (interface) -> Service implementation

Description

With the above suggestion everything related to the OAS can be associated with service object type. Both, generating Ballerina service object type from OAS as well as generating OAS from the service object type can depend on the above association. This means service implementation does not require any association with OAS related functionalities.

However, for historical reasons we can keep OAS mapping to the service implementation. I think gradually we can remove it if needed. So the associations would look like the below.

The proposed: OAS (interface) -> Service object type (interface) -> Service implementation The current: OAS (interface) -> Service implementation

In order to make this possible. We need an annotation to map the base-path to service object type as the base-path only comes into the picture during the service declaration. This is the main blocker to implement this.

Once the annotation is in place, HTTP compiler plugging can validate it against service declaration.

Interface:

@http:ServiceConfig {
   basePath: "social-media"
}
type SocialMedia service object {
    resource function get users() returns User[]|error;
    resource function get users/[int id]() returns User|UserNotFound|error;
    resource function post users(NewUser newUser) returns http:Created|error;
    resource function delete users/[int id]() returns http:NoContent|error;
    resource function get users/[int id]/posts() returns PostWithMeta[]|UserNotFound|error;
    resource function post users/[int id]/posts(NewPost newPost) returns http:Created|UserNotFound|PostForbidden|error;
};

// user representations
type User record {|
    int id;
    string name;
    @sql:Column {name: "birth_date"}
    time:Date birthDate;
    @sql:Column {name: "mobile_number"}
    string mobileNumber;
|};

public type NewUser record {|
    @constraint:String {
        minLength: 2
    }
    string name;
    time:Date birthDate;
    string mobileNumber;
|};

...

Implementation:

service SocialMedia /social\-media on new http:Listener(9095) {
}

Additionally, we can map other annotations to service type as well such as annotation to add examples, security, HATEOAS, etc. This cleans up service implementation from service interface information.

@http:ServiceConfig {
   basePath: "social-media",
   auth: [
        {
            fileUserStoreConfig: {},
            scopes: ["admin"]
        }
    ] 
}
type SocialMedia service object {

    @http:ResourceConfig {
       responseEx: "
           [
               {
                  "id":1,
                  "name":"ranga",
                  "birthDate":{
                     "year":2024,
                     "month":4,
                     "day":24
                  },
                  "mobileNumber":"+94771234001"
               },
               {
                  "id":2,
                  "name":"ravi",
                  "birthDate":{
                     "year":2024,
                     "month":4,
                     "day":24
                  },
                  "mobileNumber":"+94771234001"
               }
        ]"
    }
    resource function get users() returns User[]|error;
    resource function get users/[int id]() returns User|UserNotFound|error;
    resource function post users(NewUser newUser) returns http:Created|error;
    resource function delete users/[int id]() returns http:NoContent|error;
    resource function get users/[int id]/posts() returns PostWithMeta[]|UserNotFound|error;
    resource function post users/[int id]/posts(NewPost newPost) returns http:Created|UserNotFound|PostForbidden|error;
};

This also means we no longer need OpenAPI validator feature as the complier can validate the service against the service object type.

How is the experience of introducing separate annotations for the service object type, rather than utilizing the service implementation annotation? Because this could help streamline logic and avoid potential conflicts.

Checked the HTTP side support. Please find the below discussion points:

We need to include http:Service to the generated service type, otherwise we can not bind this to a HTTP listener:

@http:ServiceConfig {
basePath: "social-media"
}
type SocialMedia service object {
*http:Service;
resource function get users() returns User[]|error;
resource function get users/[int id]() returns User|UserNotFound|error;
resource function post users(NewUser newUser) returns http:Created|error;
resource function delete users/[int id]() returns http:NoContent|error;
resource function get users/[int id]/posts() returns PostWithMeta[]|UserNotFound|error;
resource function post users/[int id]/posts(NewPost newPost) returns http:Created|UserNotFound|PostForbidden|error;
};

The HTTP compiler plugin (with code modifier to add payload annotation) is not engaged for service object types. So the generated service type cannot be validated by the HTTP compiler for now. Btw since it is generated by the OpenAPI tool and if the tool guarantees the compatibility of the generated service type, this is not an issue.
We can have the basePath in the ServiceConfig annotation. But currently the ServiceConfig annotation is only allowed in service declaration and service objects. If we want to allow it in the service object type then we need to allow it on all types (since there is no specific annotation attachment point for service object types. Referred to the spec: https://ballerina.io/spec/lang/master/#annot-attach-points). So we may have to write a compiler validation to check this annotation usage.
When we have basePath in the service object type, do we enforce it in the service declaration or do we make that path as a default path?
1. Enforcing the base path in the service declaration means that users has to explicitly use the same base path when creating the service declaration. The compiler plugin should validate this.
```
service SocialMedia /social\-media on new http:Listener(8080) {
```
1. Make the default base path as defined in the service object type. Runtime should handle this internally.
```
service SocialMedia on new http:Listener(8080) {
```
  This base path can be also overwritten at the service declaration
```
service SocialMedia /social\-media\-dev on new http:Listener(8080) {
```
Seems like the annotations on the service object type is not visible at runtime. Need to check with @HindujaB. Due to this there is no conflicting annotations error from ballerina lang.
Usability concern: There is no code action to implement all the methods in the service declaration when we use the service type. Opened an issue to support this: https://github.com/ballerina-platform/ballerina-lang/issues/42758
Since we cannot capture the service type annotations in runtime, I have not checked on adding the examples, HATEOAS, security etc. on the ResourceConfig annotation. Will do it after talking to the runtime team

The misising annotations of the service object type at runtime seems to be a bug in the compiler. Created an issue for this: https://github.com/ballerina-platform/ballerina-lang/issues/42760

As per this comment by @MaryamZi : https://github.com/ballerina-platform/ballerina-lang/issues/42760#issuecomment-2116706356, the meta-data such as annotations are not copied to the service constructor. So we need to extract them using the HTTP compiler plugin and modify the service declaration node with them.

@lnash94 @shafreenAnfar @hasithaa @MaryamZi and myself had a discussion regarding the annotation on the service object type and decided to do the following:

Introduce a constant annotation for the values to be validated at compile time such as basePath
Pass the service object type via an annotation to the runtime to get the service object type annotation details

Constant annotation for compiler validation of base path

Since we only need basePath to be validated at the compile time, we can move that from http:ServiceConfig to some openapi tool specific annotation. Since we are passing the most of the openapi details in the openapi:ServiceInfo annotation, I am thinking about passing the base path to that annotation. The openapi:ServiceInfo currently have the following fields:

# Service validation code.
# + contract - OpenAPI contract link
# + tags - OpenAPI tags
# + operations - OpenAPI operations
# + excludeTags - Disable the OpenAPI validator for these tags
# + excludeOperations - Disable the OpenAPI validator for these operations
# + failOnErrors - Enable the OpenAPI validator
# + embed - Enable auto-inject of OpenAPI documentation to current service
# + title - Title for generated OpenAPI contract
# + version - Version for generated OpenAPI contract
public type ServiceInformation record {|
string contract = "";
string[]? tags = [];
string[]? operations = [];
string[]? excludeTags = [];
string[]? excludeOperations = [];
boolean failOnErrors = true;
boolean embed = false;
string title?;
string version?;
|};

But currently the annotation is not defined as constant, but since these information can be passed as constants, we can make this annotation constant and add a basePath field.

Other details from OpenAPI specification using annotation

Other details such as HATEOAS, auth and examples can be passed via the http:ServiceConfig and http:ResourceConfig annotation since they do not need to validated at compile time. In order to get the annotation details at runtime, we will be using the HTTP compiler plugin to add the service type descriptor to http:ServiceConfig annotation.

The service type definition will look like this:

@openapi:ServiceInfo {
    basePath: "social-media"
}
@http:ServiceConfig {
    auth: [
        {
            fileUserStoreConfig: {},
            scopes: ["admin"]
        }
    ]
}
public type Service service object {
    *http:Service;
    ...

    @http:ResourceConfig {
        name: "user",
        linkedTo: [
            {name: "user", method: http:DELETE, relation: "delete-user"},
            {name: "posts", method: http:POST, relation: "create-posts"},
            {name: "posts", method: http:GET, relation: "get-posts"}
        ],
        responseEx: "{
            "id":1,
            "name":"ranga",
            "birthDate":{
               "year":2024,
               "month":4,
               "day":24
            },
            "mobileNumber":"+94771234001"
        }"
    }
    resource function get users/[int id]() returns User|UserNotFound|error;

    ...
};

The service implementation will look like this after code modification: (Assume the service type is imported from a different package: socialMedia)

@http:ServiceConfig {
    serviceType: socialMedia:Service
}
service socialMedia:Service /social\-media on new http:Listener(8080) {
      ...
}

Annotations on the resource parameters

Service type implementation does not ensure the annotation usages in the resource parameter nodes such as @http:Payload, @http:Query, @http:Header and @http:Cache. In such scenarios, what should we do?

Since we do not have compiler validations based on the value of these annotations, we could only consider them at runtime. But we need them to validate the parameters in the service declaration:

resolving conflicts on structured parameters
validate the usage of cache on success responses

The following is the new service type which should be generated from the openapi tool:

@http:ServiceContractConfig {
    basePath: "/socialMedia"
}
@http:ServiceConfig {
    auth: [
        {
            fileUserStoreConfig: {},
            scopes: ["admin"]
        }
    ]
}
public type Service service object {
    *http:ServiceContract;
    @http:ResourceConfig {
        name: "users"
    }
    resource function get users() returns socialMedia:User[]|error;

    @http:ResourceConfig {
        name: "user",
        linkedTo: [
            {name: "user", method: http:DELETE, relation: "delete-user"},
            {name: "posts", method: http:POST, relation: "create-posts"},
            {name: "posts", method: http:GET, relation: "get-posts"}
        ]
    }
    resource function get users/[int id]() returns @http:Payload {mediaType: "application/org+json"} socialMedia:User|socialMedia:UserNotFound|error;

    @http:ResourceConfig {
        name: "users",
        linkedTo: [
            {name: "user", method: http:GET, relation: "get-user"},
            {name: "user", method: http:DELETE, relation: "delete-user"},
            {name: "posts", method: http:POST, relation: "create-posts"},
            {name: "posts", method: http:GET, relation: "get-posts"}
        ]
    }
    resource function post users(socialMedia:NewUser newUser) returns http:Created|error;

    @http:ResourceConfig {
        name: "user"
    }
    resource function delete users/[int id]() returns http:NoContent|error;

    @http:ResourceConfig {
        name: "posts"
    }
    resource function get users/[int id]/posts() returns socialMedia:PostWithMeta[]|socialMedia:UserNotFound|error;

    @http:ResourceConfig {
        name: "posts",
        linkedTo: [
            {name: "posts", method: http:POST, relation: "create-posts"}
        ]
    }
    resource function post users/[int id]/posts(socialMedia:NewPost newPost) returns http:Created|socialMedia:UserNotFound|socialMedia:PostForbidden|error;
};

The implementation will look like this: (The service type is imported via a separate package, the service type can be in the same package as well)

service socialMedia:Service "/social-media" on new http:Listener(8080) {

    resource function get users() returns socialMedia:User[]|error {
        return [];
    }

    resource function get users/[int id]() returns socialMedia:User|socialMedia:UserNotFound|error {
        return {id: 1, name: "John Doe", email: "johndoe@gmail.com"};
    }

    resource function post users(socialMedia:NewUser newUser) returns http:Created|error {
        return http:CREATED;
    }

    resource function delete users/[int id]() returns http:NoContent|error {
        return http:NO_CONTENT;
    }

    resource function get users/[int id]/posts() returns socialMedia:PostWithMeta[]|socialMedia:UserNotFound|error {
        return [];
    }

    resource function post users/[int id]/posts(socialMedia:NewPost newPost) returns http:Created|socialMedia:UserNotFound|socialMedia:PostForbidden|error {
        return http:CREATED;
    }
}

In the above service declaration,

Base path should be same as the one from the service contract type. This validation is done at compile time. Additionally a code action was introduced to add base path.
A code action is introduced to implement all the methods from service contract type
None of the HTTP annotations are allowed, all of them are inferred from the service type
Additional resources which are not defined in the service contract type, are not allowed

IMO, having two different annotations is not good. There is a usability issue here. We seem to be coming up with another annotation-like service contract due to compiler limitations.

@shafreenAnfar @sameerajayasoma @lnash94 and I had a discussion and decided not to have a separate annotation: http:ServiceContractConfig to have the basepath. The base path will be always inferred from the service contract time at runtime and specifying a base path when implementing a service contract type is disallowed. The updated service type and the implementation will look like this:

Service contract type:

@http:ServiceConfig {
    // basePath field can be only provided for service contract types, since
    // it does not have a meaning for other types or declaration
    basePath: "/socialMedia",
    auth: [
        {
            fileUserStoreConfig: {},
            scopes: ["admin"]
        }
    ]
}
public type Service service object {
    *http:ServiceContract;

     ...
};

Service contract implementation:

// No base path
service socialMedia:Service on new http:Listener(8080) {

    ...
}

ballerina-platform / ballerina-library