Streaming out large data set is not supported in Swan lake data clients

[anupama-pathirage]

Description:

$subject in jdbc and mysql modules. Basically the requirement is to send out large data set in the http response without loading them in the memory at once.

Earlier this was supported by jsonutils:fromTable for the tables. A complete example of 1.2 versions on what we need to achieve is [1]. This is one of the major use case in data clients and we need to support this.

IMO, Converting to xml or json should be supported via a util method or lang lib method for stream data types to achieve this. So it can be used with stream data returned by data clients.

[1] https://ballerina.io/learn/by-example/jdbc-streaming-big-dataset.html

Affected Versions:

Swan lake

[niveathika]

WIP code, Ballerina IO: https://github.com/niveathika/ballerina-lang/commits/streaming-json, Commits: e057b6618f769fd931af205331aab6879ec0f530, 9f5543c5491bd1928db70c4f8b22e9c5f3245519

Jsonutil: https://github.com/niveathika/module-ballerina-jsonutils/tree/streaming

[niveathika]

Testing ballerina code,

import ballerina/http;
import ballerina/sql;
import ballerinax/mysql;
import ballerina/jsonutils;

mysql:Client testDB = checkpanic new ("localhost", "root", "root",
        "testdb", 3306);
@http:ServiceConfig {
    basePath: "/stream"
}
service dataService on new http:Listener(9090) {

    @http:ResourceConfig {
        methods: ["GET"],
        path: "/"
    }
    resource function getData(http:Caller caller, http:Request req) {
        http:Response res = new;
        stream<record {}, sql:Error> selectRet = testDB->query("SELECT * FROM Data limit 5", ());

       json jsonConversionRet = jsonutils:fromStream(selectRet);
        res.setPayload(<@untainted> jsonConversionRet);

        var respondRet = checkpanic caller->respond(res);
        // if (respondRet is error) {
        //     io:println("Sending response failed", respondRet);
        // }
    }

}

The above code is not working as of now, since streams exit after one element access. Most probably this is because of the ballerina native function call using Scheduler is failing due to Async function call. Need further debugging by passing different streams

[niveathika]

This is paused from implementation due to other priority issues for now