Proton Ingest REST API supports raw/streaming modes

jovezhong commented 11 months ago

Use case

Today https://docs.timeplus.com/proton-ingest-api expects a special format as the POST payload

{
  "columns": ["id","name"],
  "data": [
    [1,"hello"],
    [2,"world"]
  ]
}

This compact data format can avoid repeating the column names. But this is too unique to Proton, and it makes it hard to integrate other HTTP service/client with Proton.

A few examples:

the awesome sensor app will post JSON object, not in this format
curl -s -H 'Accept: application/json' https://stream.wikimedia.org/v2/stream/recentchange also sends own JSON format

The workaround today is to setup a proxy server to convert the arbitrary JSON payload as the text string in raw

{
  "columns": ["raw"],
  "data": [["TEXT"]]
}

A sample code is

import { routes } from '@stricjs/app';

export function main() {
    const protonIngestEndpoint = `http://${process.env.HOST}:3218/proton/v1/ingest/streams/${process.env.STREAM}`;
    return routes()
        .post('/', c => {
            return c.text().then(a => {
                return fetch(protonIngestEndpoint, {
                    method: "POST",
                    body: `{"columns": ["raw"],"data": [["${a.replaceAll('"', '\\\"')}"]]}`,
                    headers: { "Content-Type": "application/json" },
                }).then(protonResp => new Response('status code ' + protonResp.status));
            });
        })
}

But this solution is not great and also risky. It just tries to replace " to \", but there could be other cases making the TEXT breaks the JSON format.

Describe the solution you'd like

Similar to Neutron REST API, we should support multiple mode. (default remains the current one)

format=raw, assumes there is a raw column
format=streaming Push JSON objects. Each JSON is an event.
format=lines, each line is a row for the raw column
format=compact, the current one/default one

When this is ready, loading data from wikipedia to proton will be as simple as

curl -s -H 'Accept: application/json' https://stream.wikimedia.org/v2/stream/recentchange | while read -r line; do echo "$line" | curl -s -X POST -d @- http://localhost:3218/proton/v1/ingest/streams/wiki?format=raw; done

One thing to discuss, for raw/lines, since the stream schema is fixed, can we auto-create the stream if the stream doesn't exist?

Describe alternatives you've considered

Additional context

jovezhong commented 10 months ago