Create a functioning omnisci database proxy for observable

[xmnlab]

Request (General goal)

Describe the general goal for this request. What is the high level objective of the work? What is the outcome? What does success look like?

Create a functioning omnisci database proxy for observable https://github.com/observablehq/database-proxy

We need to clone one of the existing ones for mysql, postgres, snowflake that in the lib directory in that repository and create an omnisci one that wraps our JS (MapdConnector) API. The goal is to allow observable notebooks to talk to private omnisci servers.

Some extra information https://github.com/observablehq/database-proxy/issues/16

Stakeholders

Who are the stakeholders for the different pieces that might need to clarify things

@mflaxman10

Technical Specification

Describe all the items needed to conclude this task What should the final state look like?

Input

Is there any new input data? What is its format? (e.g. data object, data file, etc) What is the fields and types? What does each field mean? Where this data would come from?

Output

Where this data should be sent? What is its format? (e.g. data object, data file, etc) What is the fields and types? What does each field mean? Where this data would come from?

Processing

Does it need any specific kind of calculation? Which one? What is the desired workflow for the functionality?

References

Add any reference resource that can help to conclude this task

[xmnlab]

some initial work here https://github.com/xmnlab/database-proxy/tree/add-support-to-omnisci

[xmnlab]

Current status:

The observable database-proxy doesn't have any documentation or tests (at least I didn't find anything), it complicates the implementation of a new backend.

Basically, the code would be very simple, we just need to create a function that receives a URL, opens a connection to the database and returns a function that will receive a response and request object with the sql to request to the database and it will populate the response with the data.

But, as the code doesn't have any docstring, it is taking some time to understand how it is populating the response object. Also, it took some time to test the postgres backend (used as reference), it needed some modifications to have it working locally.

Now, I managed to have postgres backend working locally and I am investigating the output result. It seems it uses micro response and request objects and maybe it creates automatically the output using the response object.

In the past I had some problems to include mapd-connector to database-proxy that, for some reason, was returning a buffer instead of a result. Now it seems it is working, but sometimes it has some issues related to apache-arrow.

There is an initial code and test for omniscidb, but it is still missing the response piece (query method).

So next step would be to understand the response piece of the code and implement it to the omniscidb backend.

[xmnlab]

cc @mflaxman10