Support For Apache Pinot

evidence-dev / evidence

Business intelligence as code: build fast, interactive data visualizations in pure SQL and markdown

https://evidence.dev

MIT License

3.73k stars 182 forks source link

Support For Apache Pinot #1153

Open hdulay opened 10 months ago

hdulay commented 10 months ago

What do you use this database for?

Real-Time analytical queries. Apache Pinot is an RTOLAP database (real-time online analytical process).

Are there any popular and well-maintained clients for this database?

https://socket.dev/npm/package/pinot-client

Are you interested in contributing this database connector?

yes, I'm new to this project. Any instructions for building connectors would be great. Thanks

archiewood commented 10 months ago

Hey @hdulay ,

Thanks for the suggestion!

We're currently reworking our project to allow "pluggable" connectors, and I think pinot is a really good candidate for this.

Context: currently, when we add connectors, we have to add them to the core library. This makes maintaining them a pain, as we (Evidence team) do not always have instances of these DBs to test. This new architecture will mean it's much easier for people to contribute and maintain these without being held back by the bandwidth of our team!

If you're open to it, I can update here once the new architecture is live (hopefully pretty soon!)

hdulay commented 10 months ago

Yes I look forward to it. Thanks

cyrilou242 commented 4 months ago

Hey @archiewood I'm also really interested in a Pinot connector! Were you able to make progress on the plugin architecture?

archiewood commented 4 months ago

Hey sorry for the delay!

We have totally re-architected Evidence since this issue was opened, and Evidence is indeed now "pluggable", meaning you can write your own connectors, publish them, and use them with Evidence.

Here are some resources that may help:

Docs on creating a plugin: https://docs.evidence.dev/plugins/creating-a-plugin/datasources
Template for a connector plugin: https://github.com/evidence-dev/datasource-template
An example that I spun up for pulling stats from the npm API: https://github.com/archiewood/npm-stats

If you do write a Pinot plugin,

the suggested package name is evidence-connector-[plugin] eg evidence-connector-pinot
the docs are new, so let us know if there is extra information you need
share it on the Evidence community, and we'll feature it!

cyrilou242 commented 4 months ago

Awesome thanks a lot @archiewood! I'll have a look at this.