cube-js / cube

📊 Cube — The Semantic Layer for Building Data Applications
https://cube.dev
Other
17.94k stars 1.78k forks source link

Does Cube.js support intelligent querying of a data source with multiple read replicas? #8531

Closed BeautyFades closed 1 week ago

BeautyFades commented 3 months ago

Hello there! I'm not a network engineer nor do I have plenty of experience developing production-level high performance read-replicas, so bear with me...

Problem I have an use-case for Cube.js in which it is used on top of an OLTP datastore (Postgres in AWS RDS) instead of the usual/expected OLAP warehouse.

This Postgres instance, due to performance and scalability constraints is broken into N read replicas, each having their own connection details. My question is: does Cube.js support out of the box source load balancing, or any sort of multiple connection logic that allows the same cube/query to be routed to potentially different data sources? Or can I set up a proxy or load balancer (RDS Proxy for instance) and have Cube connect to the proxy, and let it do the load balancing for me?

In the docs I never saw anything of this kind mentioned, nor did I see support for proxy connections. Has anyone ever faced this use-case before and could share details?

Thanks!

igorlukanin commented 2 months ago

Hi @BeautyFades 👋

does Cube.js support out of the box source load balancing [of data sources]

No, nothing of this kind of built into Cube.

or any sort of multiple connection logic that allows the same cube/query to be routed to potentially different data sources?

You can implement multitenancy and setup driver_factory in a way that different tenants have different credentials for the same data source. Then, you can manually randomize/round-robin through tenants, effectively spreading the load over Postgres connections.

Or can I set up a proxy or load balancer (RDS Proxy for instance) and have Cube connect to the proxy, and let it do the load balancing for me?

If Cube's Postgres driver can connect to it, then sure. I believe you can just test to see if this setup works for you.