search

rqlite / gorqlite

A Go client for rqlite, the distributed database built on SQLite
MIT License
139 stars 34 forks source link

Is it possible to use gorqlite without direct connectivity to nodes? #38

Closed jtackaberry closed 4 months ago

jtackaberry commented 4 months ago

Most of my client usage with rqlite has been using the HTTP API directly, mostly in Python (not using pyrqlite). I decided to port my client to Go, and figured I would take gorqlite out for a spin.

As you know, I host rqlite in Kubernetes, but while developing the client app, it runs from a VM outside the K8s cluster. It connects to the K8s service for rqlite, and relies on K8s internal load balancing. rqlite has always been quite well behaved with this model, transparently proxying queries and writes to the leader as needed.

Rather than submitting queries/writes to the endpoint passed in gorqlite.Open(), gorqlite just seems to use that endpoint to discover the cluster topology, and then connects to the API address advertised by each node. The problem in my case is that the nodes aren't directly accessible -- or indeed can't even be resolved in DNS -- by things outside the K8s cluster. So I get the following:

panic: tried all peers unsuccessfully. here are the results:
   peer #0: http://rqlite-1.rqlite-headless.data.svc.cluster.local:4001/db/execute?timings&level=weak&transaction failed due to Post "http://rqlite-1.rqlite-headless.data.svc.cluster.local:4001/db/execute?timings&level=weak&transaction": dial tcp: lookup rqlite-1.rqlite-headless.data.svc.cluster.local: Temporary failure in name resolution
   peer #1: http://rqlite-2.rqlite-headless.data.svc.cluster.local:4001/db/execute?timings&level=weak&transaction failed due to Post "http://rqlite-2.rqlite-headless.data.svc.cluster.local:4001/db/execute?timings&level=weak&transaction": dial tcp: lookup rqlite-2.rqlite-headless.data.svc.cluster.local: Temporary failure in name resolution
   peer #2: http://rqlite-0.rqlite-headless.data.svc.cluster.local:4001/db/execute?timings&level=weak&transaction failed due to Post "http://rqlite-0.rqlite-headless.data.svc.cluster.local:4001/db/execute?timings&level=weak&transaction": dial tcp: lookup rqlite-0.rqlite-headless.data.svc.cluster.local: Temporary failure in name resolution

Is it possible to optionally disable this implicit discovery with gorqlite and have it use the endpoint passed to Open() for statements?

otoolep commented 4 months ago

Yeah, you need to "disable cluster discovery". An example of how to do this:

`conn, err := gorqlite.Open("http://localhost:14001?disableClusterDiscovery=true")`

From the README:

When cluster discovery is disabled, only the provided URL will be used to communicate with the API instead of discovering the leader and peers and retrying failed requests with different peers. This is helpful when using a Kubernetes service to handle the load balancing of the requests across healthy nodes.

otoolep commented 4 months ago

Maybe you can close the issue if this meets your needs.

jtackaberry commented 4 months ago

Huh, staring me in the face the whole time, can't believe I missed that. Thanks Philip.

I do find the fact that it's an endpoint url query param to be a bit surprising as my intuition is that'd be used to influence server side behavior rather than client side. I looked through the API docs looking for indications in function signatures or things like options structs, but obviously I was looking in the wrong place. 🤦