scylladb / gocql

Package gocql implements a fast and robust ScyllaDB client for the Go programming language.
https://docs.scylladb.com/stable/using-scylla/drivers/cql-drivers/scylla-go-driver.html
BSD 3-Clause "New" or "Revised" License
186 stars 59 forks source link

Make tracing requests shard aware #54

Open kbr- opened 4 years ago

kbr- commented 4 years ago

Currently, all tracing requests use a single connection, the control connection; this connection might or might not be "the right" connection (in terms of which shard it's connected to) for the given session_id (which is the partition key in system_traces.sessions and system_traces.events).

Relevant code in session.go:

func (t *traceWriter) Trace(traceId []byte) {
    var (
        coordinator string
        duration    int
    )
    iter := t.session.control.query(`SELECT coordinator, duration
            FROM system_traces.sessions
            WHERE session_id = ?`, traceId)

and a couple lines below:

    iter = t.session.control.query(`SELECT event_id, activity, source, source_elapsed
            FROM system_traces.events
            WHERE session_id = ?`, traceId)

This could be improved to make the requests token/shard-aware -- pick the right connection from the available pool of connections (as we do for user requests) based on the session_id's token.

Currently tracing queries put unnecessary strain on the cluster as they require cross-shard requests.

slivne commented 4 years ago

Lets check with the java driver if it sends tracing requests only on the control connection - if it load balances them on the regular connections - we should do the same in GoCQL (and in Scylla with shard awareness it will send requests on the correct conenction to the correct shard)

mmatczuk commented 4 years ago

If java driver is similarly broken it should perhaps be fixed too?