Open ivansenic opened 2 years ago
@mpenick Could you have a look on this please?
Showcasing differences in tracing examples:
Executor:
Direct executor:
Reconfirmation needed after #2008 is merged.
The theory why is this happening that the DescribeKeyspace
calls are not offloaded to another thread, as for the CQL querying. This is most likely the reason for that strange timing. @mpenick and I think that it would be the best:
directExecutor()
This should be done for all the gRPC call that are not doing the CQL query.
We were doing some initial performance testing with the Docs API V2 and once of the most disturbing things for us so far was the metrics and the tracing reports that the
DescribeKeyspace
method on the Bridge was taking too much time from the client perspective.In the current state, we observed this metrics that showcase that the method has:
2.87ms
7.10ms
It was quite disturbing to see a method that should be a sub-millisecond method to be recorded as the
7ms
on the client side, with over4ms
overhead from the server-side on the same machine. We started exploring and we, most likely by accident, tackled the.directExecutor()
usage on theBridgeImpl
..We changed this to a dedicated executor with 16 threads:
.executor(Executors.newScheduledThreadPool(16))
, and got quite the opposite times:0.73ms
1.28ms
It's really unclear to me why is this and if usage of the direct executor is justified, but we also observed improved response times for the read and write scenarios in V2:
We should analyze if the direct executor should be used here or not.
Note that all data is from write and read docs scenario using 60
nb
threads on a single machine.