[PERF] 64% of RSCoordinator FT.SEARCH command time is associated with _FT.SEARCH command send to shards + receive and reply parsing

[filipecosta90]

Sharing the main and background thread flame charts with the #303 already solved so we focus on the new real hotspots. The first thing to notice is that the coordinator thread is the first bottleneck ( reaching 100% CPU usage faster than the main thread ).

• Main thread: https://s3.amazonaws.com/benchmarks.redislabs/internal-tasks/perf-186/RSCoordinator/json.25s.main.redis.svg
• Background thread: https://s3.amazonaws.com/benchmarks.redislabs/internal-tasks/perf-186/RSCoordinator/json.25s.background.redis.svg

test setup

Considering a 25 shards setup we see:

• Each individual shard reply to RSCoordinator ( _FT.SEARCH ) takes around 55 micros
• The total time for FT.SEARCH is 420 micros
• If we trace the time in which we send the individual _FT.SEARCH requests until we parse them we see it represents 64% of the wall clock time (270 micros out of total 420 micros ( FT.SEARCH)). Meaning 64% of wall clock time is on this block of reply sending + receiving shard replies + parsing. Meaning this is the part that we should target to optimize

tracing the network + parsing ( hiredis )

@usecs_rediscmds_detailed: 
[261, 271)          4851 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@   |
[271, 281)          5109 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[281, 291)           310 |@@@                                                 |
[291, 301)           281 |@@                                                  |
[301, 311)           135 |@                                                   |
[311, 321)           221 |@@                                                  |
[321, 331)           232 |@@                                                  |
[331, 341)            99 |@                                                   |
[341, 351)            30 |                                                    |
[351, 361)            10 |                                                    |
[361, 371)             6 |                                                    |