search

pyr / cyanite

cyanite stores your metrics
http://cyanite.io
Other
446 stars 79 forks source link

Prototype new scheduling system #237

Closed ifesdjeen closed 7 years ago

ifesdjeen commented 8 years ago

The biggest problems with event processing we faced so far is that the consumer queues get overwhelmed and start pushing the writer queue way behind.

This patch implements a micro-scheduling system (currently extremely primitive), that's using queue-per-core that polls and processes writer flushes from every worker process and "regular" digested events (currently in non-numa-aware fashion & without thread affinity) via single-producer-single-consumer queues.

For example, 4 workers are started, each worker has an affine single producer (netty event loop) single consumer (worker thread) queue. It polls events from it and ingests them.

Every worker, prior to polling it's queue, checks if there are any pending flushes / database writes. If there are any, it submits them. Results are collected asynchronously.

This will help us to move at least part of cyanite that's related to writes to more predictable / controllable env. Also, we can further improve it, make it numa-aware and make scheduling and back-pressure smarter.

tehlers320 commented 8 years ago

My configuration is a bit weird but i think it highlights a problem. I have cyanite seperated into "insert" and "query" pids

The query and insert pids are both churning on CPU with this merge.

top - 16:38:53 up 2 days, 19:15,  1 user,  load average: 21.37, 19.91, 10.72
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
Cpu(s): 95.6%us,  0.9%sy,  0.0%ni,  3.4%id,  0.0%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  30871276k total,  7473708k used, 23397568k free,   156720k buffers
Swap:        0k total,        0k used,        0k free,   482628k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                       
 35871 root      20   0 6654m 1.0g  17m S 297.9  3.4  11:30.82 java

Ill try and get jmx connected and take a look

tehlers320 commented 8 years ago

Cyanite-query:

screen shot 2016-08-22 at 11 46 02 am

Cyanite-insert:

screen shot 2016-08-22 at 11 42 15 am
ifesdjeen commented 8 years ago

@tehlers320 they should churn more idle CPU. I could insert micro-sleeps for better CPU consumption, but that'd also mean that most (or all) benefits of avoiding context-switches would be negative.

The general idea was to have a spinning scheduler, which means that we off-thread all jobs, but don't interrupt them in-between. General idea is to get 1 step closer to thread-per-core, but for that we'd need to do more work.

My measurements show that we're getting better performance generally, at cost of spinning over each queue.

I'll dig into reflector problems though. Not sure why we're getting so much related CPU spin on that...

ifesdjeen commented 7 years ago

I'm going to fix the tests and merge that one if nobody's against.

pyr commented 7 years ago

sorry, didn't react fast enough.

ifesdjeen commented 7 years ago

:) no problem! I assumed nobody was against it so merged it. Hope it was ok.