Closed ifesdjeen closed 7 years ago
My configuration is a bit weird but i think it highlights a problem. I have cyanite seperated into "insert" and "query" pids
The query and insert pids are both churning on CPU with this merge.
top - 16:38:53 up 2 days, 19:15, 1 user, load average: 21.37, 19.91, 10.72
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
Cpu(s): 95.6%us, 0.9%sy, 0.0%ni, 3.4%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 30871276k total, 7473708k used, 23397568k free, 156720k buffers
Swap: 0k total, 0k used, 0k free, 482628k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
35871 root 20 0 6654m 1.0g 17m S 297.9 3.4 11:30.82 java
Ill try and get jmx connected and take a look
Cyanite-query:
Cyanite-insert:
@tehlers320 they should churn more idle CPU. I could insert micro-sleeps for better CPU consumption, but that'd also mean that most (or all) benefits of avoiding context-switches would be negative.
The general idea was to have a spinning scheduler, which means that we off-thread all jobs, but don't interrupt them in-between. General idea is to get 1 step closer to thread-per-core, but for that we'd need to do more work.
My measurements show that we're getting better performance generally, at cost of spinning over each queue.
I'll dig into reflector problems though. Not sure why we're getting so much related CPU spin on that...
I'm going to fix the tests and merge that one if nobody's against.
sorry, didn't react fast enough.
:) no problem! I assumed nobody was against it so merged it. Hope it was ok.
The biggest problems with event processing we faced so far is that the consumer queues get overwhelmed and start pushing the writer queue way behind.
This patch implements a micro-scheduling system (currently extremely primitive), that's using queue-per-core that polls and processes writer flushes from every worker process and "regular" digested events (currently in non-numa-aware fashion & without thread affinity) via single-producer-single-consumer queues.
For example, 4 workers are started, each worker has an affine single producer (netty event loop) single consumer (worker thread) queue. It polls events from it and ingests them.
Every worker, prior to polling it's queue, checks if there are any pending flushes / database writes. If there are any, it submits them. Results are collected asynchronously.
This will help us to move at least part of cyanite that's related to writes to more predictable / controllable env. Also, we can further improve it, make it numa-aware and make scheduling and back-pressure smarter.