search

akkadotnet / akka.net

Canonical actor model implementation for .NET with local + distributed actors in C# and F#.
http://getakka.net
Other
4.69k stars 1.04k forks source link

Discussion: Port scala LightArrayRevolverScheduler (HashedWheelTimerScheduler) changes for fixed rate vs Fixed Delay? #4640

Open to11mtm opened 3 years ago

to11mtm commented 3 years ago

Akka for JVM has had changes over the last couple years so that their scheduler may operate at a Fixed Delay versus a fixed rate.

https://github.com/akka/akka/pull/27034

Related Discussion: https://github.com/akka/akka/issues/26910

It is interesting to note, thinking about @IgorFedchenko 's attempts to optimize timing in HashedWheelTimer (HWT) that Scala side is investigating issues with timing being imprecise for low values (see HERE: https://github.com/akka/akka/issues/29663)

So there is a tradeoff here of stability versus performance, at least for now.

Zetanova commented 3 years ago

One think to note is that we should test on: Windows Workstation Windows Server Linux

All 3 systems have different thread-scheduling Windows Workstation: ~16ms aka "thread quantum" Windows Server: >32ms Linux: depends on the scheduler

A good tool to measure the "windows thread quantum" is clockres.exe from sysinternals It uses the winAPI GetSystemTimeAdjustment to get it.

The Windows "thread quantum" can be set system wide with foreground or background option: image

The main problem is that most developer know this option but do not know what it does. ... now you know

Windows 10 (Foreground Services Option) Clockres v2.1 - Clock resolution display utility Copyright (C) 2016 Mark Russinovich Sysinternals

Maximum timer interval: 15.625 ms Minimum timer interval: 0.500 ms Current timer interval: 1.000 ms

Windows Server (Background Services Option) Clockres v2.1 - Clock resolution display utility Copyright (C) 2016 Mark Russinovich Sysinternals

Maximum timer interval: 15.625 ms Minimum timer interval: 0.500 ms Current timer interval: 15.625 ms

Linux (have low ex)

Zetanova commented 3 years ago

Beside the newer "SYNTHETIC TIMERS" in windows 8/10

Since end of 2018 because of Sepectre/Meltdown the QueryPerformanceCounter is increased to 10Mhz

This need to be taken into account too.

Aaronontheweb commented 3 years ago

Since end of 2018 because of Sepectre/Meltdown the QueryPerformanceCounter is increased to 10Mhz

😬

Probably not a good thing then that I do most of my benchmarking on this Ryzen machine

Zetanova commented 3 years ago

Do we have a benchmark for short and long timers? What are the real world scenarios for the scheduler especially then the new DotNetty IO tasks getting scheduled on it?

I would have some ideas that I want to test for the HashWheelScheduler

1) Remove the unnecessary LinkList from the Buckets All items of the bucket are getting touched and removed one after the other. a struct of arrays or array of structs has much better memory locality and fast interation, and maybe the compacting of items after remove can be completely skipped

2) Implement two level of timers 1.level for short-timers with the current implementation A new 2.level for long-timers where timers get stored in a unsorted list. And pulled into the first level on demand.

3) Idling when future buckets are empty

4) Align the ticks with the current system timer interval Formula for Windows Server Systems would be the MultipleOfSystemTimerInterval-1ms Current timer interval is 15.625 (windows server) Then this would result an alignment of the config value to 15ms, 31ms, 46ms, 62ms

Aaronontheweb commented 3 years ago

@Zetanova we do have a benchmark for how quickly items can be scheduled and how quickly the scheduler can work through those: https://github.com/akkadotnet/akka.net/blob/8b1d54bbf2bd297187058b9994c15e5a15a73d6d/src/core/Akka.Tests.Performance/Actor/Scheduler/DefaultSchedulerPerformanceTests.cs