search

hamadmarri / cacule-cpu-scheduler

The CacULE CPU scheduler is based on interactivity score mechanism. The interactivity score is inspired by the ULE scheduler (FreeBSD scheduler).
266 stars 32 forks source link

Public Chat #21

Closed hamadmarri closed 3 years ago

hamadmarri commented 3 years ago

Hi Everyone

Consider this as an open thread to discuss and talk about Cachy/CacULE

I would post some patches for testing too.

We could talk about how to enhance the scheduler.

Thank you.

hamadmarri commented 3 years ago

Please try the global runqueue balancer patch on top of cacule

https://github.com/hamadmarri/cacule-cpu-scheduler/blob/master/patches/CacULE/v5.9/02-grq-5.9.patch

Let me know if there is any errors on kernel log, and any performance improvements.

Thank you

raykzhao commented 3 years ago

Hi @hamadmarri

I got some page table or page map errors in dmesg with the grq patch. Sometimes the app or even the whole system will freeze. Also just after a system freeze, I need to run e2fsck in order to fix the filesystem corruption. log.txt

hamadmarri commented 3 years ago

Hi @hamadmarri

I got some page table or page map errors in dmesg with the grq patch. Sometimes the app or even the whole system will freeze. Also just after a system freeze, I need to run e2fsck in order to fix the filesystem corruption. log.txt

Hi @raykzhao

Yes, that was a problem but I thought I fixed it with the last commit c7bbf9c3f617bf5c37ce8b45ef4c99076051cc96. Which commit you tried? The last commit was pushed 20h ago.

Thank you

raykzhao commented 3 years ago

Hi @hamadmarri

I believe I used the latest patch when I reported the bug, which should include the fix you mentioned.

The out-of-tree kernel patches I'm using are here: https://gitlab.com/raykzhao/raykzhao-overlay/-/tree/master/sys-kernel/ray-cacule-sources/files. Also my kernel config should only have minor differences compared with the attachment i.e. replacing Cachy with CacULE.

config-5.10.3.zip

hamadmarri commented 3 years ago

Hi @hamadmarri

I believe I used the latest patch when I reported the bug, which should include the fix you mentioned.

The out-of-tree kernel patches I'm using are here: https://gitlab.com/raykzhao/raykzhao-overlay/-/tree/master/sys-kernel/ray-cacule-sources/files. Also my kernel config should only have minor differences compared with the attachment i.e. replacing Cachy with CacULE.

config-5.10.3.zip

Hi @raykzhao

I am compiling it with CONFIG_PREEMPT=y, it could be a reason. I have test the https://github.com/hamadmarri/cacule-cpu-scheduler/commit/c7bbf9c3f617bf5c37ce8b45ef4c99076051cc96 with CONFIG_PREEMPT_VOLUNTARY for about 24h with no error appeared. Usually (before the last commit), the error happens in random times but within few hours.

Just for the sake of cariousity, I noticed that you are using HZ=100 and CONFIG_PREEMPT, I know 100hz increase the performance but with less interactivity. Will setting CONFIG_PREEMPT with 100hz helps to overcome latency issues but maintaining performance?

Thank you

hamadmarri commented 3 years ago

Hi @hamadmarri

I believe I used the latest patch when I reported the bug, which should include the fix you mentioned.

The out-of-tree kernel patches I'm using are here: https://gitlab.com/raykzhao/raykzhao-overlay/-/tree/master/sys-kernel/ray-cacule-sources/files. Also my kernel config should only have minor differences compared with the attachment i.e. replacing Cachy with CacULE.

config-5.10.3.zip

Hi @raykzhao

I confirm this bug happens when CONFIG_PREEMPT=y. Could you please try with CONFIG_PREEMPT_VOLUNTARY = y until I fix the bug with CONFIG_PREEMPT

Please let me know if the bug still reported with CONFIG_PREEMPT_VOLUNTARY = y. In my machine no bug shown so far.

You might need to backup your files first :D

Thank you

raykzhao commented 3 years ago

Hi @hamadmarri

Unfortunately I'm still getting app/system freeze even after switching to voluntary preemption. Other than that, the system is very smooth (even smoother than Cachy/CacULE) under heavy load. Remind me of the old BFS days when it was still using a global queue.

log.txt config-5.10.3-grq.zip

hamadmarri commented 3 years ago

Hi @hamadmarri

Unfortunately I'm still getting app/system freeze even after switching to voluntary preemption. Other than that, the system is very smooth (even smoother than Cachy/CacULE) under heavy load. Remind me of the old BFS days when it was still using a global queue.

log.txt config-5.10.3-grq.zip

I removed grq from the hub since it harms the FS. I am thinking to borrow the bfs-ck and apply the hrrn on it. What do you think?

Thank you @raykzhao

raykzhao commented 3 years ago

Hi @hamadmarri

Another (maybe more up-to-date) codebase might be the old VRQ scheduler (the predecessor of PDS) before it also switched to skiplist. It seems to be a variant of BFS by fixing some locking issues. However, I have no idea how it compared to BFS since I didn't know about it at that time.

The latest VRQ scheduler I can find without skiplist is at http://cchalpha.blogspot.com/2016/08/v470472vrq3-patch-released.html. It is based on the 4.7.2 kernel source code.

hamadmarri commented 3 years ago

grq-r2.zip

Please try this full patch. global locks moved to core.c with many locking (for every possible enqueue/dequeue paths). HRRN policy, so it is basically cachy with grq.

disable auto/fair group

Thank you

raykzhao commented 3 years ago

Hi @hamadmarri

I get the following compiling errors:

kernel/sched/core.c:7640:17: error: use of undeclared identifier 'p'
        GLOBAL_RQ_LOCK(p->sched_class);
                       ^
kernel/sched/core.c:7664:19: error: use of undeclared identifier 'p'
        GLOBAL_RQ_UNLOCK(p->sched_class);

Should they become tsk->sched_class?

By the way, does this patch work with CONFIG_PREEMPT now?

raykzhao commented 3 years ago

Hi @hamadmarri

The new patch seems to work fine with CONFIG_PREEMPT, and I think #22 can be closed. The patch feels smoother than original Cachy and all variants of CacULE on my machine under heavy load. In fact, this seems to be the only scheduler that can play the video you posted at https://github.com/hamadmarri/cacule-cpu-scheduler/issues/15#issuecomment-748137399 at 1440p without any glitches when compiling the kernel with make -j5 on my 4-core laptop. Cachy and all variants of CacULE on the other hand will have several glitches at 1440p in this case (all of them should have no glitches at 1080p though).

However, it seems that the latency is slightly worse. I guess it's due to all the lockings. I will try to increase CONFIG_HZ and see if it will bring any improvement. Meanwhile I think you may cleanup the patch a bit, since I can see many codes are commented out. Also the interactivity_factor is unused and cacule_max_lifetime should be tunable via sysctl.

Thank you very much for your work!

hamadmarri commented 3 years ago

Hi @hamadmarri

I get the following compiling errors:

kernel/sched/core.c:7640:17: error: use of undeclared identifier 'p'
        GLOBAL_RQ_LOCK(p->sched_class);
                       ^
kernel/sched/core.c:7664:19: error: use of undeclared identifier 'p'
        GLOBAL_RQ_UNLOCK(p->sched_class);

Should they become tsk->sched_class?

By the way, does this patch work with CONFIG_PREEMPT now?

Hi @raykzhao

I disabled CGROUP_SCHED since it is not needed, that's why it didn't show me an error. Sorry about that. Try disable CGROUP_SCHED it could enhance the performance.

I got crash when try to set the computer to sleep with no errors. Also I got NULL reference one time on reboot. I need to fix these issues.

By the way, does this patch work with CONFIG_PREEMPT now?

So far it is working with PREEMPT, no mm/FS errors.

Thank you

hamadmarri commented 3 years ago

Hi @hamadmarri

The new patch seems to work fine with CONFIG_PREEMPT, and I think #22 can be closed. The patch feels smoother than original Cachy and all variants of CacULE on my machine under heavy load. In fact, this seems to be the only scheduler that can play the video you posted at #15 (comment) at 1440p without any glitches when compiling the kernel with make -j5 on my 4-core laptop. Cachy and all variants of CacULE on the other hand will have several glitches at 1440p in this case (all of them should have no glitches at 1080p though).

However, it seems that the latency is slightly worse. I guess it's due to all the lockings. I will try to increase CONFIG_HZ and see if it will bring any improvement. Meanwhile I think you may cleanup the patch a bit, since I can see many codes are commented out. Also the interactivity_factor is unused and cacule_max_lifetime should be tunable via sysctl.

Thank you very much for your work!

In fact, this seems to be the only scheduler that can play the video you posted at #15 (comment) at 1440p without any glitches when compiling the kernel with make -j5 on my 4-core laptop.

How about CFS, is grq better than CFS on this test?

However, it seems that the latency is slightly worse.

Based on this test https://github.com/hamadmarri/os-scheduler-responsiveness-test, the results are no different than cachy. However, I am guessing that on some parts in the scheduler, the global lock discards some interrupts. I think I need to use spin_lock_irqsave instead of just spin_lock to save interrupts during waiting/spin to hold the lock.

I will clean up this patch soon.

Thank you

raykzhao commented 3 years ago

Hi @hamadmarri

From what I read, spin_lock may cause deadlock when involving interrupts. I have encountered one system freeze so far (unfortunately the log didn't record anything useful), and I guess it could be possibly due to a deadlock.

I have replaced spin_lock with spin_lock_irqsave on my local kernel source as you suggested (and replacing spin_unlock with spin_unlock_irqrestore). So far I haven't encountered any system freeze yet.

raykzhao commented 3 years ago

How about CFS, is grq better than CFS on this test?

Well, CFS seems to be the worst in this test. There are glitches even when playing 1080p with CFS. For 1440p, the glitches will cause the video stuck for a couple of seconds. I'm using CONFIG_HZ=100 and CONFIG_SCHED_AUTOGROUP=y for my CFS build. I guess probably CONFIG_HZ=100 is too low for CFS, but I can get away with low clock frequency and have a smoother system than CFS when using alternative schedulers (PDS, BMQ, Cachy, CacULE).

hamadmarri commented 3 years ago

Hi @raykzhao

Does the system goes suspend successfully? On my machine it crashes when I try to suspend/sleep

Thank you

raykzhao commented 3 years ago

Hi @hamadmarri

Unfortunately my system also hung when trying to suspend. I didn't compile the kernel with hibernation so I didn't try sleep.

I'm recompiling the kernel with CONFIG_PM_DEBUG=y and will see if there would be any debugging information left in the log when trying to suspend.

raykzhao commented 3 years ago

Hi @hamadmarri

I tried the PM debugging based on https://www.kernel.org/doc/Documentation/power/basic-pm-debugging.txt. My system hung at the "processors" test. Based on the following suggestion from the documentation, I also got a system freeze when trying to take a CPU core offline.

If the "processors" test fails, the disabling/enabling of nonboot CPUs does not work (of course, this only may be an issue on SMP systems) and the problem should be reported. In that case you can also try to switch the nonboot CPUs off and on using the /sys/devices/system/cpu/cpu*/online sysfs attributes and see if that works.

I guess it might needs to lock in the __pick_migrate_task function. I will try on my local kernel source.

hamadmarri commented 3 years ago

Hi @hamadmarri

I tried the PM debugging based on https://www.kernel.org/doc/Documentation/power/basic-pm-debugging.txt. My system hung at the "processors" test. Based on the following suggestion from the documentation, I also got a system freeze when trying to take a CPU core offline.

If the "processors" test fails, the disabling/enabling of nonboot CPUs does not work (of course, this only may be an issue on SMP systems) and the problem should be reported. In that case you can also try to switch the nonboot CPUs off and on using the /sys/devices/system/cpu/cpu*/online sysfs attributes and see if that works.

I guess it might needs to lock in the __pick_migrate_task function. I will try on my local kernel source.

Hi @raykzhao

Please see the attached patch r3. I did some enhancements to cover less code during locks. Also I added a lock in __pick_migrate_task but didn't work too with suspend.

grq-r3.zip

hamadmarri commented 3 years ago

And this the full patch grq-r3.zip

raykzhao commented 3 years ago

Hi @hamadmarri

I occasionally get system freeze. To be safe I would still turn every lock to spin_lock_irqsave on my local kernel source for now.

hamadmarri commented 3 years ago

Hi @hamadmarri

I occasionally get system freeze. To be safe I would still turn every lock to spin_lock_irqsave on my local kernel source for now.

Hi @raykzhao

Could you please share your changes? When I added irqsave for every lock I got some kernel bug errors.

Sorry I missed up some flags names.

Thank you

hf29h8sh321 commented 3 years ago

Hi, I compiled with commit aba0dea93cbfed48092f04a4e6522abc83d17fe5 from kernel tree. I get stutters when system load spikes. For example, if I am working on some code, and frequently re-compile the code, any sound playing in the background will interrupt for around ~200ms, every time the compiler is invoked.

raykzhao commented 3 years ago

Hi @hf29h8sh321

Please try the fix at https://github.com/hamadmarri/cacule-cpu-scheduler/issues/20#issuecomment-752214217.

hamadmarri commented 3 years ago

Hi, I compiled with commit aba0dea93cbfed48092f04a4e6522abc83d17fe5 from kernel tree. I get stutters when system load spikes. For example, if I am working on some code, and frequently re-compile the code, any sound playing in the background will interrupt for around ~200ms, every time the compiler is invoked.

Hi @hf29h8sh321

Sorry I forgot to push changes on github. I have just pushed changes. Please try the last commit https://github.com/hamadmarri/linux/commit/21aa57c51799fd45189fe47caa30ea1eaa928515

Thank you

hamadmarri commented 3 years ago

rdb-r1.zip

Please try this. If performing better, I might consider it as the main patch.

It is cachy-r8 + rdb (response driven balancer) It is stable since it uses regular multi-runqueues locking approach. Removed all load statistics for the load balancer since rdb uses HRRN values to migrate tasks.

Thanks

raykzhao commented 3 years ago

Hi @hamadmarri

The rdb patch feels smoother than original Cachy/CacULE, but slightly worse than the grq patch under heavy load.

hf29h8sh321 commented 3 years ago

Applying the patch from interactivity_score_fix.zip is not as smooth. If system is loaded, audio interrupts and ui feels laggy.

The smoothest for me has been the old "rdb" branch from the kernel tree. I don't know if that branch is the same as the rdb-r1.zip patch.

raykzhao commented 3 years ago

Hi @hamadmarri

Just a minor bug in the rdb patch: the cfs_rq_last_update_time function should be removed when CONFIG_CACHY_SCHED=y. Currently the compiler will complain that cfs_rq_last_update_time doesn't have a return value on 32-bit devices. It's pretty safe to remove this function, since it's inline and its only caller sync_entity_load_avg is not used when CONFIG_CACHY_SCHED=y anyway.

hamadmarri commented 3 years ago

Hi @hamadmarri

Just a minor bug in the rdb patch: the cfs_rq_last_update_time function should be removed when CONFIG_CACHY_SCHED=y. Currently the compiler will complain that cfs_rq_last_update_time doesn't have a return value on 32-bit devices. It's pretty safe to remove this function, since it's inline and its only caller sync_entity_load_avg is not used when CONFIG_CACHY_SCHED=y anyway.

Thank you @raykzhao I will clean it up :+1:

hamadmarri commented 3 years ago

Cachy with RDB balancer updated https://github.com/hamadmarri/cacule-cpu-scheduler/blob/master/patches/Cachy/v5.9/cachy-5.9-r9.patch

hamadmarri commented 3 years ago

Applying the patch from interactivity_score_fix.zip is not as smooth. If system is loaded, audio interrupts and ui feels laggy.

The smoothest for me has been the old "rdb" branch from the kernel tree. I don't know if that branch is the same as the rdb-r1.zip patch.

Hi @hf29h8sh321 Yes, it is the same. I have added Cachy-r9 which is the rdb branch cleaned up https://github.com/hamadmarri/cacule-cpu-scheduler/blob/master/patches/Cachy/v5.9/cachy-5.9-r9.patch

Yes

hamadmarri commented 3 years ago

Some results of operarena test on phoronix-test-suite

EDIT: added kernel build https://openbenchmarking.org/result/2101082-NI-RDB32364101

hamadmarri commented 3 years ago

Compared with BMQ https://openbenchmarking.org/result/2101110-NI-RDB13468301

Salekin-1169 commented 3 years ago

Compared with BMQ https://openbenchmarking.org/result/2101110-NI-RDB13468301

Is this version of cachy different from the cacule scheduler ?

hamadmarri commented 3 years ago

Compared with BMQ https://openbenchmarking.org/result/2101110-NI-RDB13468301

Is this version of cachy different from the cacule scheduler ?

Hi @Salekin-1169

Yes, CacULE is slightly different than Cachy.

Salekin-1169 commented 3 years ago

Compared with BMQ https://openbenchmarking.org/result/2101110-NI-RDB13468301

Is this version of cachy different from the cacule scheduler ?

Hi @Salekin-1169

Yes, CacULE is slightly different than Cachy.

I see. I had the idea that Cachy was just renamed to CacULE. Are you actively developing both schedulers at once? That's impressive.

Just curious why both of them share the same github repo ?

Salekin-1169 commented 3 years ago

I found your explanation here
https://forum.xanmod.org/thread-2-page-24.html?highlight=cacule

I'll read more on it. Thanks :bow:

hamadmarri commented 3 years ago

Compared with BMQ https://openbenchmarking.org/result/2101110-NI-RDB13468301

Is this version of cachy different from the cacule scheduler ?

Hi @Salekin-1169 Yes, CacULE is slightly different than Cachy.

I see. I had the idea that Cachy was just renamed to CacULE. Are you actively developing both schedulers at once? That's impressive.

Just curious why both of them share the same github repo ?

I was planning to only support CacULE instead of Cachy because on my machine CacULE was superior on all tests to Cachy. However, some people believe that Cachy is better, therefore I have to keep cachy alive. I am planning to make one patch that contains both cachy and cacule.

raykzhao commented 3 years ago

Hi @hamadmarri

I'm wondering if the rdb can also be adapted by CacULE.

Salekin-1169 commented 3 years ago

@hamadmarri Thank you for your hard work. I appreciate it a lot.

hamadmarri commented 3 years ago

Hi @hamadmarri

I'm wondering if the rdb can also be adapted by CacULE.

Hi @raykzhao

Not yet, I am working on this. The rdb nature depends highly on the HRRN. For CacULE (uses interactivity score) the RDB will be slightly different and I hope it will be as performant as the cachy-rdb. I will share a patch of cacule-rdb for testing soon.

Thank you @raykzhao for all your help and supports. I am really appreciated.

hamadmarri commented 3 years ago

@hamadmarri Thank you for your hard work. I appreciate it a lot.

No worries @Salekin-1169

I am thankful too for all your supports and testing.

hamadmarri commented 3 years ago

Cacule-rdb patch attached. It doesn't seem as performant as cachy-rdb. Please let me know how it performs.

cacule-rdb.zip

raykzhao commented 3 years ago

Hi @hamadmarri

It seems that the patch is on top of the xanmod-cacule. In order to apply the patch to the vanilla Linux kernel, I need to apply hamadmarri/linux@148c235d6462640686bb98c6a3403700437440fe and cherry-pick hamadmarri/linux@e83a7be10623963f04f2dfedb438faad20a878f9. In addition, I also need to sync the patch without the xanmod sysctl_sched_latency tweaks.

With regards to sysctl_sched_latency, I am currently using 4ms, since it's the default value of PDS/BMQ timeslice and in zen-kernel CFS (and also the zen-kernel MuQSS rr_interval sometime ago). I feel 2ms (current xanmod-cacule default) might be too small for me since I had bad experiences with 2ms timeslice in PDS/BMQ previously. Not sure how this would affect the performance or responsiveness in Cachy/CacULE though.

With the 4ms sysctl_sched_latency, Cachy/CacULE both with the rdb feel similarly quite good for me.

hamadmarri commented 3 years ago

Hi @hamadmarri

It seems that the patch is on top of the xanmod-cacule. In order to apply the patch to the vanilla Linux kernel, I need to apply hamadmarri/linux@148c235 and cherry-pick hamadmarri/linux@e83a7be. In addition, I also need to sync the patch without the xanmod sysctl_sched_latency tweaks.

With regards to sysctl_sched_latency, I am currently using 4ms, since it's the default value of PDS/BMQ timeslice and in zen-kernel CFS (and also the zen-kernel MuQSS rr_interval sometime ago). I feel 2ms (current xanmod-cacule default) might be too small for me since I had bad experiences with 2ms timeslice in PDS/BMQ previously. Not sure how this would affect the performance or responsiveness in Cachy/CacULE though.

With the 4ms sysctl_sched_latency, Cachy/CacULE both with the rdb feel similarly quite good for me.

Hi @raykzhao

I totally forgot about the sysctl_sched_latency changes on xanmod. I believe it affected the rdb implementation on cacule. The reason I tried rdb on xanmod-cacule directly is that based on Alex tests rdb was bad on xanmod. I tested Cachy-rdb on xanomd(without cacule) and it was good same as rdb on vanilla kernel. However, this time rdb on xanmod-cacule wasn't good at all. I think it is due to sysctl_sched_latency. Thank you so much for mentioning this. I was wondering why cacule-rdb wasn't good even though the rdb implementation is almost the same to cachy-rdb. I will try to make a cacule-rdb on vanilla kernel, but first I will tweak the sysctl_sched_latency value and see if I got the same results you had.

Thank you so much :+1:

hamadmarri commented 3 years ago

cacule-rdb on vanilla kernel seems better

https://openbenchmarking.org/result/2101233-NI-CACULERDB13

https://github.com/hamadmarri/cacule-cpu-scheduler/blob/master/patches/CacULE/v5.10/cacule5.10-rdb.patch

raykzhao commented 3 years ago

Hi @hamadmarri

How does CacULE-rdb compare with Cachy-rdb? It seems that the benchmark score of CacULE-rdb is higher than the score of Cachy-rdb in your previous benchmark. Is that correct?

As long as CacULE-rdb is better than Cachy-rdb, I think it makes sense to only maintain CacULE-rdb in the future, if maintaining both Cachy/CacULE has become a burden for you. You may close #19.

hamadmarri commented 3 years ago

Hi @hamadmarri

How does CacULE-rdb compare with Cachy-rdb? It seems that the benchmark score of CacULE-rdb is higher than the score of Cachy-rdb in your previous benchmark. Is that correct?

As long as CacULE-rdb is better than Cachy-rdb, I think it makes sense to only maintain CacULE-rdb in the future, if maintaining both Cachy/CacULE has become a burden for you. You may close #19.

Hi @raykzhao ,

I reinstalled Opensuse before the last test and didn't compile cachy-rdb. I can't be sure if cacule-rdb is better because Tumbleweed is updated since cachy-rdb test, also I changed FS from btrfs to ext4. I have to compile cachy-rdb on same kernel version and test again.

I think having one patch for both cacule and cachy is possible but need time and also I have to be very careful while joining cachy to cacule in one patch. After combining them, maintaining both in one patch will be easy. I really hope that if anyone still want cachy updated to let me know, or if you all agree that cachy is no more needed also let me know, but it is hard to know that. I don't mind to have both in one patch as you suggested in #19 .

Thank you