awslabs / damo

DAMON user-space tool
https://damonitor.github.io/
GNU General Public License v2.0
148 stars 28 forks source link

could not turn DAMON with damo schemes #75

Closed honggyukim closed 9 months ago

honggyukim commented 10 months ago

I have updated my damo repo based on the current next branch, but I see an error when running damo schemes as follows.

$ cat pageout.json
[
    {
        "action": "pageout",
        "access_pattern": {
            "sz_bytes": {"min": "4.000 KiB", "max": "max"},
            "nr_accesses": {"min": "0 %", "max": "0 %"},
            "age": {"min": "1 s", "max": "max"}
        }
    }
]

$ ./damo schemes -c pageout.json
could not turn DAMON on (cannot apply kdamonds from args (filepath (/sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/schemes/0/apply_interval_us) is neither dir nor file))

It works fine on the previous version at 0c48a1fabb501ef146c976aad9050ccfd88daed9.

My kernel version is based on v6.4.

honggyukim commented 10 months ago

damo start and damo stop works fine, but only damo schemes doesn't work in this case.

honggyukim commented 10 months ago

git bisect shows the following 07c8a6a5847895b9a357563b6a50a25618d4273c is the bad commit.

07c8a6a5847895b9a357563b6a50a25618d4273c is the first bad commit
commit 07c8a6a5847895b9a357563b6a50a25618d4273c
Author: SeongJae Park <sj38.park@gmail.com>
Date:   Mon Sep 4 02:49:01 2023 +0000

    _damon_sysfs: Support DAMOS apply intervals

    Signed-off-by: SeongJae Park <sj38.park@gmail.com>

 _damon_sysfs.py                | 4 +++-
 tests/unit/test_damon_sysfs.py | 1 +
 2 files changed, 4 insertions(+), 1 deletion(-)
honggyukim commented 10 months ago

It looks apply_interval_us is newly added, but I need to investigate what it is.

honggyukim commented 10 months ago

The thing is that my kernel doesn't have apply_interval_us inside sysfs.

$ ls /sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/schemes/0/
access_pattern  action  filters  quotas  stats  tried_regions  watermarks

$ ls /sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/schemes/0/apply_interval_us
ls: cannot access '/sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/schemes/0/apply_interval_us': No such file or directory
honggyukim commented 10 months ago

It looks apply_interval_us comes from https://git.kernel.org/pub/scm/linux/kernel/git/sj/linux.git/commit/?h=damon/next&id=b4075b3601ceccc383ab32d8b06b7ad8db0dbc3d, which is not merged into the mainline.

We might have to keep the backward compatibility with and without this change.

sj-aws commented 10 months ago

Thank you for this report, Honggyu! Also thank you for further deeply investigating the root cause. As you found, this is due to the under-development feature. We're developing a feature called DAMOS apply interval, which will make DAMOS be able to be applied regardless of the aggregation interval. It's still in only the damon/next tree, and recently damo has updated to support it. However, I forgot adding check of the availability of the feature on running kernel.

I also noticed DAMON functionality tests failing on latest >=6.1 kernels. I guess this was the reason.

I just implemented the check, confirmed it working, and pushed: https://github.com/awslabs/damo/commit/b5ae1c11b5dd232c01309361dda698380627cc9a

Please let me know if it doesn't work for you.

honggyukim commented 10 months ago

Hi SeongJae,

Thanks for the quick fix. I've tested it on my home machine and see that the problem is fixed.

I will test it in the office tomorrow again, then close this issue. Thanks!

honggyukim commented 9 months ago

I've just confirmed that b5ae1c11b5dd232c01309361dda698380627cc9a fixes the problem. Thanks very much!