Closed brenns10 closed 1 month ago
So this isn't surprising and I don't know why it took me a while to realize that the "tiny" configurations use SLOB, which doesn't have the slab_cache
pointer. By design.
But I guess I'm confused about why the test fails now. Did we just never notice it?
vmtest build 205 PASSED on drgn 8ab4901d8336faf74bdd5f0b6bf8127d87d818f9, but it didn't run the 6.1.y tiny configuration, it ran the 6.1.y default configuration. Here is what it ran from one of the logs: it did not run against any tiny configurations.
Passed: local, 6.10.0-rc7-vmtest30.1default, 6.9.8-vmtest30.1default, 6.8.12-vmtest30.1default, 6.7.12-vmtest30.1default, 6.6.37-vmtest30.1default, 6.5.13-vmtest30.1default, 6.4.16-vmtest30.1default, 6.3.13-vmtest30.1default, 6.2.16-vmtest30.1default, 6.1.97-vmtest30.1default, 6.0.19-vmtest30.1default, 5.19.17-vmtest30.1default, 5.18.19-vmtest30.1default, 5.17.15-vmtest30.1default, 5.16.20-vmtest30.1default, 5.15.162-vmtest30.1default, 5.14.21-vmtest30.1default, 5.13.19-vmtest30.1default, 5.12.19-vmtest30.1default, 5.11.22-vmtest30.1default, 5.10.221-vmtest30.1default, 5.4.279-vmtest30.1default, 4.19.317-vmtest30.1default, 4.14.336-vmtest30.1default, 4.9.337-vmtest30.1default
vmtest build 206 FAILED on the current main, de8ba223defa8dc70c44385e230cf2b13623f7cd, presumably because it did run the 6.1.y tiny configuration.
Failed: 6.1.98-vmtest30.0tiny
Passed: local, 6.10.0-vmtest30.1default, 6.9.9-vmtest30.0tiny, 6.8.12-vmtest30.1default, 6.7.12-vmtest30.1default, 6.6.39-vmtest30.0tiny, 6.5.13-vmtest30.1default, 6.4.16-vmtest30.1default, 6.3.13-vmtest30.1default, 6.2.16-vmtest30.1default, 6.0.19-vmtest30.1default, 5.19.17-vmtest30.1default, 5.18.19-vmtest30.1default, 5.17.15-vmtest30.1default, 5.16.20-vmtest30.1default, 5.15.162-vmtest30.1default, 5.14.21-vmtest30.1default, 5.13.19-vmtest30.1default, 5.12.19-vmtest30.1default, 5.11.22-vmtest30.1default, 5.10.221-vmtest30.1default, 5.4.279-vmtest30.1default, 4.19.317-vmtest30.1default, 4.14.336-vmtest30.1default, 4.9.337-vmtest30.1default
The vmtest is running with -K
which is not supposed to run the tiny configurations... That's odd.
I now notice that the 6.1.98 kernel has not even been built for the default configuration, only the tiny. It appears that the test runner will use the latest 6.1 kernel version (6.1.98) and since there's no default build, it falls back to the tiny! I guess when the build job ran, the 6.1.98 kernel hadn't been tagged yet, but by the time the tiny build ran, it had?
This explains why the test ran suddenly. But also, the test shouldn't have failed, right?
Your guess sounds right to me. I'll kick off a manual vmtest build to unblock this and fix setup.py
to explicitly choose the default
flavor.
I'm not sure why the test is failing, though. I'm fairly certain I did a manual run on all flavors before cutting 0.0.27, but maybe I forgot to do that.
Ok, git bisect blames c7717280ad2f867cefc62e9e841dc0787b08d39b, and indeed, the newly added print_annotated_memory()
test fails on the 6.1.98-vmtest30.0tiny. However, the test failure is different:
======================================================================
FAIL: test_print_annotated_memory (tests.linux_kernel.helpers.test_common.TestPrintAnnotatedMemory)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/stepbren/repos/drgn/tests/linux_kernel/helpers/test_common.py", line 77, in test_print_annotated_memory
self.assertIn("slab object: drgn_test_small+0x0", f.getvalue())
AssertionError: 'slab object: drgn_test_small+0x0' not found in 'ADDRESS VALUE\nffffffffa021b7e0: ffff8880035d8710\nffffffffa021b7e8: ffff8880035d8740\nffffffffa021b7f0: ffff8880035d8770\nffffffffa021b7f8: ffff8880035d87a0\nffffffffa021b800: ffff8880035d87d0\n'
----------------------------------------------------------------------
That makes sense, the test doesn't check whether slob is active, so there will be no slab output in there.
When I run tests on 6.1.98-vmtest30.0tiny against that commit's parent, 1232a404c65c29944de05f8cd7d1c7b6908cda99, the tests do pass.
Now, looking at 48e0c51b0a0e85337595fe6bd598f080850e2f44, I see that on that commit there are the same three failures we saw in the original message. But on its parent, it's just the one. So it's this commit which actually introduces the three failures for fetching the slab_cache
field on SLOB.
So there are two issues to fix:
print_annotated_memory()
function to check for SLOB and not assert that slab objects will appear in that case._identify_address()
to avoid failing on SLOB.Oof, clearly I did not actually run the tests on the tiny flavor, then... Another followup will be to test all of the kernel flavors in CI at least some of the time. Maybe as part of the weekly vmtest build and on-demand with a test-all-kernel-flavors
tag like we have for Python versions.
Oof, clearly I did not actually run the tests on the tiny flavor, then... Another followup will be to test all of the kernel flavors in CI at least some of the time. Maybe as part of the weekly vmtest build and on-demand with a test-all-kernel-flavors tag like we have for Python versions.
Yeah, that would be a great option to have.
Regarding 48e0c51b0a0e85337595fe6bd598f080850e2f44, the commit message reads:
helper.common.memory: recognize vmap addresses in identify_address()
This requires rearranging a couple of things: symbols from modules are
also vmap addresses, so we need to check for symbols first, and we need
to open-code slab_object_info to avoid some redundant operations
I guess the open-coding of slab_object_info()
was the bit that resulted in the regression. I think the main thing that open-coding does is avoid re-testing for in_direct_map
. I think we can avoid that by shifting the check out into find_containing_slab()
from _find_containing_slab()
, and then we can drop the open-coding. In fact, a helper named in_direct_map
could be helpful anyway, and then we can simplify everything and make the tests pass too :)
Yeah, the direct map check is the only potential redundancy I see looking at it again, so that plan sounds good to me. Were you planning on working on this? No pressure, your investigation was already plenty helpful, I appreciate it!
Yeah, I am partway through it, so I can send a PR :)
I'm getting this on the
main
branch for6.1.98-vmtest30.0tiny
:It happened on my fork's CI somehow and I've verified it locally. Looking into it now but figured I'd post an issue for tracking.