support Yocto build system to save the code size

[jamin-aspeed]

If users want to install drgn package into users firmware, it is required to install pip3 package. However, pip3 package code size is very large. Besides, users firmware should install compiler for libdrgn compilation. To reduce the code size and support cross-compile, please help to support Yocto/OpenEmbedded-Core build system.

I created a new recipe and the contents of this recipe as following.

• python3-drgn_0.0.27.bb

SUMMARY = "drgn (pronounced dragon) is a debugger with an emphasis on programmability."
HOMEPAGE = "https://github.com/osandov/drgn"
LICENSE = " LGPL-2.1-or-later"
LIC_FILES_CHKSUM = "file://COPYING;md5=7c83d30e99508d6b790eacdd3abab846"

SRC_URI = "git://github.com/osandov/drgn.git;protocol=https;branch=main"
SRCREV = "a8dfc9e31f551d4b3c3d4307fa19ab6b3bfafb1c"

S = "${WORKDIR}/git"

DEPENDS = "\
    autoconf-native \
    automake-native \
    libtool-native \
    elfutils \
    "

RDEPENDS:${PN} = "\
    libdw \
    libelf \
    libgomp \
    python3-crypt \
    python3-io \
    python3-logging \
    python3-math \
    python3-pickle \
    python3-stringold \
    "

export CONFIGURE_FLAGS = "\
    --build=${BUILD_SYS}, \
    --host=${HOST_SYS}, \
    --target=${TARGET_SYS}, \
    --prefix=${prefix}, \
    --exec_prefix=${exec_prefix}, \
    --bindir=${bindir}, \
    --sbindir=${sbindir}, \
    --libexecdir=${libexecdir}, \
    --datadir=${datadir}, \
    --sysconfdir=${sysconfdir}, \
    --sharedstatedir=${sharedstatedir}, \
    --localstatedir=${localstatedir}, \
    --libdir=${libdir}, \
    --includedir=${includedir}, \
    --oldincludedir=${includedir}, \
    --infodir=${infodir}, \
    --mandir=${mandir}, \
    --with-libtool-sysroot=${STAGING_DIR_HOST} \
    "

export PYTHON_CPPFLAGS = "-I${STAGING_INCDIR}/${PYTHON_DIR}"

inherit python3native pkgconfig setuptools3

If you accept to support Yocto/OpenEmbedded-Core build system, please help to upstream it.

Thanks-Jamin

[osandov]

Hi, can you please clarify what you're requesting here? I have no objections to OpenEmbedded support, but I don't plan to submit that recipe to OpenEmbedded-Core myself.

However, if you're having issues integrating drgn's build process with OpenEmbedded, please provide details and I'd be more than happy to help with that.

[jamin-aspeed]

Thanks for suggestion.

I am asking this request because OpenEmbedded do not have drgn receipe and users cannot build it. So, I created this python3-drgb.bb to successfully build this package with OE in my project.

I don't have a plan to upstream this recipe in OE because I am not the owner of this repository. Just share this recipe to drgn owner if they want to support OE and upstream it in the future.

Thanks-Jamin

[osandov]

I assume that anyone is able to submit a recipe to OpenEmbedded, not just the upstream package maintainers, right? I don't have any experience with OpenEmbedded, so I'd need a volunteer. Side note, https://github.com/openembedded/meta-openembedded/tree/master/meta-oe/recipes-kernel seems like a better location for the recipe since that's where related tools like crash and makedumpfile are.

[jamin-aspeed]

Thanks for suggestion and review. Got it. I will help to upstream it and update progress here. If OE reviewer have a question about this repo design, can I ask you for help here?

Jamin

[osandov]

Absolutely. Feel free to Cc/tag me on anything you send.

[jamin-aspeed]

Hi @osandov

Do you know how to fix it? I run drgn with kernel debug symbol file but I encountered this warning. The latest python version in OE is 3.12. Thanks-Jamin

drgn -s /tmp/vmlinu
drgn 0.0.27 (using Python 3.12.3, elfutils 0.191, without libkdumpfile)
warning: missing some debugging symbols (see https://drgn.readthedocs.io/en/latest/getting_debugging_symbols.html):
  kernel modules (could not find loaded kernel modules: 'struct list_head' has no member 'next')

drgn 0.0.27 (using Python 3.12.3, elfutils 0.191, without libkdumpfile)
warning: missing some debugging symbols (see https://drgn.readthedocs.io/en/latest/getting_debugging_symbols.html):
  kernel modules (could not find loaded kernel modules: 'struct list_head' has no member 'next')
For help, type help(drgn).
>>> import drgn
>>> from drgn import FaultError, NULL, Object, cast, container_of, execscript, offsetof, reinterpret, sizeof, stack_trace
>>> from drgn.helpers.common import *
>>> from drgn.helpers.linux import *
>>> prog['init_task'].comm
Traceback (most recent call last):
  File "<console>", line 1, in <module>
AttributeError: 'struct task_struct' has no member 'comm'

Sorry, it seems I should place kernel debug symbol file into the following directories. After, I placed it in the following directory, this issue had been fixed.

/usr/lib/debug/boot/vmlinux-%s,
/usr/lib/debug/lib/modules/%s/vmlinux,

However, after I run these workqueue python script from kernel, I encountered the following errors from drgm. https://github.com/torvalds/linux/tree/master/tools/workqueue Could you give me any suggestion?

./wq_monitor.py
warning: missing some debugging symbols (see https://drgn.readthedocs.io/en/latest/getting_debugging_symbols.html):
  kernel modules (could not find loaded kernel modules: 'struct list_head' has no member 'next')

                            total  infl  CPUtime CPUitsv CMW/RPR  mayday rescued
Traceback (most recent call last):
  File "/usr/bin/drgn", line 8, in <module>
    sys.exit(_main())
             ^^^^^^^
  File "/usr/lib/python3.12/site-packages/drgn/cli.py", line 324, in _main
    runpy.run_path(script, init_globals={"prog": prog}, run_name="__main__")
  File "<frozen runpy>", line 286, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "./wq_monitor.py", line 175, in <module>
    main()
  File "./wq_monitor.py", line 161, in main
    for wq in list_for_each_entry('struct workqueue_struct', workqueues.address_of_(), 'list'):
  File "/usr/lib/python3.12/site-packages/drgn/helpers/linux/list.py", line 183, in list_for_each_entry
    for pos in list_for_each(head):
  File "/usr/lib/python3.12/site-packages/drgn/helpers/linux/list.py", line 151, in list_for_each
    pos = head.next.read_()
          ^^^^^^^^^
AttributeError: 'struct list_head' has no member 'next'

./wq_dump.py
warning: missing some debugging symbols (see https://drgn.readthedocs.io/en/latest/getting_debugging_symbols.html):
  kernel modules (could not find loaded kernel modules: 'struct list_head' has no member 'next')
./wq_dump.py:151: SyntaxWarning: invalid escape sequence '\ '
  print('[    workqueue     \     type   CPU', end='')
Affinity Scopes
===============
wq_unbound_cpumask=0000000f

CPU
  nr_pods  4
  pod_cpus [0]=00000001 [1]=00000002 [2]=00000004 [3]=00000008
  pod_node [0]=0 [1]=0 [2]=0 [3]=0
  cpu_pod  [0]=0 [1]=1 [2]=2 [3]=3

SMT
  nr_pods  4
  pod_cpus [0]=00000001 [1]=00000002 [2]=00000004 [3]=00000008
  pod_node [0]=0 [1]=0 [2]=0 [3]=0
  cpu_pod  [0]=0 [1]=1 [2]=2 [3]=3

CACHE (default)
  nr_pods  4
  pod_cpus [0]=00000001 [1]=00000002 [2]=00000004 [3]=00000008
  pod_node [0]=0 [1]=0 [2]=0 [3]=0
  cpu_pod  [0]=0 [1]=1 [2]=2 [3]=3

NUMA
  nr_pods  1
  pod_cpus [0]=0000000f
  pod_node [0]=0
  cpu_pod  [0]=0 [1]=0 [2]=0 [3]=0

SYSTEM
  nr_pods  1
  pod_cpus [0]=0000000f
  pod_node [0]=-1
  cpu_pod  [0]=0 [1]=0 [2]=0 [3]=0

Worker Pools
============
pool[00] ref= 1 nice=  0 idle/workers=  3/  3 cpu=  0
pool[01] ref= 1 nice=-20 idle/workers=  2/  2 cpu=  0
pool[02] ref= 1 nice=  0 idle/workers=  2/  2 cpu=  1
pool[03] ref= 1 nice=-20 idle/workers=  2/  2 cpu=  1
pool[04] ref= 1 nice=  0 idle/workers=  2/  2 cpu=  2
pool[05] ref= 1 nice=-20 idle/workers=  2/  2 cpu=  2
pool[06] ref= 1 nice=  0 idle/workers=  2/  2 cpu=  3
pool[07] ref= 1 nice=-20 idle/workers=  2/  2 cpu=  3
pool[08] ref=11 nice=  0 idle/workers=  2/  2 cpus=0000000f pod_cpus=0000000f
pool[09] ref= 4 nice=  0 idle/workers=  3/  3 cpus=0000000f pod_cpus=00000001
pool[10] ref= 4 nice=  0 idle/workers=  2/  2 cpus=0000000f pod_cpus=00000002
pool[11] ref= 4 nice=  0 idle/workers=  3/  3 cpus=0000000f pod_cpus=00000004
pool[12] ref= 4 nice=  0 idle/workers=  2/  2 cpus=0000000f pod_cpus=00000008
pool[13] ref=18 nice=-20 idle/workers=  1/  1 cpus=0000000f pod_cpus=0000000f
pool[14] ref=18 nice=-20 idle/workers=  1/  1 cpus=0000000f pod_cpus=00000001
pool[15] ref=18 nice=-20 idle/workers=  1/  1 cpus=0000000f pod_cpus=00000002
pool[16] ref=18 nice=-20 idle/workers=  1/  1 cpus=0000000f pod_cpus=00000004
pool[17] ref=18 nice=-20 idle/workers=  1/  1 cpus=0000000f pod_cpus=00000008

Workqueue CPU -> pool
=====================
[    workqueue     \     type   CPU  0  1  2  3 dfl]
Traceback (most recent call last):
  File "/usr/bin/drgn", line 8, in <module>
    sys.exit(_main())
             ^^^^^^^
  File "/usr/lib/python3.12/site-packages/drgn/cli.py", line 324, in _main
    runpy.run_path(script, init_globals={"prog": prog}, run_name="__main__")
  File "<frozen runpy>", line 286, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "./wq_dump.py", line 156, in <module>
    for wq in list_for_each_entry('struct workqueue_struct', workqueues.address_of_(), 'list'):
  File "/usr/lib/python3.12/site-packages/drgn/helpers/linux/list.py", line 183, in list_for_each_entry
    for pos in list_for_each(head):
  File "/usr/lib/python3.12/site-packages/drgn/helpers/linux/list.py", line 151, in list_for_each
    pos = head.next.read_()
          ^^^^^^^^^
AttributeError: 'struct list_head' has no member 'next

https://github.com/osandov/drgn/blob/main/drgn/helpers/linux/list.py#L151

Thanks-Jamin

[jamin-aspeed]

send path to OE (v1)

• https://patchwork.yoctoproject.org/project/oe/list/?series=27288
• https://patchwork.yoctoproject.org/project/oe/patch/20240820054044.1538795-2-jamin_lin@aspeedtech.com/

[jamin-aspeed]

send patch to OE (v2)

• https://patchwork.yoctoproject.org/project/oe/patch/20240822091940.1512585-2-jamin_lin@aspeedtech.com/
• https://patchwork.yoctoproject.org/project/oe/patch/20240822091940.1512585-1-jamin_lin@aspeedtech.com/

[jamin-aspeed]

@osandov

This recipe has been merged into meta-openembedded, so I think this issue can be close.

• https://github.com/openembedded/meta-openembedded/commit/c42a0fb650326bb3894fe2ab2d7915aa4dfc1c73
• https://github.com/openembedded/meta-openembedded/blob/master/meta-oe/recipes-kernel/drgn/python3-drgn_0.0.27.bb

Thanks-Jamin

[osandov]

Thank you for getting that merged! FYI, I posted about it on Mastodon: https://fosstodon.org/@osandov/113035633926750896.