Build ARM image for Gem5 using Buildroot

[Dauto98]

Dear Cirosantilli2,

I am learning how to use Buildroot to generate a disk image for Gem5. Sorry for the very basic questions, but I skimmed through the buildroot docs and it is not clear to me:

• What does Buildroot exactly do? In the docs, it says Buildroot is able to generate a cross-compilation toolchain, a root filesystem, a Linux kernel image and a bootloader for your target. So it means the output of Buildroot make process, if copy byte by byte to the target embedded board, it just works?
• What is the package in Buildroot make command mean?
• If I want to generate something that can work with Gem5 ARM, which configuration should I use?

The reason I don't use your kernel cheat repo because I use Gem5 in my project and I want to have greater control over it. So I want to use Buildroot to generate images separately.

Thank you.

[cirosantilli2]

Dear Cirosantilli2,

I am learning how to use Buildroot to generate a disk image for Gem5. Sorry for the very basic questions, but I skimmed through the buildroot docs and it is not clear to me:

* What does Buildroot exactly do? In the docs, it says Buildroot is able to generate a cross-compilation toolchain, a root filesystem, a Linux kernel image and a bootloader for your target. So it means the output of Buildroot `make` process, if copy byte by byte to the target embedded board, it just works?

If think the answer is yes, I don't fully understand what you mean by copying. It generates e.g. an .ext4 rootfs file and a vmlinux file which the simulator takes as input.

See also this: https://cirosantilli.com/linux-kernel-module-cheat/#buildroot-hello-world

* What is the package in Buildroot make command mean?

I don't understand the question very well. Buildroot has a package system just like Debian (manages dependencies). You can select which package will be present in the generated root filesystem image or not. Minimal packages required for a boot to shell are included by default.

* If I want to generate something that can work with Gem5 ARM, which configuration should I use?

That repo basically does uses https://github.com/cirosantilli/linux-kernel-module-cheat/blob/6efe2fb4e1a9f9f06032560d57011829797c8478/buildroot_config/default + some likely optional customizations at https://github.com/cirosantilli/linux-kernel-module-cheat/blob/6efe2fb4e1a9f9f06032560d57011829797c8478/build-buildroot#L104

The reason I don't use your kernel cheat repo because I use Gem5 in my project and I want to have greater control over it. So I want to use Buildroot to generate images separately.

I understand. Have a look at the above sections.

If you don't manage to extract a working setup from just looking, first run my scripts, check that things are working, and then extract the final generated config and run commands from my scripts, once you have a working baseline it is usually easy to do that.

For the commands you can also use --dry-run: https://cirosantilli.com/linux-kernel-module-cheat/#dry-run Every shell command being ran is also printed to stdout by default by my scripts, so you can just copy them and minimize after a working setup is obtained.

My scripts are very simple, so extracting should be easy after a run.

Thank you.

[Dauto98]

If think the answer is yes, I don't fully understand what you mean by copying. It generates e.g. an .ext4 rootfs file and a vmlinux file which the simulator takes as input.

See also this: https://cirosantilli.com/linux-kernel-module-cheat/#buildroot-hello-world

Actually, before I think make will build the buildroot tool and then use the buildroot tool through the command line to generate a disk image.

Also, when I try to run your repo

./build --download-dependencies gem5-buildroot

to see what buildroot config you use for Gem5, I got the error:

AttributeError: '_Environ' object has no attribute 'has_key':

when building Gem5, the latest version of Gem5 requires Python3. The error above occurs due to the incompatibility between Python2 and Python3. However, I can't switch to Python2 because Gem5 requires Python3 to build now.

---

That repo basically does uses https://github.com/cirosantilli/linux-kernel-module-cheat/blob/6efe2fb4e1a9f9f06032560d57011829797c8478/buildroot_config/default + some likely optional customizations at https://github.com/cirosantilli/linux-kernel-module-cheat/blob/6efe2fb4e1a9f9f06032560d57011829797c8478/build-buildroot#L104

So I looked at this config file, and made corresponding changes in make nconfig. I got the image, but when I ran it in Gem5 I got this error

<3>[   11.489158] Starting init: /bin/sh exists but couldn't execute it (error -8)
<0>[   11.489168] Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.
<4>[   11.489175] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.18.0+ #1
<4>[   11.489179] Hardware name: ARM-Versatile Express
<4>[   11.489186] [<8011221c>] (unwind_backtrace) from [<8010d37c>] (show_stack+0x10/0x14)
<4>[   11.489193] [<8010d37c>] (show_stack) from [<8086c3e0>] (dump_stack+0x78/0x8c)
<4>[   11.489200] [<8086c3e0>] (dump_stack) from [<8012e728>] (panic+0xf0/0x25c)
<4>[   11.489207] [<8012e728>] (panic) from [<80880a50>] (kernel_init+0x104/0x114)
<4>[   11.489214] [<80880a50>] (kernel_init) from [<801010b4>] (ret_from_fork+0x14/0x20)
<4>[   11.489219] Exception stack(0x8e881fb0 to 0x8e881ff8)
<4>[   11.489225] 1fa0:                                     00000000 00000000 00000000 00000000
<4>[   11.489232] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<4>[   11.489239] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000

The command I used is

./build/ARM/gem5.opt configs/example/fs.py 
    --kernel=configs/arm/binaries/vmlinux.arm 
    --machine-type=VExpress_GEM5_V1 
    --dtb-file=system/arm/dt/armv7_gem5_v1_1cpu.dtb 
    --bootloader=configs/arm/binaries/boot.arm 
    --disk-image=../buildroot-2021.02/output/images/rootfs.ext4 
    --command-line 'earlyprintk=pl011,0x1c090000 lpj=19988480 rw loglevel=8 mem=256MB root=/dev/sda console_msg_format=syslog nokaslr norandmaps panic=-1 printk.devkmsg=on printk.time=y rw console=ttyAMA0 - lkmc_home=/lkmc'

Only the disk-image is built from Buildroot, the kernel and bootloader are taken from Gem5 website.

[cirosantilli2]

If think the answer is yes, I don't fully understand what you mean by copying. It generates e.g. an .ext4 rootfs file and a vmlinux file which the simulator takes as input. See also this: https://cirosantilli.com/linux-kernel-module-cheat/#buildroot-hello-world

Actually, before I think make will build the buildroot tool and then use the buildroot tool through the command line to generate a disk image.

Also, when I try to run your repo

./build --download-dependencies gem5-buildroot

to see what buildroot config you use for Gem5, I got the error:

AttributeError: '_Environ' object has no attribute 'has_key':

when building Gem5, the latest version of Gem5 requires Python3. The error above occurs due to the incompatibility between Python2 and Python3. However, I can't switch to Python2 because Gem5 requires Python3 to build now.

I don't understand why that happens, because all my scripts are python3 e.g. https://github.com/cirosantilli/linux-kernel-module-cheat/blob/d769513efc3825ac4321f6a23f5696b3288450b4/build#L1and the reference test system, Ubuntu 20.04, only has Python 3.

Can you provide the full backtrace? What's your OS?

In any case, if that is not working on your host, just use the Docker setup instead documented at https://github.com/cirosantilli/linux-kernel-module-cheat/tree/d769513efc3825ac4321f6a23f5696b3288450b4#docker as that one just can't fail (based on Ubuntu 20.04 currently):

sudo apt-get install docker
./run-docker create && \
./run-docker sh -- ./build --download-dependencies --arch aarch64 gem5-buildroot
./run-docker sh

then:

./run --arch aarch64 --emulator gem5

and on another shell:

./run-docker sh

and:

./gem5-shell -aA

I have just tested the docker setup from scratch on my Ubuntu 20.10 host and it just worked. Please ensure that you pull the latest commit d769513efc3825ac4321f6a23f5696b3288450b4 as I've just fixed a small problem.

The built files are placed under out.docker/ so you can then just extract any images from there easily.

That repo basically does uses https://github.com/cirosantilli/linux-kernel-module-cheat/blob/6efe2fb4e1a9f9f06032560d57011829797c8478/buildroot_config/default + some likely optional customizations at https://github.com/cirosantilli/linux-kernel-module-cheat/blob/6efe2fb4e1a9f9f06032560d57011829797c8478/build-buildroot#L104

So I looked at this config file, and made corresponding changes in make nconfig. I got the image, but when I ran it in Gem5 I got this error

<3>[   11.489158] Starting init: /bin/sh exists but couldn't execute it (error -8)
<0>[   11.489168] Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.
<4>[   11.489175] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.18.0+ #1
<4>[   11.489179] Hardware name: ARM-Versatile Express
<4>[   11.489186] [<8011221c>] (unwind_backtrace) from [<8010d37c>] (show_stack+0x10/0x14)
<4>[   11.489193] [<8010d37c>] (show_stack) from [<8086c3e0>] (dump_stack+0x78/0x8c)
<4>[   11.489200] [<8086c3e0>] (dump_stack) from [<8012e728>] (panic+0xf0/0x25c)
<4>[   11.489207] [<8012e728>] (panic) from [<80880a50>] (kernel_init+0x104/0x114)
<4>[   11.489214] [<80880a50>] (kernel_init) from [<801010b4>] (ret_from_fork+0x14/0x20)
<4>[   11.489219] Exception stack(0x8e881fb0 to 0x8e881ff8)
<4>[   11.489225] 1fa0:                                     00000000 00000000 00000000 00000000
<4>[   11.489232] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<4>[   11.489239] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000

The command I used is

./build/ARM/gem5.opt configs/example/fs.py 
    --kernel=configs/arm/binaries/vmlinux.arm 
    --machine-type=VExpress_GEM5_V1 
    --dtb-file=system/arm/dt/armv7_gem5_v1_1cpu.dtb 
    --bootloader=configs/arm/binaries/boot.arm 
    --disk-image=../buildroot-2021.02/output/images/rootfs.ext4 
    --command-line 'earlyprintk=pl011,0x1c090000 lpj=19988480 rw loglevel=8 mem=256MB root=/dev/sda console_msg_format=syslog nokaslr norandmaps panic=-1 printk.devkmsg=on printk.time=y rw console=ttyAMA0 - lkmc_home=/lkmc'

Only the disk-image is built from Buildroot, the kernel and bootloader are taken from Gem5 website.

You seem to be mixing up armv7 and aarch64 components, so that the kernel can't run /bin/sh because of that.

Do you want armv7 or aarch64? Highly recommend aarch64 if you can.

E.g. for aarch64 you would use binaries/boot.arm64 aarch64. The full command can be extracted from:

./run --arch aarch64 --emulator gem5 --dry-run

but of course, it might need some small root= init= tweaking for different images.

[cirosantilli2]

The upstream images should work too however. Nightly tests run them nightly. But I prefer buildroot since I can control exactly what is in the image.

[Dauto98]

I don't understand why that happens, because all my scripts are python3 e.g. https://github.com/cirosantilli/linux-kernel-module-cheat/blob/d769513efc3825ac4321f6a23f5696b3288450b4/build#L1and the reference test system, Ubuntu 20.04, only has Python 3.

Can you provide the full backtrace? What's your OS?

Below is the stack trace. I am using Ubuntu 18.04, having both Python 2.7 and Python3.8.8. Python3.8.8 is the default

AttributeError: '_Environ' object has no attribute 'has_key':
  File "/usr/lib/scons/SCons/Script/Main.py", line 1376:
    _exec_main(parser, values)
  File "/usr/lib/scons/SCons/Script/Main.py", line 1339:
    _main(parser)
  File "/usr/lib/scons/SCons/Script/Main.py", line 1111:
    if os.environ.has_key('DH_INTERNAL_OPTIONS'):
Traceback (most recent call last):
  File "./build", line 682, in <module>
    Main().cli()
  File "/home/dauto98/Repositories/linux-kernel-module-cheat/cli_function.py", line 267, in cli
    exit_status = self.cli_noexit(*args, **kwargs)
  File "/home/dauto98/Repositories/linux-kernel-module-cheat/cli_function.py", line 258, in cli_noexit
    return self._do_main(vars(args))
  File "/home/dauto98/Repositories/linux-kernel-module-cheat/cli_function.py", line 152, in _do_main
    return self.main(**self._get_args(kwargs))
  File "/home/dauto98/Repositories/linux-kernel-module-cheat/common.py", line 1540, in main
    ret = self.timed_main()
  File "./build", line 677, in timed_main
    ret = component.build(self.env['arch'])
  File "./build", line 55, in build
    return self.build_callback()
  File "./build", line 474, in f
    return lkmc.import_path.import_path_main(component_file)(**args)
  File "/home/dauto98/Repositories/linux-kernel-module-cheat/common.py", line 739, in __call__
    return super().__call__(**kwargs)
  File "/home/dauto98/Repositories/linux-kernel-module-cheat/cli_function.py", line 149, in __call__
    return self._do_main(kwargs)
  File "/home/dauto98/Repositories/linux-kernel-module-cheat/cli_function.py", line 152, in _do_main
    return self.main(**self._get_args(kwargs))
  File "/home/dauto98/Repositories/linux-kernel-module-cheat/common.py", line 1540, in main
    ret = self.timed_main()
  File "/home/dauto98/Repositories/linux-kernel-module-cheat/common.py", line 2059, in timed_main
    return self.build()
  File "/home/dauto98/Repositories/linux-kernel-module-cheat/build-m5", line 29, in build
    self.clean()
  File "/home/dauto98/Repositories/linux-kernel-module-cheat/build-m5", line 41, in clean
    self._get_make_cmd() + ['--clean', LF],
  File "/home/dauto98/Repositories/linux-kernel-module-cheat/shell_helpers.py", line 457, in run_cmd
    raise e
Exception: Command exited with status: 2

---

You seem to be mixing up armv7 and aarch64 components, so that the kernel can't run /bin/sh because of that.

Yes indeed I was mixing up ARMv7 with ARMv8, but after changing both kernel, bootloader, and dts file to aarch64, the error become

FATAL: kernel too old
<0>[    0.686302] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00
<0>[    0.686302] 
<4>[    0.686310] CPU: 0 PID: 1 Comm: init Not tainted 4.18.0+ #1
<4>[    0.686315] Hardware name: V2P-AARCH64 (DT)
<4>[    0.686319] Call trace:
<4>[    0.686325]  dump_backtrace+0x0/0x1c0
<4>[    0.686330]  show_stack+0x14/0x20
<4>[    0.686336]  dump_stack+0x8c/0xac
<4>[    0.686340]  panic+0x130/0x288
<4>[    0.686345]  complete_and_exit+0x0/0x20
<4>[    0.686351]  do_group_exit+0x38/0xa0
<4>[    0.686356]  __wake_up_parent+0x0/0x28
<4>[    0.686361]  el0_svc_naked+0x30/0x34
<0>[    0.686366] Kernel Offset: disabled
<0>[    0.686370] CPU features: 0x21c06492
<0>[    0.686374] Memory Limit: 256 MB
<4>[    0.686381] Reboot failed -- System halted

Here is how I make the image step by step

• I take your config file from here, copy it to .config https://github.com/cirosantilli/linux-kernel-module-cheat/blob/6efe2fb4e1a9f9f06032560d57011829797c8478/buildroot_config/default
• run make oldconfig to resolve unresolved symbols. I choose aarach64 little-endian, and the default for the rest
• run make
• run Gem5 with the disk image produced by buildroot

  sudo ./build/ARM/gem5.opt configs/example/fs.py 
  --kernel=configs/thesis/arm/binaries/vmlinux.arm64 
  --machine-type=VExpress_GEM5_V1 
  --dtb-file=system/arm/dt/armv8_gem5_v1_1cpu.dtb 
  --bootloader=configs/thesis/arm/binaries/boot_v2.arm64 
  --disk-image=../buildroot-2021.02/output/images/rootfs.ext2 
  --command-line 'earlyprintk=pl011,0x1c090000 lpj=19988480 rw loglevel=8 mem=256MB root=/dev/sda console_msg_format=syslog nokaslr norandmaps panic=-1 printk.devkmsg=on printk.time=y rw console=ttyAMA0 - lkmc_home=/lkmc'

  The kernel, bootloader is taken from Gem5 website.

Last time when I tried to build Linux kernel by buildroot, it required the defconfig file. I don't know what it is and where to find it. Probably use the kernel and bootloader generated by buildroot will work.

[Dauto98]

Btw, talking about Ubuntu version, I use Ubuntu 18.04, and you use Ubuntu 20.04. In the build script, you install python-is-python3 package. This package is available for 20.04, but not in 18.04. So I comment it out when I build on my machine, otherwise, the build will fail because apt-get cannot find the package.

[cirosantilli2]

I don't understand why that happens, because all my scripts are python3 e.g. https://github.com/cirosantilli/linux-kernel-module-cheat/blob/d769513efc3825ac4321f6a23f5696b3288450b4/build#L1and the reference test system, Ubuntu 20.04, only has Python 3. Can you provide the full backtrace? What's your OS?

Below is the stack trace. I am using Ubuntu 18.04, having both Python 2.7 and Python3.8.8. Python3.8.8 is the default

AttributeError: '_Environ' object has no attribute 'has_key':
  File "/usr/lib/scons/SCons/Script/Main.py", line 1376:
    _exec_main(parser, values)
  File "/usr/lib/scons/SCons/Script/Main.py", line 1339:
    _main(parser)
  File "/usr/lib/scons/SCons/Script/Main.py", line 1111:
    if os.environ.has_key('DH_INTERNAL_OPTIONS'):
Traceback (most recent call last):
  File "./build", line 682, in <module>
    Main().cli()
  File "/home/dauto98/Repositories/linux-kernel-module-cheat/cli_function.py", line 267, in cli
    exit_status = self.cli_noexit(*args, **kwargs)
  File "/home/dauto98/Repositories/linux-kernel-module-cheat/cli_function.py", line 258, in cli_noexit
    return self._do_main(vars(args))
  File "/home/dauto98/Repositories/linux-kernel-module-cheat/cli_function.py", line 152, in _do_main
    return self.main(**self._get_args(kwargs))
  File "/home/dauto98/Repositories/linux-kernel-module-cheat/common.py", line 1540, in main
    ret = self.timed_main()
  File "./build", line 677, in timed_main
    ret = component.build(self.env['arch'])
  File "./build", line 55, in build
    return self.build_callback()
  File "./build", line 474, in f
    return lkmc.import_path.import_path_main(component_file)(**args)
  File "/home/dauto98/Repositories/linux-kernel-module-cheat/common.py", line 739, in __call__
    return super().__call__(**kwargs)
  File "/home/dauto98/Repositories/linux-kernel-module-cheat/cli_function.py", line 149, in __call__
    return self._do_main(kwargs)
  File "/home/dauto98/Repositories/linux-kernel-module-cheat/cli_function.py", line 152, in _do_main
    return self.main(**self._get_args(kwargs))
  File "/home/dauto98/Repositories/linux-kernel-module-cheat/common.py", line 1540, in main
    ret = self.timed_main()
  File "/home/dauto98/Repositories/linux-kernel-module-cheat/common.py", line 2059, in timed_main
    return self.build()
  File "/home/dauto98/Repositories/linux-kernel-module-cheat/build-m5", line 29, in build
    self.clean()
  File "/home/dauto98/Repositories/linux-kernel-module-cheat/build-m5", line 41, in clean
    self._get_make_cmd() + ['--clean', LF],
  File "/home/dauto98/Repositories/linux-kernel-module-cheat/shell_helpers.py", line 457, in run_cmd
    raise e
Exception: Command exited with status: 2

You seem to be mixing up armv7 and aarch64 components, so that the kernel can't run /bin/sh because of that.

Yes indeed I was mixing up ARMv7 with ARMv8, but after changing both kernel, bootloader, and dts file to aarch64, the error become

FATAL: kernel too old
<0>[    0.686302] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00
<0>[    0.686302] 
<4>[    0.686310] CPU: 0 PID: 1 Comm: init Not tainted 4.18.0+ #1
<4>[    0.686315] Hardware name: V2P-AARCH64 (DT)
<4>[    0.686319] Call trace:
<4>[    0.686325]  dump_backtrace+0x0/0x1c0
<4>[    0.686330]  show_stack+0x14/0x20
<4>[    0.686336]  dump_stack+0x8c/0xac
<4>[    0.686340]  panic+0x130/0x288
<4>[    0.686345]  complete_and_exit+0x0/0x20
<4>[    0.686351]  do_group_exit+0x38/0xa0
<4>[    0.686356]  __wake_up_parent+0x0/0x28
<4>[    0.686361]  el0_svc_naked+0x30/0x34
<0>[    0.686366] Kernel Offset: disabled
<0>[    0.686370] CPU features: 0x21c06492
<0>[    0.686374] Memory Limit: 256 MB
<4>[    0.686381] Reboot failed -- System halted

As the message suggests, the kernel is too old. This is a glibc check, which checks for a minimum supported kernel version, and it blows up the shell otherwise. If you use a newer kernel boot will work. Related: https://cirosantilli.com/linux-kernel-module-cheat/#fatal-kernel-too-old-failure-in-userland-simulation

Here is how I make the image step by step

* I take your config file from here, copy it to `.config` https://github.com/cirosantilli/linux-kernel-module-cheat/blob/6efe2fb4e1a9f9f06032560d57011829797c8478/buildroot_config/default

* run `make oldconfig` to resolve unresolved symbols. I choose aarach64 little-endian, and the default for the rest

* run `make`

* run Gem5 with the disk image produced by buildroot

sudo ./build/ARM/gem5.opt configs/example/fs.py 
    --kernel=configs/thesis/arm/binaries/vmlinux.arm64 
    --machine-type=VExpress_GEM5_V1 
    --dtb-file=system/arm/dt/armv8_gem5_v1_1cpu.dtb 
    --bootloader=configs/thesis/arm/binaries/boot_v2.arm64 
    --disk-image=../buildroot-2021.02/output/images/rootfs.ext2 
    --command-line 'earlyprintk=pl011,0x1c090000 lpj=19988480 rw loglevel=8 mem=256MB root=/dev/sda console_msg_format=syslog nokaslr norandmaps panic=-1 printk.devkmsg=on printk.time=y rw console=ttyAMA0 - lkmc_home=/lkmc'

The kernel, bootloader is taken from Gem5 website.

Last time when I tried to build Linux kernel by buildroot, it required the defconfig file. I don't know what it is and where to find it. Probably use the kernel and bootloader generated by buildroot will work.

defconfig is basically the config you use to build the kernel. If some configs are missing it might not boot on gem5: https://cirosantilli.com/linux-kernel-module-cheat/#kernel-configs-about

Bootloader is part of gem5 BTW. And my scripts disable Buildroot's kernel build and build it directly for convenience, as usual you could extract build commands with ./build-kernel --arch aarch64 --dry-run.

[cirosantilli2]

Btw, talking about Ubuntu version, I use Ubuntu 18.04, and you use Ubuntu 20.04. In the build script, you install python-is-python3 package. This package is available for 20.04, but not in 18.04. So I comment it out when I build on my machine, otherwise, the build will fail because apt-get cannot find the package.

OK. Yes, those things require debugging, python2 vs 3 is a mess. I currently basically just maintain a single supported Ubuntu version, and Docker should always work (though generally porting is not hard).

[Dauto98]

Hi,

I have successfully built the image using the docker method you show above. However, I got weird interaction when ran Gem5 inside the docker I followed the step above. Everything went well, except when I log in to the Gem5 VM using the last command, it did get inside Gem5 but then it froze. I couldn't do anything. It didn't receive any input from the keyboard. It just froze.

But anyway, it worked great when I run by my own Gem5 build on the host, and using the kernel, disk-image, and bootloader in out.docker/. Thank you for the help. Now I will reverse engineer the script to see how it work.

[cirosantilli2]

Great news!

I reproduced the terminal issue on my Ubuntu 20.10 host outside of Docker, I hadn't tested that properly, I think it's the gem5 bug I now documented at: https://github.com/cirosantilli/linux-kernel-module-cheat/issues/147 I'm now looking for a version of gem5 that contains that and still builds to update the repo.

[cirosantilli2]

OK, that upstream commit fixed it and I updated LKMC. Some later commit after that one broke LLD build which is my default: https://gem5.atlassian.net/browse/GEM5-943 but I'll worry about that another day.