Allow device tree overlays to override reg of /memory@0, expose SDRAM banks as separate memory regions

jarikirma commented 2 months ago

Describe the bug

This report is inspired by https://github.com/raspberrypi/linux/pull/6273 and the not so sympathetic reaction to it on the Linux kernel mailing list.

I attempted to accomplish what fake NUMA implementation in the patch did by overriding memory settings on the device tree using an overlay. I'd assume this to accomplish the task on my RPi 5 with 8 GB of RAM:

/dts-v1/;

/ {
 compatible =  "raspberrypi,5-model-b";
 fragment@0 {
  target-path = "/";
  __overlay__ {
   #address-cells = <0x00000002>;
   #size-cells = <0x00000001>;
   memory@0 {
    device_type = "memory";   
    reg = <0x00000000 0x00000000 0x3f800000 0x00000000 0x40000000 0x40000000>;
    numa-node-id = <0>;
   };
   memory@80000000 {
    device_type = "memory";   
    reg = <0x00000000 0x80000000 0x80000000>;
    numa-node-id = <1>;
   };
   memory@100000000 {
    device_type = "memory";   
    reg = <0x00000001 0x00000000 0x80000000>;
    numa-node-id = <2>;
   };
   memory@180000000 {
    device_type = "memory";   
    reg = <0x00000001 0x80000000 0x80000000>;
    numa-node-id = <3>;
   };
  };
 };
};

The issue here is that reg in /memory@0 apparently can't be overridden, or at least I failed at that, although numa-node-ids work.

Thus, I have two suggestions:

Allow memory size on region to be overridden (and possibly intelligently carve a hole in it if there's reserved memory on the region).
Even better, expose separate memory banks as separate memory regions, possibly even with numa-node-ids. Since memory information is read in the kernel very early in the boot process, this would be the most convenient way to balance memory allocations across banks without particularly intrusive (or any!) changes to the kernel mainline.

Of course this behaviour could be controlled by a bootloader configuration flag, if necessary.

Steps to reproduce the behaviour

Attempt using an overlay like

/dts-v1/;

/ {
 compatible =  "raspberrypi,5-model-b";
 fragment@0 {
  target-path = "/";
  __overlay__ {
   #address-cells = <0x00000002>;
   #size-cells = <0x00000001>;
   memory@0 {
    device_type = "memory";   
    reg = <0x00000000 0x00000000 0x3f800000 0x00000000 0x40000000 0x40000000>;
    numa-node-id = <0>;
   };
  };
 };
};

... and see how numa-node-id successfully appears on the device tree, but reg value comes from the bootloader.

Device (s)

Raspberry Pi 5

Bootloader configuration.

[all]
BOOT_UART=1
POWER_OFF_ON_HALT=0
BOOT_ORDER=0xf641

System

# rpi-eeprom-config
Raspberry Pi reference 2024-07-04
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 48efb5fc5485fafdc9de8ad481eb5c09e1182656, stage4

# vcgencmd bootloader_version
2024/06/05 16:41:49
version 6fe0b091c7cb1c5da26b7f41cc24c42840531a83 (release)
timestamp 1717602109
update-time 1722921424
capabilities 0x0000007f

# vcgencmd version
2024/06/05 16:41:49 
Copyright (c) 2012 Broadcom
version 6fe0b091 (release) (embedded)

# uname -a
Linux raspberrypi 6.6.31+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.31-1+rpt1 (2024-05-29) aarch64 GNU/Linux

Bootloader logs

No response

USB boot

No response

NVMe boot

No response

Network (TFTP boot)

No response

pelwell commented 2 months ago

on my RPi 5 with 8 GB of RAM:

The fact that you have to refer to the RAM size when talking about your overlay should give you an idea of why some of the changes from your overlay get overridden: there is one DTB per model of Pi, which the firmware patches to cater for different RAM sizes. There is some potential flexibility about the sequence of DT modifications, but making changes to that sequence often has unexpected consequences.

If you want to experiment with a hand-crafted DTB, you could install U-boot and let it load your DTB and pass it to the kernel unmolested.

jarikirma commented 2 months ago

Maybe the fact that I consider the above overlay a horrible hack was a bit too much of an implied assumption. Yet, not being able to override the value is at least rather inconsistent behaviour - but I do agree that modifying this might have unintended side effects. There are certainly many possibilities of shooting oneself on the foot with custom overlays already...

If the bootloader would describe separate memory banks as separate memory instances in the device tree it could allow kernel to make more informed decisions without questionable kernel-side patches. Especially if the performance impact of page allocation interleaving between banks on RPi 5 is really as big as implied in some benchmark comments.

raspberrypi / rpi-eeprom