Add hook for EmbeddedDevice XRT devices to allow moving the PSDDR memory region

[ld-cd]

By default the EmbeddedDevice seems to have a placeholder PSDDR region that gets added onto any other memory regions in the address map if no .xclbin file is provided. zocl of course doesn't find a reserved 256 Meg region at address 0 and falls back to the kernel CMA region which seems to be limited to 256M on most boards and is by default only 128M of which ~8M is already used by the kernel. This behavior is a tad confusing to the end user and means that only 120M of memory can by default be allocated using the default allocate region.

In addition I'm not really sure what would happen if all or near all of the CMA region were allocated and the kernel needs some more (for e.g. a framebuffer on display hotplug), but I can't imagine it would be good things and this seems difficult to debug.

This patch allows one to point the default allocation region at any reserved region that XRT is willing to allocate from. On the RFSoC 4x2 I created such a region in the bottom half of the upper two gigs of PS DDR by adding:

        reserved-memory {
                #address-cells = <2>;
                #size-cells = <2>;

                ranges;

                bufferreserved: buffer@0 {
                        no-map;

                        reg = <0x08 0x00000000 0x00 0x40000000>;
                };
        };

        zocl@0 {
                compatible = "xlnx,zocl";
                memory-region = <&bufferreserved>;
        };

To the device tree, on Alveo platforms this appears to be done by allocating hugepages on boot and using xbutil but this doesn't seem to be supported on Zynq platforms.

To use it you call the set_psddr_region(addr, size) method on an EmbeddedDevice to give it the address of a region you want to use before downloading an overlay. It does check that everything is page aligned but does not check that the region is actually in the PSDDR or that it is unused by other drivers however those checks seem like they would be better placed in zocl if they aren't already there.

It also does not check that the overlay hasn't been downloaded, is this a robust way to do that?

[ld-cd]

An alternative approach I considered was adjusting this to check for the name "PSDDR" instead of the address zero and including .xclbin file along the bitstream, but that felt like a bit of a cludge.

[skalade]

Hi @ld-cd, thanks for the contribution! Looks like this'll affect quite a few boards, so we'll need some time to evaluate the changes. Thanks! Shawn

[ld-cd]

Yeah that's what I assumed, I've tested the change on our 4x2, will give it a test on our ZCU111

[ld-cd]

I've tested this on our ZCU111 and RFSoC4x2 and it appears to work on both;

Is there anything we can do to help get this functionality or something equivalent to it into the next PYNQ release?

[skalade]

Right now pynq only supports CMA-based allocation, see device tree snippet below. https://github.com/Xilinx/PYNQ/blob/491350a321bf9bb2feb379c8285e5db0056dbce3/boards/ZCU104/petalinux_bsp/meta-user/recipes-bsp/device-tree/files/system-user.dtsi#L11

You have an alternate way of using reserved memory that we probably can't support in parallel to CMA based allocation, as we support 30+ boards with only 1 device tree each.