Open bgoglin opened 1 year ago
Notes for putting CXL info in DAX and NUMA nodes:
For CXL RAM: /sys/bus/dax/devices/dax0.0 points to something like ../../../devices/platform/ACPI0017:00/root0/decoder0.0/region0/dax_region0/dax0.0/ ("/decoder" might be an easy to detect CXL here). cat /sys/bus/cxl/devices/region0/target0 (and possibly target1... if interleaving) => "decoder4.0" /sys/bus/cxl/devices/decoder4.0 points to ../../../devices/platform/ACPI0017:00/root0/port2/endpoint4/decoder4.0/ /sys/bus/cxl/devices/endpoint4 points to ../../../../../pci0000:0c/0000:0c:00.0/0000:0d:00.0/mem1/ By the way, cat /sys/bus/cxl/devices/decoder4.0/mode => "ram".
For CXL PMEM: /sys/bus/dax/devices/dax0.0 points to ../../../devices/platform/ACPI0017:00/root0/nvdimm-bridge0/ndbus0/region0/dax0.0/dax0.0/ (not sure nvdimm-bridge is CXL specific or not) /sys/bus/cxl/devices/region0 contains target0 (and target1... if interleave) contain decoder names just like in RAM case above (except mode contains "pmem" instead of "ram").
By the way, if we locate CXL devices behind DAX/NUMA from them, we could use their PCI NUMA locality in case the DAX locality isn't correctly set (it's the case for CXL RAM device in Qemu+Linux 6.2 right now).
Starting with 2.9 (or 2.8.1) CXL memory expanders (Type 3) are detected and non-ignored (they have a dedicated PCI class 0x0502). It's not clear whether CXL Type 2 (accelerators with coherent memory) or Type 1 (devices with coherent access to host memory) will have a dedicated PCI class. If not, we'll need to dig into sysfs and/or the (non-root) PCI config space to find out whether a PCI Bridge or Device is CXL or not.
For now, they are exposed as normal PCI objects. We could switch them to CXL objects instead, either with a "CXL" subtype (for PCIDev and bridges?) and/or with a CXL upstream/downstream type (in bridges only). It's not clear whether it's useful/important or not.