Изучить маппинг память для mw типа DIR

[AndrewGavril]

• куда и как (функция) пишется информация в устройство об оффсетах
• получить эти данные (об оффсетах) на стороне qemu

[mxkrsv]

The general mapping algorithm is as follows:

1. Allocate a memory region
2. Somehow send the translation address to the peer
3. Peer does ntb_peer_mw_set_trans() with a received translation address to write it to the NTB config
4. Peer maps an outbound memory window (ntb_peer_mw_get_addr() involved)

This is the "outbound translation". Inbound translation is not supported by ntb_hw_idt driver (lacks ntb_mw_set_trans() etc.).

Implementation details

idt_ntb_peer_mw_set_trans()

This function writes the Translated Base Address (64-bit) to two 32-bit NT endpoint registers (BARLTBASE* is the lower part, BARUTBASE* is the upper).

idt_ntb_peer_mw_get_addr()

This function returns the physical memory address to where the outbound memory window is mapped (and its size).

That address and size are calculated relatively to the BAR PCI bus resource start address by shifting it by a certain constant value (known after idt_scan_mws()).

[mxkrsv]

There is also a trivial ntb_tool interface which may replicate that:

VM1:

# echo 16384 >peer0/mw_trans0
# cat peer0/mw_trans0
<addr>
# echo hello >peer0/mw0

VM2:

# echo <addr>:16384 >peer0/peer_mw_trans0
# head -c7 peer0/peer_mw0
hello

[mxkrsv]

Though I could not understand how BARs are initialized.

BARSETUP0 value is hard-coded on the QEMU side (4KiB, memory space, ...), which is fine and compliant to the IDT specification.

Others, however, are not touched in the driver at all, but are said to be configured somehow by the IDT specification. Listed methods are via EEPROM or modified BIOS(??).

The manual also says that BARx registers are writeable (page 497-...), and there are some writes to something with the same addresses:

IVSHMEM: Writing config value 0xfebd5000 to address 0x10

[mirrin00]

Others, however, are not touched in the driver at all, but are said to be configured somehow by the IDT specification. Listed methods are via EEPROM or modified BIOS(??).

Yes, in the real world BARs have to be enabled on the board (with BIOS or other methods). But it seems that in QEMU this is not necessary, and all BARs are activated by default.

BARSETUP0 values are hard-coded on the QEMU side (4KiB, memory space, ...), which is fine and compliant to the IDT specification.

BARSETUP0 is a 32-bit value in device memory. It is not the same as BAR0, which is 4KB space to communicate with IDT device.

I guess that solution is to map memory region as it is done with BAR0 and BAR2 in ivshmem device. Then when the driver tries to read/write to these BARs it will trigger functions in QEMU

[mxkrsv]

What size do we want to allocate per BAR?

Furthermore, subsequent even and odd bars can be paired to form a 64-bit bar, what do we want to do about that?

[mirrin00]

What size do we want to allocate per BAR?

I see three ways:

1. QEMU allows allocate memory region with size 0, i.e. no memory allocation, only callbacks (maybe there are other functions besides memory_region_init_io for this)
2. Allocate memory region with small size (e.g. 1 or 10 bytes). But you need to check that QEMU doesn't truncate addresses for this. For example, if memsize is 10 bytes and address request is 0xFF, QEMU may truncate that address to 0x5. This can be verified with test example or by exploring source code
3. Allocate enough memory for all possible number of windows at maximum size. (Least preferred option)

Furthermore, subsequent even and odd bars can be paired to form a 64-bit bar, what do we want to do about that?

I think I read about it in IDT documentation. Addressing an incorrect offset or using odd BARs in 64-bit mode has to cause an error to be returned

[AndrewGavril]

What size do we want to allocate per BAR?

Furthermore, subsequent even and odd bars can be paired to form a 64-bit bar, what do we want to do about that?

We have two main scenarious:

• 64b transmittion
• up to 64kb transmittion

Some initialization steps are described here: https://www.renesas.com/us/en/document/apn/510-use-non-transpared-bridging-idt-pci-express-ntb-gen1-switches

[mxkrsv]

By the way, in DIR mode there's really only 1 MW per BAR: https://elixir.bootlin.com/linux/v6.5.3/C/ident/idt_get_mw_count

It, in turn, means that the mapping physical address on the side that calls idt_ntb_peer_mw_set_trans() is exactly equal to the BAR base address.

[AndrewGavril]

Yes, i think we can configure all BARs mw in DIR mode to simplify the task

By the way, in DIR mode there's really only 1 MW per BAR: https://elixir.bootlin.com/linux/v6.5.3/C/ident/idt_get_mw_count

Great, now we should find out the offset It, in turn, means that the mapping physical address on the side that calls idt_ntb_peer_mw_set_trans() is exactly equal to the BAR base address.

[mxkrsv]

Great, now we should find out the offset

What is meant by the offset here? The offset of that single MW relatively to the BAR base address is zero.

[AndrewGavril]

Great, now we should find out the offset

What is meant by the offset here? The offset of that single MW relatively to the BAR base address is zero.

Ok, understood