MMIO-based device pass-thru

taoyuhong commented 3 years ago

The hypervisor shall support pass-thru a device with sets of page aligned MMIO resources to guest (pre-launched or post-launched), with pre-condition: there is no DMA operation and interrupt will be triggered by such device.

change high RAM start address to 4GB for post-launched VM change 64-bit mmio address space to 256G-512G for post-launched VM

taoyuhong commented 3 years ago

[External_System_ID] ACRN-6526

NanlinXie commented 3 years ago

Doc Impact Analysis:

[ ] New feature -> new HLD and usage guide
[ ] HLDs
[ ] Configuration data
[ ] Kernel module parameters
[x] Device Model (acrn-dm) parameters
[ ] Hypervisor shell commands
[ ] ACRN tools (debug and/or service)
[ ] Build system
[ ] Other documents (e.g. tutorials)

gvancuts commented 3 years ago

What about this HLD: https://projectacrn.github.io/latest/developer-guides/hld/mmio-dev-passthrough.html, should it be updated too based on these changes?

taoyuhong commented 3 years ago

These changes have no impact with HLD,

NanlinXie commented 3 years ago

@gvancuts , could we change to doc_done?

gvancuts commented 3 years ago

@gvancuts , could we change to doc_done?

It looks like these patches at least affect the Device Model HLD, e.g. the part where the e820 memmap is described.

I haven't done a very thorough check but it's the only one I found that needs updating.

NanlinXie commented 3 years ago

@JasonChenCJ, any comment?

JasonChenCJ commented 3 years ago

These two patches are actually not directly related with MMIO device pass-thru..

gvancuts commented 3 years ago

These do not match for example:

NanlinXie commented 3 years ago

@JasonChenCJ and @taoyuhong , any update according to @gvancuts's comment?

taoyuhong commented 3 years ago

"change high RAM start address to 4GB for post-launched VM" is related change, but that change matches HLD:

5: 0xe00_00000 - fff_fffff (reserved) 512MB
6: 1_000_00000 - highmem RAM highmem-4G */ before "change high RAM start address to 4GB for post-launched VM", highmem start at 5G, that does not match HLD, need update. But after #5913, we change highmem back to 4G, match HLD again

gvancuts commented 3 years ago

I was looking at the two tables but they are different today

From: https://github.com/projectacrn/acrn-hypervisor/blob/master/doc/developer-guides/hld/hld-devicemodel.rst

/*
 * Default e820 mem map:
 *
 * there is reserved memory hole for PCI hole and APIC etc
 * so the memory layout could be separated into lowmem & highmem.
 * - if request memory size <= ctx->lowmem_limit, then there is only
 *   map[0]:0~ctx->lowmem for RAM
 *   ctx->lowmem = request_memory_size
 * - if request memory size > ctx->lowmem_limit, then there are
 *   map[0]:0~ctx->lowmem_limit & map[2]:4G~ctx->highmem for RAM
 *   ctx->highmem = request_memory_size - ctx->lowmem_limit
 *
 *             Begin      End         Type         Length
 * 0:             0 -     0xef000     RAM          0xEF000
 * 1        0xef000 -     0x100000    (reserved)   0x11000
 * 2       0x100000 -     lowmem      RAM          lowmem - 0x100000
 * 3:        lowmem -     bff_fffff   (reserved)   0xc00_00000-lowmem
 * 4:   0xc00_00000 -     dff_fffff   PCI hole     512MB
 * 5:   0xe00_00000 -     fff_fffff   (reserved)   512MB
 * 6:   1_000_00000 -     highmem     RAM          highmem-4G
 */

From https://github.com/projectacrn/acrn-hypervisor/blob/master/devicemodel/core/sw_load_common.c

/*
 * Default e820 mem map:
 *
 * there is reserved memory hole for PCI hole and APIC etc
 * so the memory layout could be separated into lowmem & highmem.
 * - if request memory size <= ctx->lowmem_limit, then there is only
 *   map[0]:0~ctx->lowmem for RAM
 *   ctx->lowmem = request_memory_size
 * - if request memory size > ctx->lowmem_limit, then there are
 *   map[0]:0~ctx->lowmem_limit & map[2]:4G~ctx->highmem for RAM
 *   ctx->highmem = request_memory_size - ctx->lowmem_limit
 *
 *             Begin    Limit        Type            Length
 * 0:              0 -  0xA0000      RAM             0xA0000
 * 1:       0x100000 -  lowmem part1 RAM             0x0
 * 2:   SW SRAM_bot  -  SW SRAM_top  (reserved)      SOFTWARE_SRAM_MAX_SIZE
 * 3:   gpu_rsvd_bot -  gpu_rsvd_top (reserved)      0x4004000
 * 4:   lowmem part2 -  0x80000000   (reserved)      0x0
 * 5:     0xE0000000 -  0x100000000  MCFG, MMIO      512MB
 * 6:  HIGHRAM_START_ADDR -  mmio64 start  RAM       ctx->highmem
 *
 * FIXME: Do we need to reserve DSM and OPREGION for GVTD here.
 */

But after #5913, we change highmem back to 4G, match HLD again

This is a reference to this ticket, did you mean another one?

taoyuhong commented 3 years ago

The e820 layout has changed, does not match HLD, need update HLD. I just mean these mismatch does not caused by mmio device passthrough patches. Mismatch has happened since commit 643513f3d484454fc5d0ca627e3b94da952c7198 Author: Peter Fang peter.fang@intel.com Date: Fri Mar 15 01:59:56 2019 -0700

And after that it keep changed several times. I am not sure if it will change again soon. So I don't know if we should update HLD now.

gvancuts commented 3 years ago

The e820 layout has changed, does not match HLD, need update HLD. I just mean these mismatch does not caused by mmio device passthrough patches. Mismatch has happened since commit 643513f Author: Peter Fang peter.fang@intel.com Date: Fri Mar 15 01:59:56 2019 -0700

Oh, I see. Thanks for the clarification.

And after that it keep changed several times. I am not sure if it will change again soon. So I don't know if we should update HLD now.

We should. And we should again the next time it changes ;-)

liuhang-bit commented 3 years ago

https://github.com/projectacrn/acrn-hypervisor/pull/6404 this PR fix the issue

NanlinXie commented 3 years ago

@gvancuts , please confirm whether we could change to doc:done.

gvancuts commented 3 years ago

@gvancuts , please confirm whether we could change to doc:done.

Yes

amyreye commented 2 years ago

@NanlinXie @dbkinder Can we close this issue?

projectacrn / acrn-hypervisor

MMIO-based device pass-thru #5913