search

hermit-os / kernel

A Rust-based, lightweight unikernel.
http://hermit-os.org
Apache License 2.0
1.2k stars 86 forks source link

Support 4GiB of RAM with 2MiB pages #490

Open jounathaen opened 2 years ago

jounathaen commented 2 years ago

I call my application with

/usr/libexec/qemu-kvm \                                                                                                                                                                                              
    -cpu qemu64,apic,fsgsbase,rdtscp,xsave,xsaveopt,fxsr \                                                                                                                                                           
    -device isa-debug-exit,iobase=0xf4,iosize=0x04 \                                                                                                                                                                 
    -display none \                                                                                                                                                                                                  
    -serial stdio \                                                                                                                                                                                                  
    -kernel rusty-loader/target/x86_64/debug/rusty-loader \                                                                                                                                                          
    -netdev tap,id=net0,ifname=tap10,script=no,downscript=no,vhost=on \                                                                                                                                              
    -device virtio-net-pci,netdev=net0,disable-legacy=on \                                                                                                                                                           
    -smp 1 \                                                                                                                                                                                                         
    -m 3.5G \                                                                                                                                                                                                         
    -initrd target/x86_64-unknown-hermit/release/myapp

which fails with 

[LOADER] Loader: [0x100000 - 0x1200a8]                                                                                                                                                                               
[LOADER] Found Multiboot information at 0x9500                                                                                                                                                                       
[LOADER] Found module: [0x122000 - 0x1f33548]                                                                                                                                                                        
[LOADER] Module length: 0x1e11548                                                                                                                                                                                    
[LOADER] Found an ELF module at 0x122000                                                                                                                                                                             
[LOADER] Map 222 pages at 0x122000 (page size 4 KByte)                                                                                                                                                               
[LOADER] Map 15 pages at 0x200000 (page size 2048 KByte)                                                                                                                                                             
[LOADER] Parsing kernel from ELF at 0x122000..0x1f33548 (31528264 B)                                                                                                                                                 
[LOADER] Loading kernel to 0x2000000                                                                                                                                                                                 
[LOADER] TLS is at 0x22f4ba8..0x22f4c20 (120 B)                                                                                                                                                                      
[LOADER] Use stack address 0xa000                                                                                                                                                                                    
[LOADER] BootInfo located at 0x120000                                                                                                                                                                                
[LOADER] Jumping to HermitCore Application Entry Point at 0x2230b80                                                                                                                                                  
[0][INFO] Welcome to HermitCore-rs 0.4.3                                                                                                                                                                             
[0][INFO] Kernel starts at 0x2000000                                                                                                                                                                                 
[0][INFO] BSS starts at 0x2420000                                                                                                                                                                                    
[0][INFO] TLS starts at 0x22f4ba8 (size 120 Bytes)                                                                                                                                                                   
[0][INFO] Enable FSGSBASE support                                                                                                                                                                                    
[0][INFO] Found Multiboot info at 0x9500                                                                                                                                                                             
[0][INFO] Found cmdline at 0x121049 (size 46)                                                                                                                                                                        
[0][INFO] Total memory size: 14335 MB                                                                                                                                                                                
[0][INFO] Kernel region: [0x2000000 - 0x2600000]                                                                                                                                                                     
[0][INFO] A pure Rust application is running on top of HermitCore!                                                                                                                                                   
[0][INFO] Heap: size 12838 MB, start address 0x2600000                                                                                                                                                               
[0][INFO] Heap is located at 0x2600000 -- 0x324c00000 (0 Bytes unmapped)                                                                                                                                             
[0][INFO]                                                                                                                                                                                                            
[0][INFO] ===================== PHYSICAL MEMORY FREE LIST ======================                                                                                                                                     
[0][INFO] 0x0000004020C000 - 0x00000040400000                                                                                                                                                                        
[0][INFO] 0x000000BFC00000 - 0x000000BFFE0000                                                                                                                                                                        
[0][INFO] 0x000002BFC00000 - 0x000002C0000000                                                                                                                                                                        
[0][INFO] ======================================================================                                                                                                                                     
[0][INFO]                                                                                                                                                                                                            
[0][INFO]                                                                                                                                                                                                            
[0][INFO] ================== KERNEL VIRTUAL MEMORY FREE LIST ===================                                                                                                                                     
[0][INFO] 0x00000324C00000 - 0x00800000000000                                                                                                                                                                        
[0][INFO] ======================================================================                                                                                                                                     
[0][INFO]                                            
[0][INFO]                                            
[0][INFO] ========================== CPU INFORMATION ===========================
[0][INFO] Model:                   QEMU Virtual CPU version 2.5+
[0][INFO] Frequency:               3391 MHz (from Measurement)
[0][INFO] SpeedStep Technology:    Not Available
[0][INFO] Features:                MMX SSE SSE2 SSE3 MCE FXSR XSAVE RDTSCP CLFLUSH X2APIC HYPERVISOR FSGSBASE 
[0][INFO] Physical Address Width:  46 bits
[0][INFO] Linear Address Width:    48 bits
[0][INFO] Supports 1GiB Pages:     No
[0][INFO] ======================================================================
[0][INFO] 
[0][INFO] HermitCore-rs booted on 2022-07-13 15:31:26.0 +00:00:00
[0][WARN] PCI Device @8086:7000 has multiple functions! Currently only one is handled.
[0][INFO] 
[0][INFO] ======================== PCI BUS INFORMATION =========================
[0][INFO] 00:00 Host bridge [0600]: Intel Corporation 440FX - 82441FX PMC [Natoma] [8086:1237]
[0][INFO] 00:01 ISA bridge [0601]: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] [8086:7000]
[0][INFO] 00:02 VGA compatible controller [0300]: Unknown Vendor Unknown Device [1234:1111], MemoryBar: 0xfd000000 (size 0x1000000), MemoryBar: 0xfebd0000 (size 0x1000)
[0][INFO] 00:03 Ethernet controller [0200]: Red Hat, Inc. Virtio network device [1AF4:1041], IRQ 11, MemoryBar: 0xfebd1000 (size 0x1000), MemoryBar: 0xfe000000 (size 0x4000)
[0][INFO] ======================================================================
[0][INFO] 
[0][INFO] Found an ACPI revision 0 table at 0xF59E0 with OEM ID "BOCHS "
[0][INFO] IOAPIC v17 has 24 entries
[0][INFO] Disable IOAPIC timer
[0][INFO] 
[0][INFO] ===================== MULTIPROCESSOR INFORMATION =====================
[0][INFO] APIC in use:             x2APIC
[0][INFO] Initialized CPUs:        1
[0][INFO] ======================================================================
[0][INFO] 
[0][INFO] Compiled with PCI support
[0][INFO] Compiled with ACPI support
[0][INFO] Compiled with FSGSBASE support
[0][INFO] Compiled with SMP support
[0][INFO] HermitCore is running on common system!
[0][INFO] Found virtio network device with device id 0x1041
[0][WARN] Currently only mapping of 64 bit BAR's is supported!
[0][INFO] Non Virtio PCI capability with id 11 found. And NOT used.
[0][ERROR] Found virtio capability whose BAR is not mapped or non existing. Capability of type 5 and id 0 for device 1041, can not be used!
[0][INFO] Feature set wanted by network driver are in conformance with specification.
[0][INFO] Feature set wanted by network driver are in conformance with specification.
[0][INFO] Driver found a subset of features for virtio device 1041. Features are: [VIRTIO_NET_F_MAC, VIRTIO_NET_F_STATUS, VIRTIO_F_RING_INDIRECT_DESC, VIRTIO_F_VERSION_1]
[0][INFO] Features have been negotiated between virtio network device 1041 and driver.
[0][INFO] Created SplitVq: idx=0, size=256
[0][INFO] Created SplitVq: idx=1, size=256
[0][PANIC] panicked at 'called `Result::unwrap()` on an `Err` value: AllocError', src/mm/mod.rs:247:66

Works when using 3.49G or less.

mkroening commented 2 years ago

Running rusty-demo with more than that works fine for me:

qemu-system-x86_64 \
                        -cpu host -enable-kvm \
                        -smp 1 -m 8G \
                        -device isa-debug-exit,iobase=0xf4,iosize=0x04 \
                        -display none -serial stdio \
                        -kernel loader/target/x86_64/debug/rusty-loader \
                        -initrd target/x86_64-unknown-hermit/release/rusty_demo

This might be triggered by some specific allocation behavior of your app.

jounathaen commented 2 years ago

Sure, because the rusty-demo doesn't allocate any memory. If you change it to e.g.:

#[cfg(target_os = "hermit")]
use hermit_sys as _;

fn main() {
    println!("Hello World!");
    let mut v = vec!(1, 2);
    println!("{v:?}");
    v.push(3);
    println!("{v:?}");
}

it fails

mkroening commented 2 years ago

I still can't reproduce... :thinking:

For me, it works with 8G. Are you on latest master?

mkroening commented 2 years ago

While I did not find the root cause, I found a workaround:

The issue does not happen, if no network device is created or the host CPU model is used.

To properly solve this issue, it might be interesting to find what the triggering difference in CPU features between -cpu host and -cpu qemu64,apic,fsgsbase,rdtscp,xsave,xsaveopt,fxsr is.

mkroening commented 2 years ago

The issue can be worked around with activating 1GB pages with pdpe1gb.