includeos / IncludeOS

A minimal, resource efficient unikernel for cloud services
https://www.includeos.org
Apache License 2.0
4.86k stars 358 forks source link

Add back 78 commits from master, make it boot. #2241

Closed alfreb closed 4 weeks ago

alfreb commented 1 month ago

Why?

The current master has been developed in the direction of fully integrating pthreads with the kernel, but it is not yet completed. It also has the side effect of deprecating 32-bit support, as well as the original SMP API where tasks could run fully in parallel with no concept of threads. After some discussion we've agreed that while the pthread kernel is an interesting approach that we'd some like to integrate as its own platform, we want to focus on the pure SMP idea as the main direction for now. The cleanest way to get there was to branch off from v0.15.1, and then cherry-pick in the recent work on nix on top of that.

The plan

Testing this branch

  1. nix-build example.nix
    • Will build a bootable example, without a bootloader.

Attach a bootloader and start qemu. One of two ways:

A) using vmrunner / boot tool

  1. Install vmrunner, with the latest non-conan patches, found in the non-conan branch
  2. Follow the instructions in the readme and add boot to your path
  3. boot -g --create-bridge ./result/bin/hello_includeos.elf.bin. This will attach a grub bootloader, create a network bridge (which requires sudo - and is really unnecessary in this case, but it's the default behavior of the tool as of now.). You should see IncludeOS output below.

B) Using just grubify.sh and qemu directly

Clone vmrunner non-conan branch mentioned above.

BIN=hello_includeos.elf.bin
cp ./result/bin/$BIN ./$BIN
chmod 755 ./$BIN
strip $BIN

# Attach grub bootloader (requires sudo and "grub-pc" installed)
<your-vmrunner-path>/bin/grubify.sh $BIN 

There's a helper script here, for reference, that will pick boot if it's present, otherwise use grubify. This, however, depends on grubify.sh to be present in the vmbuild directroy, which we currently don't have. https://github.com/alfreb/IncludeOS/blob/it-boots-v0-15-0/minimal_test.sh

Instead of reviving that script I think we should add vmrunner as a nix dependency for running tests and then have some sort of test.nix that does all of this. But let's revisit that once we've aligned on the new direction this PR is taking.

You should see something like this

Either approach should give the same output

================================================================================

                           #include<os> // Literally

================================================================================
     [ Kernel ] Stack: 0x1ffc38
     [ Kernel ] Boot magic: 0x2badb002, addr: 0x10000
     [ x86_64 ] Initializing paging
                * Supported page sizes: 4.000_KiB, 2.000_MiB
                * Adding 512 1GiB entries @ 0x0 -> 0x8000000000
                * Identity mapping 0 -> 0, size 16.000_GiB, flags 0x83 (8192 pages á 2.000_MiB)
                * Marking page 0 as not present
                * Allowing execute on 0x201000 -> 0x2d47f8
                * Passing page tables to CPU
<Multiboot>Booted with multiboot
    * Boot flags: 0x1a67
    * Valid memory (130559 Kib):
      0x00000000 - 0x0009fbff (639 Kib)
      0x00100000 - 0x07fdffff (129920 Kib)

    * Booted with parameters @ 0x10078:
    * Multiboot provided memory map  (7 entries @ 0x100a4)
      0x0000000000 - 0x000009fbff FREE (639 Kb.)
      0x000009fc00 - 0x000009ffff RESERVED (1 Kb.)
      0x00000f0000 - 0x00000fffff RESERVED (64 Kb.)
      0x0000100000 - 0x0007fdffff FREE (129920 Kb.)
      0x0007fe0000 - 0x0007ffffff RESERVED (128 Kb.)
      0x00feffc000 - 0x00feffffff RESERVED (16 Kb.)
      0x00fffc0000 - 0x00ffffffff RESERVED (256 Kb.)

     [ Kernel ] Total memory detected as 95.652_MiB
                Assigning fixed memory ranges (Memory map)
                * Assigning heap 0x339000 -> 0x5fa6fff
     [ Kernel ] Virtual memory map
                0x0000001000 - 0x0000007fff,  28.000_KiB (100.0 %) Page tables
                0x0000010000 - 0x000009d3ff, 565.000_KiB (100.0 %) Stack
                0x000009fc00 - 0x000009ffff,   1.000_KiB (100.0 %) Reserved (Multiboot)
                0x00000f0000 - 0x00000fffff,  64.000_KiB (100.0 %) Reserved (Multiboot)
                0x0000201000 - 0x00002d4fff, 848.000_KiB (100.0 %) ELF .text
                0x0000339000 - 0x0005fa6fff,  92.430_MiB (0.304 %) Dynamic memory
                0x0007fe0000 - 0x0007ffffff, 128.000_KiB (100.0 %) Reserved (Multiboot)
                0x00feffc000 - 0x00feffffff,  16.000_KiB (100.0 %) Reserved (Multiboot)
                0x00fffc0000 - 0x00ffffffff, 256.000_KiB (100.0 %) Reserved (Multiboot)
       [ ACPI ] Reading headers
                OEM: BOCHS  Rev. 0
       [ ACPI ] Reading APIC information
                LAPIC base: 0xfee00000  (flags: 0x1)
                I/O APIC 0   ADDR 0xfec00000  INTR 0x0
                IRQ redirect for bus 0 from IRQ 0 to VEC 2
                IRQ redirect for bus 0 from IRQ 5 to VEC 5
                IRQ redirect for bus 0 from IRQ 9 to VEC 9
                IRQ redirect for bus 0 from IRQ 10 to VEC 10
                IRQ redirect for bus 0 from IRQ 11 to VEC 11
        [ SMP ] Found 1 APs
        [ x86 ] Setting up GDT, TLS, IST
        [ x86 ] Initializing CPU 0
   [ I/O APIC ] Initializing ID=0
                Addr: 0xfec00000  Version: 0x11  Intr: 0  Entries: 24
     [ x2APIC ] Enabling x2APIC @ 0xfee00900
                APIC id: 0  ver: 50014
        [ x86 ] Enabling interrupts
        [ x86 ] Setting up kernel clock sources
        [ x86 ] KVM PV clocks initialized
                +--> 2999.998000 MHz
       [ APIC ] Measuring APIC timer...
                Enabled redirected entry 2 ioapic 0 -> 0 on apic 0
                Initializing drivers
[ PCI Manager ] Probing PCI bus
                |
                +--[ Bridge device, Intel (0x1237) ]
                +--[ Bridge device, Intel (0x7000) ]
                +--[ Storage controller, Intel (0x7010) ]
                +--[ Bridge device, Intel (0x7113) ]
                +--[ Display controller, Unknown vendor (0x1111) ]
                +--[ Network controller, VirtIO (0x1000) ]
                o
                |- Initializing Storage controller
                |--[ Storage controller, Intel (0x8086:0x7010) ]
                |  +-x Driver not found
                o
                |- Initializing Network controller
                |--[ Network controller, VirtIO, Ethernet (0x1af4:0x1000) ]
                |
     [ Virtio ] Attaching to  PCI addr 0x18
                [x] Vendor ID is VIRTIO
                [x] Device ID 0x1000 is in a valid range (Virtio LEGACY)
                [x] Device Revision ID (0x0) supported
                |  |- BAR0 I/O @ 0xc000, size 32
                |  |- BAR1 Mem @ 0xfebf1000, size 4096
                |  |- BAR4 Mem @ 0xfe000000, size 16384
                [x] Unit has valid I/O base (0xc000)
                [*] Reset device
                [x] Device has 4 MSI-X vectors
                MSI-X vector 0 pointing to cpu 0 intr 66
                MSI-X vector 1 pointing to cpu 0 intr 67
                MSI-X vector 2 pointing to cpu 0 intr 68
                MSI-X vector 3 pointing to cpu 0 intr 69
     [ Virtio ] Initialization complete
  [ VirtioNet ] Driver initializing
                [x] Negotiated needed features
                [x] Negotiated wanted features
                [x] Device handles packets w. partial checksum
                [x] Guest handles packets w. partial checksum
                [x] There's a control queue
                [x] Queue can handle any header/data layout
                [x] We can use indirect descriptors
                [x] There's a Ring Event Index to use
                [ ] There are multiple queue pairs
                [x] Merge RX buffers
                [x] RX queue (256) assigned (0x3b5000) to device
                [x] TX queue (256) assigned (0x3b9000) to device
                [x] CTRL queue (64) assigned (0x3bd000) to device
  [ VirtioNet ] Adding 128 receive buffers of size 2048
                [x] Valid Mac address: 52:54:00:12:34:56
                [x] Signalled driver OK
                [x] Link up
                o
    [ Machine ] Listing registered devices
                |
                +--+ NIC eth0
                |
                o
  [ SystemLog ] Initializing System Log
                Created @ 0x1fffffff0000 (63 kB)
                Data @ 0x1fffffff0028 (65496 bytes)
     [ Kernel ] Initializing RNG
     [ Kernel ] Initializing plugins
     [ Kernel ] Running service constructors
--------------------------------------------------------------------------------
================================================================================
 IncludeOS VERY_DIRTY (x86_64 / 64-bit)
 +--> Running [ Hello world - OS included ]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Args =
Try giving the service less memory, eg. 5MB in vm.json