search

GaloisInc / HaLVM

The Haskell Lightweight Virtual Machine (HaLVM): GHC running on Xen
BSD 3-Clause "New" or "Revised" License
1.05k stars 88 forks source link

HaLVMs in Amazon's Cloud #17

Closed acw closed 8 years ago

AFDudley commented 9 years ago

What is the status of this? What things need to be added to HaLVM to make this work?

acw commented 9 years ago

It should work, but has never been tested.

AFDudley commented 9 years ago

It doesn't look like there is a HaLVM AMI available. it needs to work with AKI, like mirageOS does. Would this script get me 90% of the way there? https://github.com/mirage/mirage/blob/master/scripts/ec2.sh I rather not build HaLVM on bare metal, can I run the fedora build host via HVM on AWS?

acw commented 9 years ago

My plan was to use that script as a base, yes.

And yes, you should be able to build the build host on AWS, you just won't be able to run HaLVMs on it.

acw commented 9 years ago

OK, update: I was able to get an upload and boot (I think) a HaLVM on EC2 ... it doesn't work well. After some investigation, it looks like EC2 uses an old version of Xen that the HaLVM doesn't presently support. It's possible that the problem stems from that. So the next step is to figure out a way to build Xen 3.4.3 for a platform I can mess around with, and make sure that all those issues are accounted for.

AFDudley commented 9 years ago

Wow. Okay, good to know. I built a build host on EC2 using ubuntu 14.04 it took around 4 hours so I didn't get a chance to actually build any of the same projects yet, thanks for the update.

AFDudley commented 9 years ago

my Docker host is running in a t2.micro instance which has this version of Xen: [ 0.000000] DMI: Xen HVM domU, BIOS 4.2.amazon 06/02/2014 [ 0.000000] Hypervisor detected: Xen HVM [ 0.000000] Xen version 4.2. [ 0.000000] Xen Platform PCI: I/O protocol version 1

I am going to continue building HaLVM.

acw commented 9 years ago

Hmmm. Looks like t2.micro instances are HVM-based, whereas the HaLVM is paravirt-based. It's also not clear if they support PV-GRUB or a similar system or not.

AFDudley commented 9 years ago

Right, that was just the build host; I wasn't trying to run HaLVM on that machine, just build it on that machine. Amazon does support PV-GRUB, just not on that instance type. I am now realizing that Xen version has nothing to do with the Xen version they run on their PV instances... sorry! http://www.brendangregg.com/blog/2014-05-09/xen-feature-detection.html

AFDudley commented 9 years ago

I was able to use a modified version of the ec2.sh to boot the GetId sample program on a m3.medium instance. Do you have another unikernel I should be testing with? I will try some more advanced examples today.

acw commented 9 years ago

If you could get the WebServer to start, that would be amazing!

https://github.com/GaloisInc/HaLVM/wiki/HaLVM-Web-Server-Quick-Start

AFDudley commented 9 years ago

Building network-hans-0.2... Preprocessing library network-hans-0.2... [1 of 7] Compiling Hans.Socket.Handle ( src/Hans/Socket/Handle.hs, dist/build/Hans/Socket/Handle.o ) [2 of 7] Compiling Network.Socket.Internal ( src/Network/Socket/Internal.hs, dist/build/Network/Socket/Internal.o ) [3 of 7] Compiling Network.Socket.ByteString.Lazy ( src/Network/Socket/ByteString/Lazy.hs, dist/build/Network/Socket/ByteString/Lazy.o ) [4 of 7] Compiling Network.Socket.ByteString ( src/Network/Socket/ByteString.hs, dist/build/Network/Socket/ByteString.o ) [5 of 7] Compiling Network.BSD.ServiceDB ( src/Network/BSD/ServiceDB.hs, dist/build/Network/BSD/ServiceDB.o )

my system runs of memory here. it has 2gb, how much do you recommend?

johnpmayer commented 9 years ago

You could also add swap space On Feb 23, 2015 3:14 PM, "A. F. Dudley" notifications@github.com wrote:

Building network-hans-0.2... Preprocessing library network-hans-0.2... [1 of 7] Compiling Hans.Socket.Handle ( src/Hans/Socket/Handle.hs, dist/build/Hans/Socket/Handle.o ) [2 of 7] Compiling Network.Socket.Internal ( src/Network/Socket/Internal.hs, dist/build/Network/Socket/Internal.o ) [3 of 7] Compiling Network.Socket.ByteString.Lazy ( src/Network/Socket/ByteString/Lazy.hs, dist/build/Network/Socket/ByteString/Lazy.o ) [4 of 7] Compiling Network.Socket.ByteString ( src/Network/Socket/ByteString.hs, dist/build/Network/Socket/ByteString.o ) [5 of 7] Compiling Network.BSD.ServiceDB ( src/Network/BSD/ServiceDB.hs, dist/build/Network/BSD/ServiceDB.o )

my system runs of memory here. it has 2gb, how much do you recommend?

— Reply to this email directly or view it on GitHub https://github.com/GaloisInc/HaLVM/issues/17#issuecomment-75622260.

AFDudley commented 9 years ago

Adding Swap is a good idea, still don't know how much to add. To answer my own question it looks like 4gb of RAM would have done the trick.

acw commented 9 years ago

I'm not sure. You might try adding "--disable-split-objs" to see if that helps decrease your memory requirements (at the cost of a larger binary).

AFDudley commented 9 years ago

Error from WebServer: Xen Minimal OS!

start_info: 0x112e000(VA)

nr_pages: 0xf0000

shared_inf: 0x7db69000(MA)

 pt_base: 0x1131000(VA)

nr_pt_frames: 0xd

mfn_list: 0x9ae000(VA)

mod_start: 0x0(VA)

 mod_len: 0

   flags: 0x0

cmd_line: root=/dev/sda1 ro 4

stack: 0x96d840-0x98d840

MM: Init

  _text: 0x0(VA)

 _etext: 0x7dc7d(VA)

_erodata: 0x9a000(VA)

 _edata: 0x9fce0(VA)

stack start: 0x96d840(VA)

   _end: 0x9ade40(VA)

start_pfn: 1141

max_pfn: f0000

Mapping memory range 0x1400000 - 0xf0000000

setting 0x0-0x9a000 readonly

skipped 0x1000

MM: Initialise page allocator for 18ba000(18ba000)-f0000000(f0000000)

MM: done

Demand map pfns at f0001000-20f0001000.

Heap resides at 20f0002000-40f0002000.

Initialising timer interface

Initialising console ... done.

gnttab_table mapped at 0xf0001000.

Initialising scheduler

Thread "Idle": pointer: 0x20f0002050, stack: 0x2070000

Thread "xenstore": pointer: 0x20f0002800, stack: 0x2080000

xenbus initialised on irq 1 mfn 0x1ba6fb1

Thread "shutdown": pointer: 0x20f0002fb0, stack: 0x2090000

Dummy main: start_info=0x98d940

Thread "main": pointer: 0x20f0003760, stack: 0x20a0000

"main" "root=/dev/sda1" "ro" "4"

vbd 2049 is hd0

*** BLKFRONT for device/vbd/2049 **

backend at /local/domain/0/backend/vbd/1397/2049

10240 sectors of 512 bytes

vbd 2064 is hd1

*** BLKFRONT for device/vbd/2064 **

backend at /local/domain/0/backend/vbd/1397/2064

8377344 sectors of 512 bytes



GNU GRUB  version 0.97  (3932160K lower / 0K upper memory)

+-------------------------------------------------------------------------+||||||||||||||||||||||||+-------------------------------------------------------------------------+

Use the ^ and v keys to select which entry is highlighted.

Press enter to boot the selected OS, 'e' to edit the

commands before booting, or 'c' for a command-line.  Mirage                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       The highlighted entry will be booted automatically in 1 seconds.     Booting 'Mirage'

root (hd0)

Filesystem type is ext2fs, using whole disk

kernel /boot/halvm.gz

block error -1 for op 0

============= Init TPM Front ================

Tpmfront:Error Unable to read device/vtpm/0/backend-id during tpmfront initialization! error = ENOENT

Tpmfront:Info Shutting down tpmfront

xc: error: panic: xc_dom_core.c:621: xc_dom_find_loader: no loader found: Invalid kernel

xc_dom_parse_image returned -1

close(3)

Error 9: Unknown boot failure

Press any key to continue... I can provide access to the AMIs or anything else that might help with this issue.

AFDudley commented 9 years ago

GetId works just fine without debugging in xen turned on, is this also the case for WebServer or would that cause it to break?

acw commented 9 years ago

I don't think debugging would matter.

Yes, can you open up the AMI? And how did you get that console output?

AFDudley commented 9 years ago

Here is the ami-ID I have made it public, let me know if you need something more: ami-00e6b568

from the web ui: actions -> instance settings -> Get system log (at the very bottom)

If you are using something like boto, you can pull it that way as well. I avoid the CLI tools, so I don't know how to do it from there.

acw commented 9 years ago

Oh! Excellent on both fronts. I was despairing how I was going to debug these.

I'm seeing the same error as you in my instance, which is good, I suppose. :)

AFDudley commented 9 years ago

The issue was the loopback defined in https://github.com/AFDudley/ec2-halvm/blob/master/ec2-halvm.sh was too small.

It boots now and sees a NIC: root (hd0)

Filesystem type is ext2fs, using whole disk

kernel /boot/halvm.gz

============= Init TPM Front ================

Tpmfront:Error Unable to read device/vtpm/0/backend-id during tpmfront initialization! error = ENOENT

Tpmfront:Info Shutting down tpmfront

close blk: backend=/local/domain/0/backend/vbd/1122/2049 node=device/vbd/2049

close blk: backend=/local/domain/0/backend/vbd/1122/2064 node=device/vbd/2064

Found 1 NIC

22:00:0B:7F:CF:94

acw commented 9 years ago

OK. As you might think, this suggests that it found an Ethernet card, but then either failed initializing it or failed to complete a DHCP handshake.

If this is an older version of Xen, then this could just be a matter of back-porting support for older NIC backends.

AFDudley commented 9 years ago

[ 0.000000] DMI: Xen HVM domU, BIOS 4.2.amazon 06/02/2014 [ 0.000000] Hypervisor detected: Xen HVM [ 0.000000] Xen version 4.2. [ 0.000000] Xen Platform PCI: I/O protocol version 1 [ 0.000000] Netfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated NICs. [ 0.000000] Blkfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated disks. [ 0.000000] ACPI: RSDP 00000000000ea020 000024 (v02 Xen) [ 0.000000] ACPI: XSDT 00000000fc00f710 00005C (v01 Xen HVM 00000000 HVML 00000000) [ 0.000000] ACPI: FACP 00000000fc00f260 0000F4 (v04 Xen HVM 00000000 HVML 00000000) [ 0.000000] ACPI: DSDT 00000000fc0035e0 00BBF6 (v02 Xen HVM 00000000 INTL 20090123) [ 0.000000] ACPI: APIC 00000000fc00f360 0000D8 (v02 Xen HVM 00000000 HVML 00000000) [ 0.000000] ACPI: SRAT 00000000fc00f4b0 000170 (v01 Xen HVM 00000000 HVML 00000000) [ 0.000000] ACPI: HPET 00000000fc00f620 000038 (v01 Xen HVM 00000000 HVML 00000000) [ 0.000000] ACPI: WAET 00000000fc00f660 000028 (v01 Xen HVM 00000000 HVML 00000000) [ 0.000000] ACPI: SSDT 00000000fc00f690 000031 (v02 Xen HVM 00000000 INTL 20090123) [ 0.000000] ACPI: SSDT 00000000fc00f6d0 000031 (v02 Xen HVM 00000000 INTL 20090123) [ 0.000000] Booting paravirtualized kernel on Xen HVM [ 0.000000] xen: PV spinlocks enabled [ 0.000000] xen:events: Xen HVM callback vector for event delivery is enabled [ 0.264007] Xen: using vcpuop timer interface [ 0.264014] installing Xen timer for CPU 0 [ 0.560094] xen:balloon: Initialising balloon driver [ 0.564038] xen_balloon: Initialising balloon driver [ 0.568079] xen:balloon: Cannot add additional memory (-17) [ 0.610123] Switched to clocksource xen [ 0.638675] xen: --> pirq=16 -> irq=8 (gsi=8) [ 0.638717] xen: --> pirq=17 -> irq=12 (gsi=12) [ 0.638741] xen: --> pirq=18 -> irq=1 (gsi=1) [ 0.638769] xen: --> pirq=19 -> irq=6 (gsi=6) [ 0.638799] xen: --> pirq=20 -> irq=4 (gsi=4) [ 1.061693] xen: --> pirq=21 -> irq=28 (gsi=28) [ 1.061765] xen:grant_table: Grant tables using version 1 layout [ 1.209680] xen_netfront: Initialising Xen virtual ethernet driver [ 1.358828] xenbus_probe_frontend: Device with no driver: device/vfb/0

I am running in a m3.medium. Which uses Xen PVHVM

copumpkin commented 8 years ago

:+1: on documenting this and getting it on the readme/front page. Would greatly facilitate experimentation and tickle the right buzzwords (Unikernel, EC2, AWS, etc.)

dysinger commented 8 years ago

Is this still an issue?

acw commented 8 years ago

Nope, it's done ... until someone finds a bug.