search

solo-io / bumblebee

Get eBPF programs running from the cloud to the kernel in 1 line of bash
Apache License 2.0
1.26k stars 78 forks source link

Vagrant Example Fails on Mac M1 (arm64) #40

Closed nickschuetz closed 2 years ago

nickschuetz commented 2 years ago

Here are my results based on the example provided in the README:

% vagrant up
Bringing machine 'impish' up with 'vmware_desktop' provider...
==> impish: Checking if box 'generic/ubuntu2110' version '3.6.4' is up to date...
==> impish: Verifying vmnet devices are healthy...
==> impish: Preparing network adapters...
==> impish: Starting the VMware VM...
An error occurred while executing `vmrun`, a utility for controlling
VMware machines. The command and output are below:

Command: ["start", "/Users/nschuetz/Projects/bumblebee/.vagrant/machines/impish/vmware_desktop/577e0bde-07ab-41e0-b964-2a3a9092f21a/generic-ubuntu2110-vmware.vmx", "nogui", {:notify=>[:stdout, :stderr], :timeout=>45}]

Stdout: 2022-01-06T11:01:00.524| ServiceImpl_Opener: PID 28446
Error: The operation was canceled

Stderr:

Then I tried using this setup with an actual arm64 box: https://gist.github.com/sbailliez/f22db6434ac84eccb6d3c8833c85ad92

Here are my results:

vmrun version 1.17.0 build-18656771

Vagrant File:

Vagrant.configure("2") do |config|
    config.vm.box = "spox/ubuntu-arm"
    config.vm.box_version = "1.0.0"
    config.vm.define :impish
    config.vm.hostname = "impish"
    config.vm.synced_folder ".", "/source"
    config.vm.provision "shell", inline: <<-SHELL
      # install llvm:
      export DEBIAN_FRONTEND=noninteractive
      export LLVM_VERSION=13
      wget https://apt.llvm.org/llvm.sh
      bash ./llvm.sh "$LLVM_VERSION"
      rm ./llvm.sh
      apt-get -qq update
      # bpf related deps:
      apt-get -qq install linux-headers-$(uname -r) linux-tools-$(uname -r) libbpf-dev
      # dev tools:
      apt-get -qq install -y golang docker.io make
      usermod -aG docker vagrant
      # add headers:
      bpftool btf dump file /sys/kernel/btf/vmlinux format c > /usr/local/include/vmlinux.h
      cp /source/builder/solo_types.h /usr/local/include/
    SHELL
end

% vagrant up
Bringing machine 'impish' up with 'vmware_desktop' provider...
==> impish: Cloning VMware VM: 'spox/ubuntu-arm'. This can take some time...
==> impish: Checking if box 'spox/ubuntu-arm' version '1.0.0' is up to date...
==> impish: Verifying vmnet devices are healthy...
==> impish: Preparing network adapters...
WARNING: The VMX file for this box contains a setting that is automatically overwritten by Vagrant
WARNING: when started. Vagrant will stop overwriting this setting in an upcoming release which may
WARNING: prevent proper networking setup. Below is the detected VMX setting:
WARNING:
WARNING:   ethernet0.pcislotnumber = "160"
WARNING:
WARNING: If networking fails to properly configure, it may require this VMX setting. It can be manually
WARNING: applied via the Vagrantfile:
WARNING:
WARNING:   Vagrant.configure(2) do |config|
WARNING:     config.vm.provider :vmware_desktop do |vmware|
WARNING:       vmware.vmx["ethernet0.pcislotnumber"] = "160"
WARNING:     end
WARNING:   end
WARNING:
WARNING: For more information: https://www.vagrantup.com/docs/vmware/boxes.html#vmx-allowlisting
==> impish: Starting the VMware VM...
==> impish: Waiting for the VM to receive an address...
==> impish: Forwarding ports...
    impish: -- 22 => 2222
==> impish: Waiting for machine to boot. This may take a few minutes...
    impish: SSH address: 127.0.0.1:2222
    impish: SSH username: vagrant
    impish: SSH auth method: private key
    impish:
    impish: Vagrant insecure key detected. Vagrant will automatically replace
    impish: this with a newly generated keypair for better security.
    impish:
    impish: Inserting generated public key within guest...
    impish: Removing insecure key from the guest if it's present...
    impish: Key inserted! Disconnecting and reconnecting using new SSH key...
==> impish: Machine booted and ready!
==> impish: Setting hostname...
==> impish: Configuring network adapters within the VM...
==> impish: Waiting for HGFS to become available...
==> impish: Enabling and configuring shared folders...
    impish: -- /Users/nschuetz/Projects/bumblebee: /source
==> impish: Running provisioner: shell...
    impish: Running: inline script
    impish: --2022-01-06 16:54:35--  https://apt.llvm.org/llvm.sh
    impish: Resolving apt.llvm.org (apt.llvm.org)... 199.232.6.49, 2a04:4e42:3e::561
    impish: Connecting to apt.llvm.org (apt.llvm.org)|199.232.6.49|:443... connected.
    impish: HTTP request sent, awaiting response... 200 OK
    impish: Length: 3751 (3.7K) [application/octet-stream]
    impish: Saving to: ‘llvm.sh’
    impish:
    impish:      0K ...                                                   100% 4.48M=0.001s
    impish:
    impish: 2022-01-06 16:54:36 (4.48 MB/s) - ‘llvm.sh’ saved [3751/3751]
    impish:
    impish: + needed_binaries=(lsb_release wget add-apt-repository)
    impish: + missing_binaries=()
    impish: + for binary in "${needed_binaries[@]}"
    impish: + which lsb_release
    impish: + for binary in "${needed_binaries[@]}"
    impish: + which wget
    impish: + for binary in "${needed_binaries[@]}"
    impish: + which add-apt-repository
    impish: + [[ 0 -gt 0 ]]
    impish: + LLVM_VERSION=13
    impish: + '[' 1 -eq 1 ']'
    impish: + LLVM_VERSION=13
    impish: ++ lsb_release -is
    impish: + DISTRO=Ubuntu
    impish: ++ lsb_release -sr
    impish: + VERSION=20.04
    impish: + DIST_VERSION=Ubuntu_20.04
    impish: + [[ 0 -ne 0 ]]
    impish: + declare -A LLVM_VERSION_PATTERNS
    impish: + LLVM_VERSION_PATTERNS[9]=-9
    impish: + LLVM_VERSION_PATTERNS[10]=-10
    impish: + LLVM_VERSION_PATTERNS[11]=-11
    impish: + LLVM_VERSION_PATTERNS[12]=-12
    impish: + LLVM_VERSION_PATTERNS[13]=-13
    impish: + LLVM_VERSION_PATTERNS[14]=
    impish: + '[' '!' _ ']'
    impish: + LLVM_VERSION_STRING=-13
    impish: + case "$DIST_VERSION" in
    impish: + REPO_NAME='deb http://apt.llvm.org/focal/     llvm-toolchain-focal-13   main'
    impish: + apt-key add -
    impish: + wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key
    impish: --2022-01-06 16:54:36--  https://apt.llvm.org/llvm-snapshot.gpg.key
    impish: Resolving apt.llvm.org (apt.llvm.org)... 199.232.6.49, 2a04:4e42:3e::561
    impish: Connecting to apt.llvm.org (apt.llvm.org)|199.232.6.49|:443... connected.
    impish: HTTP request sent, awaiting response... Warning: apt-key output should not be parsed (stdout is not a terminal)
    impish: 200 OK
    impish: Length: 3145 (3.1K) [application/octet-stream]
    impish: Saving to: ‘STDOUT’
    impish:
    impish:      0K ...                                                   100%  221M=0s
    impish:
    impish: 2022-01-06 16:54:36 (221 MB/s) - written to stdout [3145/3145]
    impish:
    impish: OK
    impish: + add-apt-repository 'deb http://apt.llvm.org/focal/     llvm-toolchain-focal-13   main'
    impish: Hit:1 http://us.ports.ubuntu.com/ubuntu-ports focal InRelease
    impish: Hit:2 http://us.ports.ubuntu.com/ubuntu-ports focal-updates InRelease
    impish: Hit:3 http://us.ports.ubuntu.com/ubuntu-ports focal-backports InRelease
    impish: Hit:4 http://us.ports.ubuntu.com/ubuntu-ports focal-security InRelease
    impish: Get:5 https://apt.llvm.org/focal llvm-toolchain-focal-13 InRelease [6,821 B]
    impish: Get:6 https://apt.llvm.org/focal llvm-toolchain-focal-13/main arm64 Packages [9,891 B]
    impish: Fetched 16.7 kB in 2s (10.7 kB/s)
    impish: Reading package lists...
    impish: + apt-get update
    impish: Hit:2 http://us.ports.ubuntu.com/ubuntu-ports focal InRelease
    impish: Hit:3 http://us.ports.ubuntu.com/ubuntu-ports focal-updates InRelease
    impish: Hit:1 https://apt.llvm.org/focal llvm-toolchain-focal-13 InRelease
    impish: Hit:4 http://us.ports.ubuntu.com/ubuntu-ports focal-backports InRelease
    impish: Hit:5 http://us.ports.ubuntu.com/ubuntu-ports focal-security InRelease
    impish: Reading package lists...
    impish: + apt-get install -y clang-13 lldb-13 lld-13 clangd-13
    impish: E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 2548 (unattended-upgr)
    impish: E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?
    impish: E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 2548 (unattended-upgr)
    impish: E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?
    impish: E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 2548 (unattended-upgr)
    impish: E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?
    impish: usermod: group 'docker' does not exist
    impish: /tmp/vagrant-shell: line 14: bpftool: command not found```
nickschuetz commented 2 years ago

I created a new ubuntu 21.10 arm64 box and updated the Vagrantfile a bit:

Vagrant.configure("2") do |config|
    config.vm.box = "nickschuetz/ubuntu-21.10-arm64"
    config.vm.box_version = "1.0.0"
    config.vm.define :impish
    config.vm.hostname = "impish"
    config.vm.synced_folder ".", "/source"
    config.vm.provision "shell", inline: <<-SHELL
      # install llvm:
      export DEBIAN_FRONTEND=noninteractive
      export LLVM_VERSION=13
      curl -sL https://apt.llvm.org/llvm.sh "$LLVM_VERSION" | bash
      apt-get -qq update
      # bpf related deps:
      apt-get -qq install linux-headers-$(uname -r) linux-tools-$(uname -r) libbpf-dev
      # dev tools:
      apt-get -qq install -y golang docker.io make
      # add headers:
      bpftool btf dump file /sys/kernel/btf/vmlinux format c > /usr/local/include/vmlinux.h
      cp /source/builder/solo_types.h /usr/local/include/
    SHELL
end

All that needs to be done now is clone bumblebee on your new vm and modify the Makefile (%s/amd64/arm64/g), make build-cli and away you go.

yuval-k commented 2 years ago

i believe this as addressed by #43

themarcelor commented 1 year ago

Hi @nickschuetz , any plans to release a new box? https://fridge.ubuntu.com/2022/07/19/ubuntu-21-10-impish-indri-end-of-life-reached-on-july-14-2022/

themarcelor commented 1 year ago

If anyone else wants to try the tutorial from https://www.solo.io/blog/get-started-with-ebpf-using-bumblebee/.

  1. Edit your Vagrantfile to utilize a different vagrant box (e.g., rethinc-oss/ubuntu-2204-arm64).
  2. vagrant up + vagrant ssh.
  3. git clone https://github.com/solo-io/bumblebee.git + cd bumblebee.
  4. checkout the latest release: git checkout v0.0.14 (otherwise your build will produce a binary whose version doesn't match a builder Docker img later on, e.g., builder:0.0.14-4-gb693411-dirty)
  5. sed -i 's/amd64/arm64/g' Makefile.
  6. make build-cli.
  7. Double-check the version: 
./_output/bee-linux-arm64 version
0.0.14
  1. sudo cp _output/bee-linux-arm64 /usr/bin/bee
  2. sudo apt install emacs
  3. Use the bee CLI to build the skeleton program:
 INFO  Selected Language: C
 INFO  Selected Program Type: Network
 INFO  Selected Map Type: RingBuffer
 INFO  Selected Output Type: print
 INFO  Selected Output Type: BPF Program File Location hello.c
  1. Edit the file with emacs -nw hello.c
  2. Follow the rest of the tutorial. 🎉 
➜ bumblebee (main) ✗ sudo bee build hello.c hello:v1
 SUCCESS  Successfully compiled "hello.c" and wrote it to "hello.o"
 SUCCESS  Saved BPF OCI image to hello:v1

➜ bumblebee (main) ✗ sudo bee list
Name     | OS    | OS Version             | Arch
hello:v1 | Linux | 5.13.19-051319-generic | aarch64

➜ bumblebee (main) ✗ sudo setcap cap_sys_resource,cap_sys_admin,cap_bpf+eip $(which bee)
➜ bumblebee (main) ✗ echo $?
0

➜ bumblebee (main) ✗ sudo bee run hello:v1
 SUCCESS  Fetching program from registry: hello:v1

(just vagrant ssh from some other terminal and run curl solo.io to see the magic 🪄 ).

        __
       /\ \                                                         Program location: hello:v1
       \ \ \____     __     __
        \ \ '__'\  /'__'\ /'__'\                      // \
         \ \ \L\ \/\  __//\  __/                      \\_/ //
          \ \_,__/\ \____\ \____\   ''-.._.-''-.._.. -(||)(')       <ctrl-n>   Select next table
           \/___/  \/____/\/____/                     '''           <ctrl-p>   Select previous table
                                                                    <ctrl-c>   Quit
                                         (powered by solo.io)

╔════════════════════════════════════════════════════════════════events════════════════════════════════════════════════════════════════╗
║message                                                               pid                                                             ║
║Hello World!                                                          11385                                                           ║
║Hello World!                                                          11416
nickschuetz commented 1 year ago

Hi @nickschuetz , any plans to release a new box? https://fridge.ubuntu.com/2022/07/19/ubuntu-21-10-impish-indri-end-of-life-reached-on-july-14-2022/

No, sorry, I do not.