hashicorp / packer

Packer is a tool for creating identical machine images for multiple platforms from a single source configuration.
http://www.packer.io
Other
15.09k stars 3.33k forks source link

Virtual box linux kernel modules missing #538

Closed andrewdeandrade closed 10 years ago

andrewdeandrade commented 11 years ago

I'm using packer to build a Vagrant box for use in VirtualBox. I have an error that could be a Packer or VirtualBox problem, but I am not sure since I haven't seen this error with previous VirtualBoxes used with Vagrant (but my prior experience is admittedly limited). Below are the details. FWIW, my suspicion is that one of the Packer scripts or a tool it calls is kicking off some sort of kernel recompile when packaging up the vmdk, causing the loss of vboxsf.

When building the VM, I know that installation of VBoxAdditions via my shell provisioning script is working perfectly (no errors, log below. All dependencies like build-essentials, dkms and linux-headers-generic--uname -r are available as is X and XFree86). After installing, I write the result of lsmod | grep vbox to a log file:

vagrant@vagrant-precise64:~$ cat 30_vbox_system_modules.log 
vboxvideo              12658  0 
drm                   287564  1 vboxvideo
vboxsf                 39556  0 
vboxguest             235481  2 vboxsf

I then install Chef, do a bunch of cleanup and right before returning exit 0 at the end of the script, I write at the result of lsmod | grep vbox to a second log and get the same exact value as before, confirming that the linux kernel extensions are there before Packer takes over and passes the virtual machine through the Vagrant post-processor.

Despite all this, when I add the Vagrant box and try to boot it with the --no-provision flag to keep things simple, I get the following error:

$ vagrant up proxy --no-provision
Bringing machine 'proxy' up with 'virtualbox' provider...
[proxy] Importing base box 'precise64'...
[proxy] Matching MAC address for NAT networking...
[proxy] Setting the name of the VM...
[proxy] Clearing any previously set forwarded ports...
[proxy] Creating shared folders metadata...
[proxy] Clearing any previously set network interfaces...
[proxy] Preparing network interfaces based on configuration...
[proxy] Forwarding ports...
[proxy] -- 22 => 2222 (adapter 1)
[proxy] Running 'pre-boot' VM customizations...
[proxy] Booting VM...
[proxy] Waiting for machine to boot. This may take a few minutes...
[proxy] Machine booted and ready!
[proxy] Configuring and enabling network interfaces...
[proxy] Mounting shared folders...
[proxy] -- /vagrant
Failed to mount folders in Linux guest. This is usually beacuse
the "vboxsf" file system is not available. Please verify that
the guest additions are properly installed in the guest and
can work properly. The command attempted was:

mount -t vboxsf -o uid=`id -u vagrant`,gid=`getent group vagrant | cut -d: -f3` /vagrant /vagrant
mount -t vboxsf -o uid=`id -u vagrant`,gid=`id -g vagrant` /vagrant /vagrant

In order to triage this, I've first tried the "keep_input_artifact": true flag on the vagrant post-processor and loaded the ovf/vmdk directly in VirtualBox and after logging and running lsmod | grep vbox I get back nothing. I then tried adding a second shell provisioning after the first that does nothing but log the output of lsmod | grep vbox to another log to see where this kernel extension modules are getting lost.

When in the vm modprobe vboxdrv returns FATAL: Module vboxdrv not found.

When I look in /etc/init.d/ for the vboxdrv binary, it is not there.

When I try to run the VirtualBox command at the $ prompt I get:

root@vagrant-precise64:~# VirtualBox
The program 'VirtualBox' is currently not installed.  You can install it by typing:
apt-get install virtualbox-qt

The kernel ring buffer returns nothing when I pipe dmesg through a grep with vboxdrv, vbox or even just vb.

FWIW, the only vbox related items in /etc/init.d/ are vboxadd, vboxadd-service and vboxadd-x11

vagrant@vagrant-precise64:/var/log$ cat VBoxGuestAdditions.log 

Starting the VirtualBox Guest Additions ...done.
Starting VirtualBox Guest Addition service  ...done.
vagrant@vagrant-precise64:/var/log$ cat vboxadd-install.log
Uninstalling modules from DKMS
Attempting to install using DKMS

Creating symlink /var/lib/dkms/vboxguest/4.2.18/source ->
                 /usr/src/vboxguest-4.2.18

DKMS: add completed.

Kernel preparation unnecessary for this kernel.  Skipping...

Building module:
cleaning build area....
make KERNELRELEASE=3.8.0-29-generic -C /lib/modules/3.8.0-29-generic/build M=/var/lib/dkms/vboxguest/4.2.18/build............
cleaning build area....

DKMS: build completed.

vboxguest:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.8.0-29-generic/updates/dkms/

vboxsf.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.8.0-29-generic/updates/dkms/

vboxvideo.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.8.0-29-generic/updates/dkms/

depmod....

DKMS: install completed.
Creating user for the Guest Additions.
Creating udev rule for the Guest Additions kernel module.
vagrant@vagrant-precise64:/var/log$ cat vboxadd-install-x11.log 
VirtualBox Guest Additions installation, Window System and desktop setup

FWIW, I installed VirtualBox Guest additions with the following snippet:

vbga="/mnt/VBoxGuestAdditions"
mkdir -p $vbga
mount -o loop /home/$account/VBoxGuestAdditions.iso $vbga
yes|sh $vbga/VBoxLinuxAdditions.run
umount $vbga
unset vbga
rm -f /home/$account/VBoxGuestAdditions.iso

I also found the source of vboxsf in /usr/src/vboxguest-4.2.18/vboxsf

andrewdeandrade commented 11 years ago

Furthermore, when I add asecond provisioner after my main shell script:

  "provisioners": [{
    "type": "shell",
    "script": "./configs/packer-bootstrap.sh",
    "execute_command" : "echo {{user `ssh_pass`}} | sudo -S sh '{{ .Path }}'"
  },{
    "type": "shell",
    "script": "./configs/temp.sh",
    "execute_command" : "echo {{user `ssh_pass`}} | sudo -S sh '{{ .Path }}'"
  }],

and add the following two lines to it:

sleep 10

lsmod | grep vbox > /home/vagrant/36_vbox_system_modules.log

It sits and appears stuck with almost 100% CPU across 4 processor cores on my Macbook Air 13" (kernel_task)

2013/10/15 20:28:09 /Users/me/go/bin/packer-builder-virtualbox: 2013/10/15 20:28:09 remote command exited with '0': echo vagrant | sudo -S sh '/tmp/script.sh'
2013/10/15 20:28:09 127374 bytes written for 'stdout'
2013/10/15 20:28:09 0 bytes written for 'stderr'
2013/10/15 20:28:09 127374 bytes written for 'stdout'
2013/10/15 20:28:09 0 bytes written for 'stderr'
2013/10/15 20:28:09 /Users/me/go/bin/packer-command-build: 2013/10/15 20:28:09 127374 bytes written for 'stdout'
2013/10/15 20:28:09 /Users/me/go/bin/packer-command-build: 2013/10/15 20:28:09 0 bytes written for 'stderr'
2013/10/15 20:28:09 /Users/me/go/bin/packer-provisioner-shell: 2013/10/15 20:28:09 0 bytes written for 'stderr'
2013/10/15 20:28:09 /Users/me/go/bin/packer-provisioner-shell: 2013/10/15 20:28:09 127374 bytes written for 'stdout'
==> vagrant-precise64: Provisioning with shell script: ./configs/temp.sh
2013/10/15 20:28:09 ui: ==> vagrant-precise64: Provisioning with shell script: ./configs/temp.sh
2013/10/15 20:28:09 /Users/me/go/bin/packer-provisioner-shell: 2013/10/15 20:28:09 Opening ./configs/temp.sh for reading
2013/10/15 20:28:09 /Users/me/go/bin/packer-builder-virtualbox: 2013/10/15 20:28:09 opening new ssh session
2013/10/15 20:28:10 /Users/me/go/bin/packer-builder-virtualbox: 2013/10/15 20:28:10 Starting remote scp process:  scp -vt /tmp
2013/10/15 20:28:11 /Users/me/go/bin/packer-builder-virtualbox: 2013/10/15 20:28:11 Started SCP session, beginning transfers...
2013/10/15 20:28:11 /Users/me/go/bin/packer-builder-virtualbox: 2013/10/15 20:28:11 Copying input data into in-memory buffer so we can get the length
2013/10/15 20:28:12 /Users/me/go/bin/packer-builder-virtualbox: 2013/10/15 20:28:12 Opening conn for SSH to tcp 127.0.0.1:3213

That last log entry is at 20:28:12 and it is now ~20:49:00.

I don't know if this is or is not related to the problem above. I was hoping to check for the presence of vboxsf in a second provisioning shell script.

andrewdeandrade commented 11 years ago

I think I found the solution. So in order to install VBoxGuestAdditions, you need to have build-essentials, linux-headers-generic-$(uname -r) and dkms installed. All three are needed for rebuilding the kernel. Many vagrant box build scripts I've seen often uninstall them afterwards. However in my case above the removal of dkms caused the linux kernel modules not to be loaded on reboot. This to me suggests that the vmdk packaging step is triggering the recompiling of dynamically built kernel modules.

mitchellh commented 11 years ago

Packer itself doesn't do anything within the VM (executes no scripts on its own) other than those that you specify. Furthermore, the packaging is step is doing a literal VBoxManage export of that VM. Packer is extremely not sneaky. :)

One issue that may be happening as well is if you don't have a proper shutdown_command specified then you could be losing data.

Other than that I'm not sure what could be going wrong here. The 100% CPU thing is interesting, but unless I can reproduce it (which I haven't been able to), then I can't move forward on that. :(

Any more info?

mitchellh commented 10 years ago

No reply. Closing.

dindinoh commented 10 years ago

sorry to confirm on an already closed issue, but we've had some big issues with this.

but I will confirm that using:

"shutdown_command": "echo 'packer' | sudo -S shutdown -P now"

or the like to actually use a proper shutdown solves the problem with not compiling guest additions.

benissimo commented 10 years ago

I got this same error trying to do "vagrant up" after upgrading from virtualbox 4.3.2 to 4.3.4. I have these plugins: vagrant-omnibus (1.1.2) vagrant-vbguest (0.9.0)

and my Vagrantfile also has a 'config.vm.provision "chef_solo"' block.

Even though no provisioning is needed (has already been provisioned and I'm using vagrant 1.3.5), chef_solo provisioner always tries to copy cookbooks, etc. into the shared folder, even when provisioning is not happening (which seems like a bug).

I performed the shutdown command listed above by @dindinoh, tried vagrant reload, and still kept getting the same error. Finally what worked for me was to comment out the chef_solo block in my Vagrantfile and do a reload.

Otherwise I kept getting this error. FYI I'm on a Mac host (10.6.8) with Ubuntu guest (12.0.4, headless).

[default] Running 'pre-boot' VM customizations...
[default] Booting VM...
GuestAdditions versions on your host (4.3.4) and guest (4.3.2) do not match.
stdin: is not a tty
Reading package lists...
Building dependency tree...
Reading state information...
dkms is already the newest version.
linux-headers-3.2.0-57-generic is already the newest version.
The following package was automatically installed and is no longer required:
  linux-image-3.2.0-55-generic
Use 'apt-get autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 8 not upgraded.
Copy iso file /Applications/VirtualBox.app/Contents/MacOS/VBoxGuestAdditions.iso into the box /tmp/VBoxGuestAdditions.iso
stdin: is not a tty
mount: block device /tmp/VBoxGuestAdditions.iso is write-protected, mounting read-only
Installing Virtualbox Guest Additions 4.3.4 - guest version is 4.3.2
stdin: is not a tty
Verifying archive integrity... All good.
Uncompressing VirtualBox 4.3.4 Guest Additions for Linux............
VirtualBox Guest Additions installer
Removing installed version 4.3.4 of VirtualBox Guest Additions...
Copying additional installer modules ...
Installing additional modules ...
Removing existing VirtualBox DKMS kernel modules ...done.
Removing existing VirtualBox non-DKMS kernel modules ...done.
Building the VirtualBox Guest Additions kernel modules
The headers for the current running kernel were not found. If the following
module compilation fails then this could be the reason.

Building the main Guest Additions module ...fail!
(Look at /var/log/vboxadd-install.log to find out what went wrong)
Doing non-kernel setup of the Guest Additions ...done.
Installing the Window System drivers
Installing X.Org Server 1.11 modules ...done.
Setting up the Window System to use the Guest Additions ...done.
You may need to restart the hal service and the Window System (or just restart
the guest system) to enable the Guest Additions.

Installing graphics libraries and desktop services components ...done.
An error occurred during installation of VirtualBox Guest Additions 4.3.4. Some functionality may not work as intended.
stdin: is not a tty
[default] Waiting for machine to boot. This may take a few minutes...
[default] Machine booted and ready!
[default] Configuring and enabling network interfaces...
[default] Mounting shared folders...
[default] -- /tmp/vagrant-chef-1/chef-solo-2/roles
Failed to mount folders in Linux guest. This is usually beacuse
the "vboxsf" file system is not available. Please verify that
the guest additions are properly installed in the guest and
can work properly. The command attempted was:

mount -t vboxsf -o uid=`id -u vagrant`,gid=`getent group vagrant | cut -d: -f3` /tmp/vagrant-chef-1/chef-solo-2/roles /tmp/vagrant-chef-1/chef-solo-2/roles
mount -t vboxsf -o uid=`id -u vagrant`,gid=`id -g vagrant` /tmp/vagrant-chef-1/chef-solo-2/roles /tmp/vagrant-chef-1/chef-solo-2/roles

p.s. The error message contains a typo "usually beacuse" (sic).

Update: in my case it looks like the real problem is that 4.3.4 guest additions do not build:

https://www.virtualbox.org/ticket/12358

DKMS make.log for vboxguest-4.3.4 for kernel 3.2.0-57-generic (x86_64)
Tue Dec  3 11:33:30 UTC 2013
make: Entering directory `/usr/src/linux-headers-3.2.0-57-generic'
  LD      /var/lib/dkms/vboxguest/4.3.4/build/built-in.o
  LD      /var/lib/dkms/vboxguest/4.3.4/build/vboxguest/built-in.o
  CC [M]  /var/lib/dkms/vboxguest/4.3.4/build/vboxguest/VBoxGuest-linux.o
In file included from include/linux/slub_def.h:14:0,
                 from include/linux/slab.h:185,
                 from /var/lib/dkms/vboxguest/4.3.4/build/vboxguest/r0drv/linux/the-linux-kernel.h:72,
                 from /var/lib/dkms/vboxguest/4.3.4/build/vboxguest/VBoxGuest-linux.c:26:
include/linux/kmemleak.h:1:1: warning: null character(s) ignored [enabled by default]
In file included from include/linux/slab.h:185:0,
                 from /var/lib/dkms/vboxguest/4.3.4/build/vboxguest/r0drv/linux/the-linux-kernel.h:72,
                 from /var/lib/dkms/vboxguest/4.3.4/build/vboxguest/VBoxGuest-linux.c:26:
include/linux/slub_def.h: In function ‘kmalloc_order’:
include/linux/slub_def.h:224:2: error: implicit declaration of function ‘kmemleak_alloc’ [-Werror=implicit-function-declaration]
In file included from include/linux/mm.h:16:0,
                 from /var/lib/dkms/vboxguest/4.3.4/build/vboxguest/r0drv/linux/the-linux-kernel.h:88,
                 from /var/lib/dkms/vboxguest/4.3.4/build/vboxguest/VBoxGuest-linux.c:26:
include/linux/range.h: At top level:
include/linux/range.h:1:1: warning: null character(s) ignored [enabled by default]
In file included from /var/lib/dkms/vboxguest/4.3.4/build/vboxguest/r0drv/linux/the-linux-kernel.h:88:0,
                 from /var/lib/dkms/vboxguest/4.3.4/build/vboxguest/VBoxGuest-linux.c:26:
include/linux/mm.h:1303:8: warning: ‘struct range’ declared inside parameter list [enabled by default]
include/linux/mm.h:1303:8: warning: its scope is only this definition or declaration, which is probably not what you want [enabled by default]
cc1: some warnings being treated as errors
make[2]: *** [/var/lib/dkms/vboxguest/4.3.4/build/vboxguest/VBoxGuest-linux.o] Error 1
make[1]: *** [/var/lib/dkms/vboxguest/4.3.4/build/vboxguest] Error 2
make: *** [_module_/var/lib/dkms/vboxguest/4.3.4/build] Error 2
make: Leaving directory `/usr/src/linux-headers-3.2.0-57-generic'
mhamrah commented 10 years ago

I ran into this same issue. The cause was my post-install shell provisioner for packer.

In order to install vbox additions, yo need the kernel-headers-uname r and kernel-devel-uname -r packages (along with the some others). In order for these packages to be usable you must reboot the instance, otherwise, the older kernel versions will be used.

Packer supports reboots via the shell provisioner, you just need to separate the scripts and add reboot/sleep commands. I added this to my packer provisioning configuration.