nixbox 18.09 not on Vagrant Cloud

[MaCXyLo]

see here: https://app.vagrantup.com/nixos

[BrianHicks]

I ran into this as well, after using the instructions in the README, specifically: vagrant init nixos/nixos-18.09-x86_64 then vagrant up.

The solution, for now, seems to be to downgrade to one that is available: nixos/nixos-18.03-x86_64

[zimbatm]

The main reason is that Hashicorp has stopped offering CI builders and other CIs don't support VirtualBox. On top of that I don't have VirtualBox installed on my system anymore.

[zimbatm]

Does anyone know of a CI that offers support for VirtualBox?

[MaCXyLo]

nixops supports deployment of virtualbox images. hydra is the build server of nixos. but i have no experience with both.

next question is, if it's a good idea, to create a chicken or the egg problem?...

gitlab also seems to offer support for virtualbox images using packer: https://gitlab.com/gitlab-org/gitlab-packer/commit/15a926b58205dfa2fc7cb8bb8e166d742f4015ba

[zimbatm]

Following on that link, looks like GitLab stopped using packer to build images: https://gitlab.com/gitlab-org/gitlab-ce/issues/164#note_75698

Maybe we should just generate the images with Nix directly instead of using Packer, it's not that hard as well. The only difficult thing is to wrap the vagrant config properly on the output. With Nix it's possible to generate VirtualBox images without having VirtualBox installed on the host.

[wpoosanguansit]

Is there a solution for this issue?

[zimbatm]

something like https://github.com/nix-community/nixos-generators could be a good replacement. What's left is to wrap the output in a format that Vagrant can understand.

[gurjeet]

@zimbatm I have VirtualBox installed on my system, and I wouldn't mind supporting this effort for future versions of NixOS as well.

Let me know what's involved in publishing these images to the https://app.vagrantup.com/nixos repo.

[zimbatm]

Thanks for the offer @gurjeet. The best way forward would be to generate the images using Nix instead. Once that is done it will be possible to push them using the CI directly.

The image would include the vagrant.nix nixos module and output a OVA file, which is then wrapped in a .tar.gz with the metadata packed with it. The package format is described over here: https://www.vagrantup.com/docs/boxes/format.html

[zimbatm]

I just made a PR, hopefully it will be a bit clearer: https://github.com/nix-community/nixos-generators/pull/22

If that works, then we can deprecate this project and then create a new one to periodically publish the Vagrant boxes

[gurjeet]

@zimbatm I'm new to NixOS in general, and haven't used nix-generators before. How should I test your patch to that repo.

I am on macOS 10.13.6, and have nix(-darwin) installed. I have cloned the nixis-generators repo, and in that directory executing./nixos-generate --list shows vagrant as one of the formats.

The next step is to provide a configuration.nix to specify the NixOS derivation I need. I don't see a suitable one on my macOS. Can you please suggest which one I should use.

[zimbatm]

Yes that's a good start.

The principle for testing would be to run ./nixos-generate -f vagrant, get the .box image and try to start it with vagrant somehow (I think vagrant import is needed to import the box). This will fail because the format is probably wrong.

Once you have that test phase working, download one of the original .box files, unpack it with tar and look at the differences. The metadata file needs to be completed, the .ova file name needs to be fixed I guess. Then open formats/vagrant.nix and fix the build instructions to match that.

If that's too much low-level for you, ping me on IRC and I can walk you through it a bit more.

[PierreR]

@zimbatm serait-il possible d'enlever le postprocessor "vagrant-cloud" des fichiers json puisqu'apparemment cela ne sert plus à rien ? Si ok je peux pousser un tel changement.

[zimbatm]

sounds good @PierreR

[zimbatm]

fixed in dfc1beca4bef917756f2c9f2a7a7a4abdfbcfa47

[PierreR]

@zimbatm you also want to remove the vagrant box post-processors ? It was useful for me to create a vagrant box compatible with virtualbox. FWIW I don't use this post-processors with vmware because AFAIK there is no good free vagrant plugin for vmware.

[zimbatm]

The project is about creating vagrant boxes and adds all the cruft necessary for that to work like the vagrant users, a bunch of system packages... If all you want is to generate a VM image then it's probably better to use the nixos-generators project.

[PierreR]

The project is about creating vagrant boxes

I was just trying to point it out that dfc1bec is removing the creation of such a vagrant box

[zimbatm]

woops, reverted. I don't see the vagrant-cloud post-processor

[gurjeet]

... getting back to the ideal world I am trying to create for myself.

The command you suggested fails with the following error. I had a look at the patch before running the command, and the contents of formats/vagrant.nix seem to imply that the nixos-generate command needs to be executed on a Linux system.

$ ./nixos-generate -f vagrant
error: assertion failed at /nix/store/v6aja9ar864688529ayznix8mvv3ih10-nixpkgs-19.09pre184256.5b7e3718ee3/nixpkgs/pkgs/os-specific/linux/kernel/generic.nix:56:1
(use '--show-trace' to show detailed location information)

EDIT: To clarify a bit more, the line 56 in that file is asserting this:

assert stdenv.isLinux;

[zimbatm]

Oh right, the project doesn't cross-compile right now, it has to be executed on a linux machine

[zimbatm]

Does anyone want to try out https://github.com/nix-community/nixos-generators/pull/50 ? You will have to also checkout nixpkgs to the associated PR and then build the .box file

[LucianU]

@zimbatm, I'd like to try that PR, but it's not clear to me how to do it. Is it enough to clone your nixos-generators fork?

[zimbatm]

Clone https://github.com/nix-community/nixos-generators/pull/50 and https://github.com/NixOS/nixpkgs/pull/76071

Then in the nixos-generators, override the configuration.nix to

{ config, lib, pkgs, modulesPath, ... }:
{
  documentation.man.enable = false;
  documentation.nixos.enable = false;
  sound.enable = false;
}

Finally run ./nixos-generators -I nixpkgs=../path/to/nixpkgs-clone -f vagrant-virtualbox`.

At the end if should give you a vbox file that you can import into virtualbox and try out.

[LucianU]

I'm getting this error

./nixos-generate -I nixpkgs=../nixpkgs/ -f vagrant-virtualbox
error: attribute 'vagrantVirtualbox' in selection path 'config.system.build.vagrantVirtualbox' not found

I'm running on a Mac, btw.

[zimbatm]

thanks, fixed in https://github.com/NixOS/nixpkgs/pull/76071/commits/686125dbf9e1b5e933d3e74a7843bd9b1939d272

You will need a Linux remote builder to produce a compatible image. Do you know how to set that up?

[LucianU]

Do you mean I can only run the command on Linux? I could do it in a VM, if that's possible.

[zimbatm]

You can run the command on macOS if you have a Linux remote builder machine setup. You will have to add ./nixos-generate -I nixpkgs=../nixpkgs/ -f vagrant-virtualbox --system linux-x86_64. Or run it in a VM that works too.

[LucianU]

I tried building inside a NixOS VM managed with VirtualBox and I got this error:

WARNING: Image format was not specified for 'nixos.raw' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.
qemu-system-x86_64: cannot set up guest memory 'pc.ram': Cannot allocate memory
builder for '/nix/store/jpnka8baajbkkazqpqd40v7sxzy0g65s-nixos-ova-20.03.git.686125d-x86_64-linux.drv' failed with exit code 1
cannot build derivation '/nix/store/q76pbx6gfmhv976kw1gppy9nrxyar0xk-virtualbox-vagrant.box.drv': 1 dependencies couldn't be built
error: build of '/nix/store/q76pbx6gfmhv976kw1gppy9nrxyar0xk-virtualbox-vagrant.box.drv' failed

I think that's because the VM is managed by VirtualBox.

Next, I tried on a DigitalOcean instance running Ubuntu. I got this error:

error: a 'x86_64-linux' with features {kvm} is required to build '/nix/store/jpnka8baajbkkazqpqd40v7sxzy0g65s-nixos-ova-20.03.git.686125d-x86_64-linux.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, nixos-test}

[zimbatm]

I'm afraid there are going to be many rounds to trial.

For the first issue, I think giving more RAM to the VirtualBox VM should solve it.

The second issue is because the Nix relies on KVM to build the image. Most hosting providers don't allow nested virtualization. I know Google Cloud or Azure do propose that option.

[zimbatm]

I published a version of the image over here: https://app.vagrantup.com/nixos/boxes/nixos-20.03-test

Virtualbox seems to be broken on NixOS at the moment so I am not able to test it unfortunately.

[Br1ght0ne]

I get this error when running vagrant up on Windows 10 and latest VirtualBox. Everything works with 18.03 image.

Vagrantfile: https://gist.github.com/filalex77/dc3305055a3990494778177f7059effe#file-vagrantfile

``` > vagrant up Bringing machine 'default' up with 'virtualbox' provider... ==> default: Importing base box 'nixos/nixos-20.03-test'... ==> default: Matching MAC address for NAT networking... ==> default: Checking if box 'nixos/nixos-20.03-test' version '0.0.1' is up to date... ==> default: Setting the name of the VM: holochain-vagrant_default_1586268793281_37112 ==> default: Clearing any previously set network interfaces... ==> default: Preparing network interfaces based on configuration... default: Adapter 1: nat ==> default: Forwarding ports... default: 22 (guest) => 2222 (host) (adapter 1) ==> default: Running 'pre-boot' VM customizations... ==> default: Booting VM... ==> default: Waiting for machine to boot. This may take a few minutes... default: SSH address: 127.0.0.1:2222 default: SSH username: vagrant default: SSH auth method: private key default: default: Vagrant insecure key detected. Vagrant will automatically replace default: this with a newly generated keypair for better security. default: default: Inserting generated public key within guest... default: Removing insecure key from the guest if it's present... default: Key inserted! Disconnecting and reconnecting using new SSH key... ==> default: Machine booted and ready! ==> default: Checking for guest additions in VM... default: The guest additions on this VM do not match the installed version of default: VirtualBox! In most cases this is fine, but in rare cases it can default: shared folder errors, please make sure the guest additions within the default: virtual machine match the version of VirtualBox you have installed on default: your host and reload your VM. default: default: Guest Additions Version: 6.0.14 default: VirtualBox Version: 6.1 ==> default: Mounting shared folders... default: /vagrant => C:/Users/brightone/holochain-vagrant The following SSH command responded with a non-zero exit status. Vagrant assumes that this means the command failed! mkdir -p /vagrant Stdout from the command: Stderr from the command: ```

[LucianU]

From what I can tell, the command mkdir -p /vagrant failed. I would log into the VM and see if that directory exists. I don't know what it means, if it does though.

[Br1ght0ne]

@LucianU The directory doesn't actually exist, yes. In vagrant ssh, sudo asks for root password, which I do not have...

[LucianU]

That's strange. It shouldn't ask for your password, because it should normally log in with an SSH key. What does vagrant ssh-config show?

[Br1ght0ne]

@LucianU

What does vagrant ssh-config show?

> vagrant ssh-config
Host default
  HostName 127.0.0.1
  User vagrant
  Port 2222
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile C:/Users/brightone/holochain-vagrant/.vagrant/machines/default/virtualbox/private_key
  IdentitiesOnly yes
  LogLevel FATAL

[LucianU]

Strange. I tried your Vagrantfile now and I got the same error about mkdir -p /vagrant. From what I can tell, it's running the command without sudo and failing with permission denied.

Also, I realized I misunderstood what you said about sudo.

The user should have passwordless sudo, but the machine wasn't built that way. I think your only option at this point is to build your own machine with the tools in this repo. I did it in the past and it was straightforward.

[FranklinYu]

The nixos/nixos-20.03-test on Vagrant Cloud didn’t work for me. I got this error message during vagrant up phase, when running on macOS:

There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.

Command: ["startvm", "d5e91f69-bba7-4b13-a7b6-d7796e21e052", "--type", "headless"]

Stderr: VBoxManage: error: Implementation of the USB 2.0 controller not found!
VBoxManage: error: Because the USB 2.0 controller state is part of the saved VM state, the VM cannot be started. To fix this problem, either install the 'Oracle VM VirtualBox Extension Pack' or disable USB 2.0 support in the VM settings.
VBoxManage: error: Note! This error could also mean that an incompatible version of the 'Oracle VM VirtualBox Extension Pack' is installed (VERR_NOT_FOUND)
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component ConsoleWrap, interface IConsole

and the machine is powered off. Fortunately current master branch works (I built it with Packer myself).

By the way I guess #28 can be closed in favor of this?

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/building-openstack-images-and-development-vms-from-a-macos-host/9522/1

[joseph-long]

Crossposting from the PR on nixpkgs: https://github.com/NixOS/nixpkgs/pull/76071#issuecomment-711420610

The VBoxManage step failure can be worked around by letting vagrant create the VM, error out with that message, and then using the UUID in VBoxManage modifyvm [the uuid] --usb off. Booting should get further.

The problem with mkdir -p /vagrant failing has to do with the /etc/sudoers generated by the expression for the VM:

Defaults:root,%wheel env_keep+=LOCALE_ARCHIVE
Defaults:root,%wheel env_keep+=NIX_PATH
Defaults:root,%wheel env_keep+=TERMINFO_DIRS
Defaults env_keep+=SSH_AUTH_SOCK
Defaults lecture = never
root   ALL=(ALL) SETENV: ALL
%wheel ALL=(ALL) NOPASSWD: ALL, SETENV: ALL

# Don't edit this file. Set the NixOS options ‘security.sudo.configFile’
# or ‘security.sudo.extraRules’ instead.

# Keep SSH_AUTH_SOCK so that pam_ssh_agent_auth.so can do its magic.
Defaults env_keep+=SSH_AUTH_SOCK

# "root" is allowed to do anything.
root        ALL=(ALL:ALL) SETENV: ALL

# extraRules
%wheel  ALL=(ALL:ALL)   SETENV: ALL

# Keep terminfo database for root and %wheel.
Defaults:root,%wheel env_keep+=TERMINFO_DIRS
Defaults:root,%wheel env_keep+=TERMINFO

The sudoers format is powerful and I may not be interpreting it correctly, but I think the line %wheel ALL=(ALL) NOPASSWD: ALL, SETENV: ALL is being superseded by %wheel ALL=(ALL:ALL) SETENV: ALL later in the file.

Fortunately there is now an option, security.sudo.wheelNeedsPassword = false;, that covers that.