NixOS / nixops

NixOps is a tool for deploying to NixOS machines in a network or cloud.
https://nixos.org/nixops
GNU Lesser General Public License v3.0
1.87k stars 364 forks source link

Nixops not respecting ec2.ebsInitialRootDiskSize option #1212

Closed thomashoneyman closed 4 years ago

thomashoneyman commented 5 years ago

I attempted to use the deployment.ec2.ebsInitialRootDiskSize option to deploy a new t2.nano EC2 instance with 50GB of disk space instead of the default, which I believed was 8GB.

However, regardless of what I set this option to, the Elastic Beanstalk volume is always created with the size 3GB (confirmed with df -h and lsblk while ssh-ed into the EC2, and manually in the AWS console).

Steps to reproduce

The simple-twitter repository serves as a simple, deployable reproduction of this issue. The issue linked below has a full description of my attempts to use this option to increase the disk size of an EC2: https://github.com/Gabriel439/simple-twitter/issues/1#issue-531586406

For posterity's sake, here are excerpted steps from that issue.

# clone the repository and ensure nixops is available
git clone git@github.com:Gabriel439/simple-twitter
cd simple-twitter
nix-shell -p nixops # unstable channel produces nixops version 1.7

# create a deployment and deploy to ec2
nixops create --deployment simple-twitter simple-twitter.nix
nixops deploy --deployment simple-twitter --allow-reboot

In my case this fails with the error error: writing to file: No space left on device. That appears to be because the EC2 has only 3GB of disk space and it becomes full during the deployment.

[root@ip:~]# lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0   3G  0 disk 
└─xvda1 202:1    0   3G  0 part /

I then try to bump up the disk space using ebsInitialRootDiskSize:

 ec2 = {
   inherit accessKeyId region;

-  instanceType = "t2.nano";
+  instanceType = "t2.micro";
+
+  ebsInitialRootDiskSize = 50;

   keyPair = resources.ec2KeyPairs.my-key-pair;
};

The end result is the identical disk as before -- the option appears to have no effect. If I then use nix-collect-garbage on the EC2 to save some disk space:

[root@ip:~] nix-collect-garbage
...
deleting unused links...
note: currently hard linking saves -0.00 MiB
209 store paths deleted, 1840.89 MiB freed

[root@ip:~] df -h
Filesystem                Size  Used Avail Use% Mounted on
devtmpfs                   50M     0   50M   0% /dev
tmpfs                     495M     0  495M   0% /dev/shm
tmpfs                     248M  4.6M  243M   2% /run
tmpfs                     495M  360K  494M   1% /run/wrappers
/dev/disk/by-label/nixos  3.0G  1.1G  1.8G  38% /
tmpfs                     495M     0  495M   0% /sys/fs/cgroup

...it's clear that /dev/disk/by-label/nixos is being used for the Nix store, and that running nix-collect-garbage reduces its previous 100% use down to 38%. The disk only has 3GB of space, the the same as the xvda disk displayed by lsblk, and nowhere near the 50GB I would expect to see having used ebsInitialRootDiskSize.

This is confirmed by looking in the AWS console after deploying the instance, where I see a running volume of size 3GB instead of the volume of size 50GB I would expect.

Ideal outcome

I would like to use the ebsInitialRootDiskSize or learn about another option available in nixops that would allow me to create a new EC2 instance with more than 3GB of disk space so that I can deploy the simple-twitter project.

thomashoneyman commented 4 years ago

It looks like version 1.7 of NixOps (which I was using) has the ebsInitialRootDiskSize option available. On master, however, NixOps has switched to requiring the nixops-aws plugin in order to make this option available.

If I build NixOps from the master branch and include the aws plugin, as in:

$ nix-build release.nix -A build.x86_64-linux --arg p "(p: [ p.aws ])"

...then, my above issue disappears. Using the ebsInitialRootDiskSize = 20; option produces a volume of size 20GB.

I am still not sure why the version I installed from the 19.09 release channel, version 1.7, states that the option is available in the manual but does not work if the option is actually used. However, given that this appears to work correctly on master I'll close this issue.