Issues with available space when deploying

[thomashoneyman]

When I attempt to deploy an EC2 instance using the readme commands, the following error prevents the deploy from succeeding:

error: writing to file: No space left on device
error: writing to file: Broken pipe

Steps to reproduce

I cloned the repository and made sure nixops was available in my environment with:

git clone git@github.com:Gabriel439/simple-twitter
cd simple-twitter
nix-shell -p nixops

This provides, on my system, nixops-1.7. Next, I followed the readme commands:

nixops create --deployment simple-twitter simple-twitter.nix
nixops deploy --deployment simple-twitter --allow-reboot

This appears to build briefly on my machine and then begin copying files over to the new EC2 via SSH. Next, the 'no space left on device' error appears.

Troubleshooting

I tried bumping up the available space by switching to a t2.micro instance and bumping the disk space from the default of 8gb to 50gb instead:

 ec2 = {
   inherit accessKeyId region;

-  instanceType = "t2.nano";
+  instanceType = "t2.micro";
+
+  ebsInitialRootDiskSize = 50;

   keyPair = resources.ec2KeyPairs.my-key-pair;
};

However, this fails again with the same error. I checked the disk space on the target system:

[root@ip:~]# df -h
Filesystem                Size  Used Avail Use% Mounted on
devtmpfs                   50M     0   50M   0% /dev
tmpfs                     495M     0  495M   0% /dev/shm
tmpfs                     248M  4.6M  243M   2% /run
tmpfs                     495M  360K  494M   1% /run/wrappers
/dev/disk/by-label/nixos  3.0G  3.0G     0 100% /
tmpfs                     495M     0  495M   0% /sys/fs/cgroup

Here, it looks like /dev/disk/by-label/nixos has used 100% of the 3gb available and that's mounted at /; I checked lsblk as well:

[root@ip:~]# lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0   3G  0 disk 
└─xvda1 202:1    0   3G  0 part /

So I certainly don't seem to have 50gb available, despite the use of ebsInitialRootDiskSize = 50. Instead, it looks like just 3gb is available. According to the manual for ebsInitialRootDiskSize:

Preferred size (G) of the root disk of the EBS-backed instance. By default, EBS-backed images have a size determined by the AMI. Only supported on creation of the instance.

From this I expect that 3gb is the default size given to a t2.nano or t2.micro instance and that ebsInitialRootDiskSize = 50 would override that and produce a 50gb volume, but that doesn't appear to be the case.

Given only 3gb of disk space I am not surprised that this deploy would run out of disk space, but I am surprised that you were able to deploy this without seeing a similar error.

If I then use nix-collect-garbage on the destination machine:

...
deleting unused links...
note: currently hard linking saves -0.00 MiB
209 store paths deleted, 1840.89 MiB freed

[root@ip:~]# df -h
Filesystem                Size  Used Avail Use% Mounted on
devtmpfs                   50M     0   50M   0% /dev
tmpfs                     495M     0  495M   0% /dev/shm
tmpfs                     248M  4.6M  243M   2% /run
tmpfs                     495M  360K  494M   1% /run/wrappers
/dev/disk/by-label/nixos  3.0G  1.1G  1.8G  38% /
tmpfs                     495M     0  495M   0% /sys/fs/cgroup

it does appear that the limited 3gb disk is the reason this is failing.

[Gabriella439]

@thomashoneyman: You might want to report this as a bug against the nixops repository, referencing this repository as a reproducing test case. This seems like nixops is sometimes not respecting the ebsInitialRootDiskSize option

[thomashoneyman]

@Gabriel439 I've created and linked an issue in the nixops repository. If it seems like we're correct and the issue is there then I'll close this issue; in the meantime I'll leave this open.

[thomashoneyman]

I've confirmed that the simple-twitter deployment works IF built on a Linux machine, because in that case the build occurs on the user's machine and only the closure is copied over to the EC2 (taking 2.2GB of space in total).

Users will only encounter this disk size issue if building on a non-Linux machine. In that case the build takes place on the EC2 which requires more disk space.

[thomashoneyman]

I can also confirm that I can use the ebsInitialRootDiskSize option in versions of NixOps after 1.6 (I was using 1.7) if NixOps is built with the aws plugin.

NixOps versions until 1.6 shipped with options for various cloud providers available, including AWS. After that version, though, cloud providers were split out into various "plugins" and only if the plugin is installed do you get access to the options. The AWS plugin exists in the separate nixops-aws repository.

If I clone the master branch of the NixOps repository and build it with this command:

$ nix-build release.nix -A build.x86_64-linux --arg p "(p: [ p.aws ])"

and then re-deploy with ebsInitialRootDiskSize = 20; then I properly see a 20GB volume.

[thomashoneyman]

Given that I'm now able to deploy this from MacOS by bumping up the disk size using the master branch of NixOps or from a Linux machine without having to do anything else, I'm going to close this issue.