No space left on device

wmertens commented 4 years ago

Running nixops as regular user id 1000 with these specs:

$ df -hl
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         50M     0   50M   0% /dev
tmpfs           494M     0  494M   0% /dev/shm
tmpfs           247M  4.2M  243M   2% /run
tmpfs           494M  336K  494M   1% /run/wrappers
/dev/vda2        23G   12G   11G  52% /
/dev/vda2        23G   12G   11G  52% /nix/store
tmpfs           494M     0  494M   0% /sys/fs/cgroup
tmpfs            99M   16K   99M   1% /run/user/1000
tmpfs            99M     0   99M   0% /run/user/1003
tmpfs            99M     0   99M   0% /run/user/0

causes the opaque error tar: no space left on device

The reason is that nixops uses TMPDIR which points to /run and is limited to 100M.

What would be a good way to tell the user about this problem?

GTrunSec commented 3 years ago

So, How to solve this problem, Does anyone idea here?

tewfik-ghariani commented 3 years ago

Can you share more information about the issue? Does the error occur when you launch a 'nixops deploy' operation? Can you check if you have enough space in the deployment server itself? Can you try to run a nix-collect-garbage to free up some space from the nix store?

GTrunSec commented 3 years ago

Can you share more information about the issue? Does the error occur when you launch a 'nixops deploy' operation?

machine setting:

{
  photo =
    { config, pkgs, lib, ... }:
    {
      deployment.targetEnv = "virtualbox";
      deployment.virtualbox.memorySize = 6028; # megabytes
      deployment.virtualbox.vcpu = 4; # number of cpus
      virtualisation.virtualbox.guest.enable = lib.mkForce false;
      boot.tmpOnTmpfs = true;
    };
}

check machine info:

nixops check -d photo    
Machines state:
+-------+--------+-----+-----------+----------+----------------+------------------------------------------+-------+
| Name  | Exists | Up  | Reachable | Disks OK | Load avg.      | Units                                    | Notes |
+-------+--------+-----+-----------+----------+----------------+------------------------------------------+-------+
| photo | Yes    | Yes | Yes       | N/A      | 0.39 0.28 0.16 | ● home.mount [failed]                    |       |
|       |        |     |           |          |                |   sys-fs-fuse-connections.mount [failed] |       |
|       |        |     |           |          |                |   sys-kernel-config.mount [failed]       |       |
+-------+--------+-----+-----------+----------+----------------+------------------------------------------+-------+
Non machines resources state:
+------+--------+
| Name | Exists |
+------+--------+
+------+--------+

nixops ssh -d photo photo (build package test failed!)

nix-shell -p git 
git clone https://github.com/GTrunSec/photoprism-flake
cd photoprism-flake/
nix-build

error output:

go: downloading github.com/dsoprea/go-photoshop-info-format v0.0.0-20200610045659-121dd752914dgo: downloading github.com/go-xmlfmt/xmlfmt v0.0.0-20191208150333-d5b6f63a941b-> unzip /build/go/pkg/mod/cache/download/github.com/tensorflow/tensorflow/@v/v1.15.2.zip: write /build/go/pkg/mod/github.com/tensorflow/tensorflow@v1.15.2.tmp-911731347/tensorflow/python/kernel_tests/sparse_conditional_accumulator_test.py: no space left on device
github.com/photoprism/photoprism/internal/classify imports
        github.com/tensorflow/tensorflow/tensorflow/go: unzip /build/go/pkg/mod/cache/download/github.com/tensorflow/tensorflow/@v/v1.15.2.zip: write /build/go/pkg/mod/github.com/tensorflow/tensorflow@v1.15.2.tmp-911731347/tensorflow/python/kernel_tests/sparse_conditional_accumulator_test.py: no space left on device
github.com/photoprism/photoprism/internal/nsfw imports
        github.com/tensorflow/tensorflow/tensorflow/go/op: unzip /build/go/pkg/mod/cache/download/github.com/tensorflow/tensorflow/@v/v1.15.2.zip: write /build/go/pkg/mod/github.com/tensorflow/tensorflow@v1.15.2.tmp-911731347/tensorflow/python/kernel_tests/sparse_conditional_accumulator_test.py: no space left on device

df check

df -ih
Filesystem               Inodes IUsed IFree IUse% Mounted on
/dev/disk/by-label/nixos   3.2M  210K  3.0M    7% /
tmpfs                      732K    18  732K    1% /sys/fs/cgroup
devtmpfs                   730K   379  730K    1% /dev
tmpfs                      732K     1  732K    1% /dev/shm
tmpfs                      732K   898  731K    1% /run
tmpfs                      732K    39  732K    1% /run/wrappers
tmpfs                      147K    15  147K    1% /run/user/0
tmpfs                      732K    10  732K    1% /tmp

Thanks for your helping.

GTrunSec commented 3 years ago

solved by resizing the tmp

tewfik-ghariani commented 3 years ago

Cool!

cc @wmertens did you face the exact same issue?

wmertens commented 3 years ago

yes but I solved it by setting TMPDIR=/tmp. I don't have tmpOnTmpfs.

wmertens commented 3 years ago

Perhaps there should be a check the available disk space on $TMPDIR, if it's less than e.g. 500MB print a warning?

NixOS / nixops

No space left on device #1402