no more space on GRNET VM

[backeb]

Hi @kkoumantaros @yan0s cc @lorincmeszaros @sebastian-luna-valero

I’ve been doing some work around testing the CMEMS downloads on the GRNET VM and have run out of space building a docker image 😊

When I do a df -h I get:

(base) [centos@hisea-test-vm ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        40G   40G  320K 100% /
devtmpfs         16G     0   16G   0% /dev
tmpfs            16G     0   16G   0% /dev/shm
tmpfs            16G  1.7G   14G  11% /run
tmpfs            16G     0   16G   0% /sys/fs/cgroup
tmpfs           3.2G     0  3.2G   0% /run/user/1000

Assuming /dev/vda1 is our main volume and it tells me its 100% full.

But when I do du -shc * I get

(base) [centos@hisea-test-vm ~]$ du -shc *
4.6G    anaconda3
2.9M    data
23G     _old
8.0K    playground
403M    repos
28G     total

Then checking the docker images by doing docker image ls I get

(base) [centos@hisea-test-vm ~]$ docker image ls
REPOSITORY           TAG       IMAGE ID       CREATED         SIZE
postprocess          latest    c1e973dfcd06   10 months ago   1.36GB
centos               latest    300e315adb2f   19 months ago   209MB
getera               1.0       161b2a634e62   21 months ago   1.23GB
fmphyschem           2.0       f5a5f87eb6dc   21 months ago   1.22GB
fmtide               2.0       4732902cb3dd   21 months ago   1.22GB
deltares/delft3dfm   latest    c03f786c0c91   2 years ago     1.1GB

But the docker images only sum to about 5GB so I’m not sure what’s using up the remaining 7GB.

I am busy scp’ing the data in _old to a server at Deltares to free up some space, but I see in https://ui.cloud.grnet.gr/project/volumes/ we have set up the nfs-server-extra, but its not mounted to anything. Can we change that so we can use the extra storage for our download tests?

Thanks

[sebastian-luna-valero]

To refine this, do

sudo -i
cd /
du -shc *

and keep going down the tree to find out what folder consumes more data.

[backeb]

Thanks @sebastian-luna-valero for the fast response, below the output from your suggestion

(base) [centos@hisea-test-vm ~]$ sudo -i
[root@hisea-test-vm ~]# cd /
[root@hisea-test-vm /]# du -shc *
0       bin
121M    boot
0       data
0       dev
35M     etc
29G     home
0       lib
0       lib64
0       media
0       mnt
655M    opt
0       path
du: cannot access ‘proc/11689/task/11689/fd/3’: No such file or directory
du: cannot access ‘proc/11689/task/11689/fdinfo/3’: No such file or directory
du: cannot access ‘proc/11689/fd/3’: No such file or directory
du: cannot access ‘proc/11689/fdinfo/3’: No such file or directory
0       proc
48K     root
1.7G    run
0       sbin
0       srv
0       sys
380K    tmp
1.2G    usr
9.8G    var
42G     total

The data in home is expected as described in https://github.com/c-scale-community/use-case-hisea/issues/29#issuecomment-1198258197

following var > lib > docker and doing du -shc * there I get

[root@hisea-test-vm docker]# du -shc *
72K     buildkit
12M     containers
17M     image
44K     network
9.0G    overlay2
0       plugins
0       runtimes
0       swarm
0       tmp
0       trust
36K     volumes
9.0G    total

There are a whole bunch of files in overlay2. What do those do? And can they be removed?

root@hisea-test-vm overlay2]# du -shc *
8.0K    007d2b11addca787edd775f5c3d371cca33200c7321d96232abe22163bb85b41
8.0K    007d2b11addca787edd775f5c3d371cca33200c7321d96232abe22163bb85b41-init
1.9M    00d1ae8affaecd20b339943280c826aba2630175f1a7d65320d315598b049cef
8.0K    00d1ae8affaecd20b339943280c826aba2630175f1a7d65320d315598b049cef-init
48K     03c857be45267bf3c0309d711be5719e5f3a624f975093866a383f743be2fc94
8.0K    03c857be45267bf3c0309d711be5719e5f3a624f975093866a383f743be2fc94-init
8.0K    042e6d163fae07e866d93f419d40205f432c803f665aa7fba695a467de72206c
8.0K    047d8ef1d27609a42dcad946943850f6b49d05987bbbfdd460e042e88693b0cb
8.0K    047d8ef1d27609a42dcad946943850f6b49d05987bbbfdd460e042e88693b0cb-init
132K    04ab398867c3799067506a688196412ca9211bda13817600516039b0a3c26b9f
176M    07fd5e70dc488ca17019e448d15752aa256e07509cd5d3eba38a8ec1ce4773f6
8.0K    0844ca89db362ab7c522b814cfe13d6be4fccc96c9c29f7fb01514fb59b36459

[backeb]

A bit of googling... https://stackoverflow.com/questions/46672001/is-it-safe-to-clean-docker-overlay2

when I do docker system df i get

[root@hisea-test-vm overlay2]# docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          6         6         4.69GB    1.222GB (26%)
Containers      130       1         1.232GB   44.45MB (3%)
Local Volumes   6         6         0B        0B
Build Cache     0         0         0B        0B

And then, I kind of stupidly did

docker image prune --all
docker system prune -a

So now I've removed all docker images 🙈

Oh well, at least we have some space again

[root@hisea-test-vm /]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        40G   31G  9.1G  78% /
devtmpfs         16G     0   16G   0% /dev
tmpfs            16G     0   16G   0% /dev/shm
tmpfs            16G  1.7G   14G  11% /run
tmpfs            16G     0   16G   0% /sys/fs/cgroup
tmpfs           3.2G     0  3.2G   0% /run/user/1000

[backeb]

@lorincmeszaros I hope we can easily recreate the docker images I deleted... especially these ones:

fmphyschem           2.0       f5a5f87eb6dc   21 months ago   1.22GB
fmtide               2.0       4732902cb3dd   21 months ago   1.22GB
deltares/delft3dfm   latest    c03f786c0c91   2 years ago     1.1GB

[sebastian-luna-valero]

FYI: https://docs.docker.com/storage/storagedriver/overlayfs-driver/

What's the output of:

sudo docker container list -a

I believe if you clean up unused docker containers, it should free up space under /var/lib/docker/overlay2

[sebastian-luna-valero]

sorry, I only see https://github.com/c-scale-community/use-case-hisea/issues/29#issuecomment-1198298712 now...

[backeb]

Just learnt a whole bunch about docker...

[backeb]

@kkoumantaros @yan0s please provide support to

• [ ] mount the nfs-server-extra

[kkoumantaros]

Hi @Bjorn I do not seem to have access to this vm to help could you please add my key ?

Cheers

K.

On 28 Jul 2022, at 6:28 PM, Bjorn Backeberg @.**@.>> wrote:

@kkoumantaroshttps://github.com/kkoumantaros @yan0shttps://github.com/yan0s please provide support to

[backeb]

Hi @kkoumantaros,

How do I do that? And what's you key?

I use the following command to log on: ssh -i /path/to/hisea-key-pair.pem centos@62.217.83.202

[lorincmeszaros]

@lorincmeszaros I hope we can easily recreate the docker images I deleted... especially these ones:

fmphyschem           2.0       f5a5f87eb6dc   21 months ago   1.22GB
fmtide               2.0       4732902cb3dd   21 months ago   1.22GB
deltares/delft3dfm   latest    c03f786c0c91   2 years ago     1.1GB

no worries. the last one is the model docker image pulled from Docker Hub. The other two are the pre-processign ones and have been updated anyway, so no worries

[kkoumantaros]

Dear all,

I’ve allocated an extra volume on this machine to avoid the issue with not enough space you can find it under /dev/vdb and you can mount it as you wish

if you need my help please add my key (see bellow) in the vm so that I can login .

Cheers

K.

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC49ce6u32rJ/C+Y0Lt5z30LaHN2iQueR0UdUVVsIioIkehiGR80b/ATQWgERwNEbENdbQzmr5Wjw7LfCuQ/uU6TqCd0ih6hx9VZkFSvnc5hfUkE4eouhPF3i/6MEGK87KOlPnj3p8DjO2kiPmw9VMz4jopXWksiE5GYfd+lLHuGFVgyjnzFkaBOI9RWg8XGtZr5MMrgj2BANePfw5jhkIiH2tH/LfDgsMDzSealX658jnDhH1g9e1AXZ2HQC39WR4FA/Xy1D3OugDz1Mc6XLZtzRDLFdhzMQaMGlUNdlRY8gGUMhKbpT4Tv82w1pm9w+cWlt2qMDrIMbewzhp9O/3J @.**@.>

On 29 Jul 2022, at 1:15 PM, Bjorn Backeberg @.**@.>> wrote:

Hi @kkoumantaroshttps://github.com/kkoumantaros,

How do I do that? And what's you key?

I use the following command to log on: ssh -i /path/to/hisea-key-pair.pem @.**@.>

— Reply to this email directly, view it on GitHubhttps://github.com/c-scale-community/use-case-hisea/issues/29#issuecomment-1199108538, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AASE2AFPAGMZ33VMUMIILQLVWOVM5ANCNFSM545SR5DQ. You are receiving this because you were mentioned.Message ID: @.***>

[backeb]

Thanks @kkoumantaros

2 questions:

1. How do I mount the additional volume?
2. Where in the below do I add your public key? authorized_keys?

(base) [centos@hisea-test-vm .ssh]$ ll
total 16
-rw-------. 1 centos centos  381 Apr 28 23:59 authorized_keys
-rw-------. 1 centos centos 3243 Oct 27  2021 id_rsa
-rw-r--r--. 1 centos centos  764 Oct 27  2021 id_rsa.pub
-rw-r--r--. 1 centos centos  974 Oct 27  2021 known_hosts

[kkoumantaros]

you can add it on authorized_keys

and I will mount the extra volume

On 2 Aug 2022, at 12:09 PM, Bjorn Backeberg @.**@.>> wrote:

authorized_keys

[backeb]

@kkoumantaros done, I've add your key.

[sebastian-luna-valero]

Hi, is this issue solved?

[backeb]

as far as I can tell, @kkoumantaros still needs to mount the extra volume...

[kkoumantaros]

I've just mounted the extra volume at

/dev/vdb1 400G 33M 400G 1% /extra I will see if I can mount also the nfs volume on /nfs-volume later on.

[backeb]

Hi @kkoumantaros I see you've mounted the nfs-volume, thank you! Is the nfs-volume visible from your HPC?

[kkoumantaros]

I did not mange to mount the nfs-volume yet we can see if it can be mounted by HPC in a user space later on if the Security policies allow it. Otherwise we can set up rsync to transfer data from one system to the other.

[backeb]

oh interesting!

In the VM I see:

And now when I do df -h I see it mounted:

we can see if it can be mounted by HPC in a user space later on if the Security policies allow it.

That would be cool and could really facilitate the hybrid cloud-HPC workflow we are working towards.

[backeb]

😆 love it ❤️

[kkoumantaros]

yes just mounted it, not tested yet 😆

[backeb]

I can use https://github.com/c-scale-community/use-case-hisea/blob/main/scripts/run_downloadtest.sh to download data to it?

[backeb]

Closing for now - all the additional storage has been mounted.