IBT-FMI / gebuilder

Gentoo System and Image Builder
GNU General Public License v3.0
11 stars 0 forks source link

No space left on device errors #24

Closed TheChymera closed 6 years ago

TheChymera commented 6 years ago

I keep getting a lot of these errors:

ERROR:root:Failed to copy file: _parsed_options=Namespace(group=-1, mode=420, owner=-1, preserve_timestamps=False), source=b'dsurqec_40micron_mask.nii', dest_dir=b'/var/tmp/portage/sci-biology/mouse-brain-atlases-0.1[193/1924$
image/usr/share/mouse-brain-atlases'                                                                                                                                                                                              
Traceback (most recent call last):                                                                                                                                                                                                
  File "/usr/lib/portage/python3.5/doins.py", line 209, in run                                                                                                                                                                    
    copyfile(source, dest)                                                                                                                                                                                                        
  File "/usr/lib64/python3.5/site-packages/portage/util/file_copy/__init__.py", line 30, in _optimized_copyfile                                                                                                                   
    _file_copy(src_file.fileno(), dst_file.fileno())                                                                                                                                                                              
OSError: [Errno 28] No space left on device                                                                                                                                                                                       
ERROR:root:Failed to copy file: _parsed_options=Namespace(group=-1, mode=420, owner=-1, preserve_timestamps=False), source=b'dsurqec_40micron_masked.nii', dest_dir=b'/var/tmp/portage/sci-biology/mouse-brain-atlases-0.1.2018062
3/image/usr/share/mouse-brain-atlases'                                                                                                                                                                                            
Traceback (most recent call last):                                                                                                                                                                                                
  File "/usr/lib/portage/python3.5/doins.py", line 209, in run                                                                                                                                                                    
    copyfile(source, dest)                                                                                                                                                                                                        
  File "/usr/lib64/python3.5/site-packages/portage/util/file_copy/__init__.py", line 30, in _optimized_copyfile                                                                                                                   
    _file_copy(src_file.fileno(), dst_file.fileno())                                                                                                                                                                              
OSError: [Errno 28] No space left on device                                                                                                                                                                                       
ERROR:root:Failed to copy file: _parsed_options=Namespace(group=-1, mode=420, owner=-1, preserve_timestamps=False), source=b'lambmc_15micron.nii', dest_dir=b'/var/tmp/portage/sci-biology/mouse-brain-atlases-0.1.20180623/image/
usr/share/mouse-brain-atlases'                                                                                                                                                                                                    
Traceback (most recent call last):                                                                                                                                                                                                
  File "/usr/lib/portage/python3.5/doins.py", line 209, in run                                                                                                                                                                    
    copyfile(source, dest)                                                                                                                                                                                                        
  File "/usr/lib64/python3.5/site-packages/portage/util/file_copy/__init__.py", line 30, in _optimized_copyfile                                                                                                                   
    _file_copy(src_file.fileno(), dst_file.fileno())                                                                                                                                                                              
OSError: [Errno 28] No space left on device                                                                                                                                                                                       
ERROR:root:Failed to copy file: _parsed_options=Namespace(group=-1, mode=420, owner=-1, preserve_timestamps=False), source=b'lambmc_180micron.nii', dest_dir=b'/var/tmp/portage/sci-biology/mouse-brain-atlases-0.1.20180623/image
/usr/share/mouse-brain-atlases'                                                                                                                                                                                                   
Traceback (most recent call last):                                                                                                                                                                                                
  File "/usr/lib/portage/python3.5/doins.py", line 209, in run                                                                                                                                                                    
    copyfile(source, dest)                                                                                                                                                                                                        
  File "/usr/lib64/python3.5/site-packages/portage/util/file_copy/__init__.py", line 30, in _optimized_copyfile                                                                                                                   
    _file_copy(src_file.fileno(), dst_file.fileno())                                                                                                                                                                              
OSError: [Errno 28] No space left on device                                                                                                                                                                                       
ERROR:root:Failed to copy file: _parsed_options=Namespace(group=-1, mode=420, owner=-1, preserve_timestamps=False), source=b'lambmc_45micron.nii', dest_dir=b'/var/tmp/portage/sci-biology/mouse-brain-atlases-0.1.20180623/image/
usr/share/mouse-brain-atlases'                                                                                                                                                                                                    
Traceback (most recent call last):                                                                                                                                                                                                
  File "/usr/lib/portage/python3.5/doins.py", line 209, in run                                                                                                                                                                    
    copyfile(source, dest)                                                                                                                                                                                                        
  File "/usr/lib64/python3.5/site-packages/portage/util/file_copy/__init__.py", line 30, in _optimized_copyfile                                                                                                                   
    _file_copy(src_file.fileno(), dst_file.fileno())                                                                                                                                                                              
OSError: [Errno 28] No space left on device                                                                                                                                                                                       
/usr/lib/portage/python3.5/isolated-functions.sh: line 272: echo: write error: No space left on device                                                                                                                            
 * ERROR: sci-biology/mouse-brain-atlases-0.1.20180623::science failed (install phase):                                                                                                                                           
/usr/lib/portage/python3.5/isolated-functions.sh: line 272: echo: write error: No space left on device                                                                                                                            
 *   doins failed                                                                                                                                                                                                                 
/usr/lib/portage/python3.5/isolated-functions.sh: line 272: echo: write error: No space left on device                                                                                                                            
 *                                                                                                                                                                                                                                
/usr/lib/portage/python3.5/isolated-functions.sh: line 272: echo: write error: No space left on device                                                                                                                            
 * If you need support, post the output of `emerge --info '=sci-biology/mouse-brain-atlases-0.1.20180623::science'`,                                                                                                              
/usr/lib/portage/python3.5/isolated-functions.sh: line 272: echo: write error: No space left on device                                                                                                                            
 * the complete build log and the output of `emerge -pqv '=sci-biology/mouse-brain-atlases-0.1.20180623::science'`.                                                                                                               
/usr * Messages for package media-fonts/liberation-fonts-2.00.1-r3:                                                                                                                                                               
 * The following fontconfig configuration files have been installed:                                                                                                                                                              
 *                                                                                                                                                                                                                                
 *   60-liberation.conf                                                                                                                                                                                                           
 *                                                                                                                                                                                                                                
 * Use `eselect fontconfig` to enable/disable them.                                                                                                                                                                               

 * IMPORTANT: 13 news items need reading for repository 'gentoo'.                                                                                                                                                                 
 * Use eselect news read to view new 

Strangely enough (and this is with #DELETE_ON_FAIL=1) df -h gives me:

Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         10M     0   10M   0% /dev
tmpfs           7.7G     0  7.7G   0% /dev/shm
tmpfs           7.7G  328K  7.7G   1% /run
/dev/vda1       100G   15G   82G  16% /
cgroup_root      10M     0   10M   0% /sys/fs/cgroup
none            6.0G     0  6.0G   0% /usr/share/gebuilder/roots/e6966aabd76aeb9189f3f03a02c59681a37ffdbe2d89c8d842a071807422ccda/root/var/tmp/portage
none            1.0G     0  1.0G   0% /usr/share/gebuilder/roots/e6966aabd76aeb9189f3f03a02c59681a37ffdbe2d89c8d842a071807422ccda/root/tmp

@Doeme ideas?

TheChymera commented 6 years ago

Chrooting into the file and starting the emerge process, it really does look like it's filling up:

newbuilder / # df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         10M     0   10M   0% /dev
tmpfs           7.7G     0  7.7G   0% /dev/shm
none            6.0G  6.0G     0 100% /var/tmp/portage
none            1.0G     0  1.0G   0% /tmp

I looked into the occurences of 6G, and it appears that this is the only file where such a size is specified, editing it (to 16G, see below), however, did not fix the problem:

newbuilder /usr/share/gebuilder # cat scripts/common/mount_basic.sh
#!/bin/bash

for file in dev dev/pts proc sys var/tmp/portage tmp
do
        on_exit "umount -R \"${ROOT}/$file\""
done

pushd "${ROOT}"
debug "recursively bind-mount /dev to dev/"
mount --rbind /dev/ dev
debug "mount proc/"
mount -t proc none proc
debug "mount sys/"
mount -t sysfs none sys
debug "mount devpts with gid=5 for glibc[-suid] at /dev/pts/"
mount -t devpts -o gid=5 none dev/pts/
debug "mount tmpfd on /var/tmp/portage"
mkdir -p var/tmp/portage
mount -t tmpfs -o size=16G none var/tmp/portage
debug "mounting tmpfs on /tmp/"
mount -t tmpfs -o size=1G none tmp
popd
Doeme commented 6 years ago

Yeah, maybe we should just stop mounting var/tmp/portage as tmpfs in the first place. This might be detrimental when building openstack images, because we build some smaller software in a tight space-constraint, but should not have a large impact otherwise.

Usually, packages (when sufficiently resource consuming) check whether they will exceed the capacity of /var/tmp/portage/ via check-reqs.eclass, but sci-biology/mouse-brain-atlases doesn't seem to do that.

TheChymera commented 6 years ago

I'll first try to fix it package-side, then. It's my package.

TheChymera commented 6 years ago

Defining the respective variables for the package does not seem to have done the trick:

# Copyright 1999-2018 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

EAPI=6

inherit check-reqs

DESCRIPTION="A collection of mouse brain atlases in NIfTI format"
HOMEPAGE="http://imaging.org.au/AMBMC/Model"
SRC_URI="http://chymera.eu/pkgdata/${P}.zip"

LICENSE="fairuse"
SLOT="0"
KEYWORDS="~amd64 ~x86 ~amd64-linux ~x86-linux"
IUSE=""

RDEPEND=""
DEPEND=""

CHECKREQS_DISK_BUILD=4G
CHECKREQS_DISK_USR=4G
CHECKREQS_DISK_VAR=8G

src_install() {
        insinto "/usr/share/${PN}"
        doins *
}

Results in (note how it does check for space a the beginning):

 * Package:    sci-biology/mouse-brain-atlases-0.1.20180623
 * Repository: science
 * Maintainer: horea.christ@gmail.com sci@gentoo.org
 * USE:        abi_x86_64 amd64 elibc_glibc kernel_linux userland_GNU
 * FEATURES:   preserve-libs sandbox userpriv usersandbox
 * Checking for at least 4 GiB disk space at "/var/tmp/portage/sci-biology/mouse-brain-atlases-0.1.20180623/temp" ...
 [ ok ]
 * Checking for at least 4 GiB disk space at "//usr" ...
 [ ok ]
 * Checking for at least 8 GiB disk space at "//var" ...
 [ ok ]
>>> Unpacking source...
>>> Unpacking mouse-brain-atlases-0.1.20180623.zip to /var/tmp/portage/sci-biology/mouse-brain-atlases-0.1.20180623/work
>>> Source unpacked in /var/tmp/portage/sci-biology/mouse-brain-atlases-0.1.20180623/work
>>> Preparing source in /var/tmp/portage/sci-biology/mouse-brain-atlases-0.1.20180623/work/mouse-brain-atlases-0.1.20180623 ...
>>> Source prepared.
>>> Configuring source in /var/tmp/portage/sci-biology/mouse-brain-atlases-0.1.20180623/work/mouse-brain-atlases-0.1.20180623 ...
>>> Source configured.
>>> Compiling source in /var/tmp/portage/sci-biology/mouse-brain-atlases-0.1.20180623/work/mouse-brain-atlases-0.1.20180623 ...
>>> Source compiled.
>>> Test phase [not enabled]: sci-biology/mouse-brain-atlases-0.1.20180623

>>> Install mouse-brain-atlases-0.1.20180623 into /var/tmp/portage/sci-biology/mouse-brain-atlases-0.1.20180623/image/ category sci-biology
ERROR:root:Failed to copy file: _parsed_options=Namespace(group=-1, mode=420, owner=-1, preserve_timestamps=False), source=b'dsurqec_40micron_mask.nii', dest_dir=b'/var/tmp/portage/sci-biology/mouse-brain-atlases-0.1.20180623/image/usr/share/mouse-brain-atlases'
Traceback (most recent call last):
  File "/usr/lib/portage/python3.5/doins.py", line 209, in run
    copyfile(source, dest)
  File "/usr/lib64/python3.5/site-packages/portage/util/file_copy/__init__.py", line 30, in _optimized_copyfile
    _file_copy(src_file.fileno(), dst_file.fileno())
OSError: [Errno 28] No space left on device
ERROR:root:Failed to copy file: _parsed_options=Namespace(group=-1, mode=420, owner=-1, preserve_timestamps=False), source=b'dsurqec_40micron_masked.nii', dest_dir=b'/var/tmp/portage/sci-biology/mouse-brain-atlases-0.1.20180623/image/usr/share/mouse-brain-atlases'
Traceback (most recent call last):
  File "/usr/lib/portage/python3.5/doins.py", line 209, in run
    copyfile(source, dest)
  File "/usr/lib64/python3.5/site-packages/portage/util/file_copy/__init__.py", line 30, in _optimized_copyfile
    _file_copy(src_file.fileno(), dst_file.fileno())
OSError: [Errno 28] No space left on device
ERROR:root:Failed to copy file: _parsed_options=Namespace(group=-1, mode=420, owner=-1, preserve_timestamps=False), source=b'lambmc_15micron.nii', dest_dir=b'/var/tmp/portage/sci-biology/mouse-brain-atlases-0.1.20180623/image/usr/share/mouse-brain-atlases'
Traceback (most recent call last):
  File "/usr/lib/portage/python3.5/doins.py", line 209, in run
    copyfile(source, dest)
  File "/usr/lib64/python3.5/site-packages/portage/util/file_copy/__init__.py", line 30, in _optimized_copyfile
    _file_copy(src_file.fileno(), dst_file.fileno())
OSError: [Errno 28] No space left on device
ERROR:root:Failed to copy file: _parsed_options=Namespace(group=-1, mode=420, owner=-1, preserve_timestamps=False), source=b'lambmc_180micron.nii', dest_dir=b'/var/tmp/portage/sci-biology/mouse-brain-atlases-0.1.20180623/image/usr/share/mouse-brain-atlases'
Traceback (most recent call last):
  File "/usr/lib/portage/python3.5/doins.py", line 209, in run
    copyfile(source, dest)
  File "/usr/lib64/python3.5/site-packages/portage/util/file_copy/__init__.py", line 30, in _optimized_copyfile
    _file_copy(src_file.fileno(), dst_file.fileno())
OSError: [Errno 28] No space left on device
ERROR:root:Failed to copy file: _parsed_options=Names
Doeme commented 6 years ago

Interesting. Not sure why the check-reqs doesn't do its job here...

Anyways, checking the resources only results in an earlier build error, and does not really address the problem. We either need to enlargen the tmpfs or just not mount it at all.

TheChymera commented 6 years ago

you want to make the choice yourself or should we discuss pro/cons?

TheChymera commented 6 years ago

Fixed as per https://github.com/IBT-FMI/gebuilder/commit/2a6d939d08b650aa7144c47af59bfc2f245fce0a --- reopen if you want to discuss this further.