building centos-8-kvm failed with "No space left on device" at [Download kernel and initrd] task

[n1ete]

finally i got some time to give danube another try on my infrastructure especiallly with "esdc-factory" this time. so i spun up a builder zone on hn and followed README instructions to setup ssh, webserver and and git repo.

make init & make check reported no errors.

wanted to test this with a recent gitlab-ce build on centos-8 kvm instead of the default centos-7 and made changes according to gitlab version 13.6.1 and centos 8 (can pr the changes if needed) at

vars/build/os/contrib-gitlab-ce.yml
vars/build/vm/contrib-gitlab-ce.yml
roles/contrib-gitlab-ce/tasks/main.yml

as far as i understand i need first an centos-8-kvm base image present at webroot on builder so i can build with it contrib-gitlab-ce (?) so i did an make base-centos-8-kvm but this failes with an IOError: [Errno 28] No space left on device (see verbose builder log) factory_no_space_left_error_verbose.log it failes at the point where sha256sum module is used to check boot files for cloud init. the ansible module is depreciated and replaced by 'checksum' i think. will try to update the parts accordingly to see if this could be the cause.

i tried aswell an make contrib-gitlab-ce and make contrib-centos-8-kvm these both are presented me with

TASK [Set base image facts for download and import operations (CentOS)] ********************************************************************************************************************************************
Sunday 29 November 2020  23:51:01 +0000 (0:00:00.042)       0:00:05.103 ******* 
fatal: [buildnode]: FAILED! => {"msg": "An unhandled exception occurred while templating '[{'boot': True, 'compression': \"{{ build_disk_compression | default('lz4') }}\", 'image_size': '{{ base_centos_imgmanifest.image_size }}', 'image_uuid': '{{ base_centos_imgmanifest.uuid }}', 'refreservation': '{{ base_centos_imgmanifest.image_size }}'}]'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while templating '{{ lookup('pipe', 'curl -s {{ builder_base_centos_imgmanifest_url }}') | from_json }}'. Error was a <class 'json.decoder.JSONDecodeError'>, original message: Expecting value: line 1 column 1 (char 0)"}

so i need obviously some lessons and help for using the factory correct and missing probably something obvious.

again a big thank you for sharing this with the community! 🥇

p.s.

two unrelated questions that is not worth another open issue i think:

1. is nfs/smb working from a zone? i think there was an update on this, in a smartos update but i couldnt find it anymore....
2. can i configure lofs mounts over webgui in danube? thanks in advance!

[YanChii]

Hi @n1ete

the No space left error is probably related to default ansible temp dir that points to /root/.ansible/tmp and with USB boot, the rootfs memory disk space is very limited. Add this line to esdc-factory/ansible/ansible.cfg under [defaults] section:

remote_tmp = /var/tmp/ansible_tmp

You've seen correctly that the gitlab image build is based on base-centos-7 image as you can see here and here as default value (you might find useful this command: ./search_var.sh builder_base_centos_imgmanifest_url).

Most of final images are based on some base-* image (classic multistage build used nowadays also in docker). If you don't understand something, just ask, no problem. We will all benefit when more people will use esdc-factory.

And PRs are very welcome, too.

To unrelated questions:

1. AFAIK it should work. This work was done some time ago in illumos. But I didn't test it.
2. not sure I understand this one. Web GUI in Danube is in mgmt01.local VM and this is CentOS. Obviously there's no lofs possible as it's not SunOS zone. But maybe I'm missing you main point why do you need lofs mounts.

Jan

[n1ete]

hey @YanChii thanks for that! your were right!

147 are some proposed changes according to gitlab on centos 8 & deprecated sha256sum replaced with checksum.

the other questions:

1. didnt got it to work till now... so do i use 'zfs share' in zones or do i use 'sharemgr'?
2. i have a large second raidz pool on a node with media content and would love to lofs mount folders into 'lx' or 'joyent' zone via webui at the creation of machine if the storage is localy availble on the machine, so can make use of stateless services sharing localy some folders together from different zones.

[YanChii]

1. as I didn't try it, I cannot tell without experimenting, too. We didn't do NFS sharing in DC so maybe we're not the right place to find out :(. This question however fits for smartos-discuss or omnios-discuss.

2. I understand your aim. This is not supported by DC. It's not even supported by vmadm. You are able to do it manually using zoneadm. But then make sure you don't migrate the VM to other node as it can yield undefined results. It is possible to implement it in DC but it's quite a work because we'd have to issue zoneadm commands which are not part of vmadm and also create / migrate / backup / snapshot / delete calls need be aware of such configuration. However, if you'd like to help us with this implementation, we can have a call on how to make changes in DC code and we can help you with that. Another possibility is to wait as there are rumors that bhyve will be able to directly mount a filesystem from node. That would give us much more flexibility.

[n1ete]

1. will try to find out more... thanks anyways
2. are you sure? i think a lofs definition would just look like this i think in an zonemanifest

   "filesystems": [
   {
     "source": "/somezpool/data/grafana/config",
     "target": "/config",
     "type": "lofs",
     "options": [
       "nodevice"
     ]
   }
   ],

   isnt it easy as that?

edit: had to reread your concerns and understand that this isnt as easy as i thought....:D edit2: ignoring lofs mounts completely for these tasks and handle these zones as any other zone might be easy? usually you want to handle the source data of these mountpoints different anyways.

[YanChii]

You're right. filesystems property in vmadm was made modifiable. So zoneadm is not needed.

If you want to ignore lofs mounts completely by DC, you can just modify the VM manually by vmadm and use it like that. DC will ignore the settings :). Even migrations might work out of the box if you setup the lofs destination fs on the target node.

[n1ete]

closing this here for now since the main issues is resolved thanks to your help ;) mountpoint managment opinion:

i thought /and still think) mount point configuration over webui in danube would be awesome...containers/zones are a powerfull addition to kvm machines, managing moutpoints should be fairly easy to implement since everything is already working or? thanks anyways for your great support here btw. ;)

[YanChii]

If you explore vmadm commands that are needed for mounts management, I'll explore how to add it into GUI/API. In case the implementation is easy, we can do it.