Closed mirabilos closed 8 months ago
I will fix it
Hi !
Maybe also installing automake
, autoconf
and libtool
, as they are often used (we use it for instance on the PPP project).
Thank you for your VMs !
@RICCIARDI-Adrien Here are 2 opertunities to install packages:
if the pkg can be installed by pkgutil
, you can just add a VM_PRE_INSTALL_PKGS
item, in the conf file: "https://github.com/vmactions/solaris-builder/blob/main/conf/solaris-11.4.conf"
Example:
https://github.com/vmactions/netbsd-builder/blob/main/conf/netbsd-9.3.conf#L16C1-L16C20
Otherwise, you can install it in the hooks/postBuild.sh
, this scrip will be executed in the vm as soon as it's built successully.
https://github.com/vmactions/solaris-builder/blob/main/hooks/postBuild.sh
Example:
https://github.com/vmactions/freebsd-builder/blob/main/hooks/postBuild.sh
Please clone and send PR. I can include it in next release.
Is the PR https://github.com/vmactions/solaris-builder/pull/2 OK for both of you @mirabilos and @Neilpang ?
@RICCIARDI-Adrien if it does what it says then yes ;-) thank you <3
the image size is doubled, see my comments: https://github.com/vmactions/solaris-builder/pull/2
what do you think?
That’s to be expected. But do:
zerofree
on it afterwardsqcow2
file
qemu-img
to copy the qcow2 file to another qcow2 file in which the now-zero-filled blocks will not be present any more: qemu-img convert -O qcow2 original_image.qcow2 deduplicated_image.qcow2
qcow2
can also do compression but it uses zlib and isn’t very useful later on when the VM runs, so keep xz
ing it, I supposeSeeing that zerofree
is for ext{2,3,4} only and Solaris uses… what? UFS?, you’ll have to find another tool. If none exists, the “wood hammer” method would be:
dd if=/dev/zero of=/mnt/tmpf1 bs=1048576
followed by dd if=/dev/zero of=/mnt/tmpf2
, dd if=/dev/zero of=/mnt/tmpf3
, … (to free up subblocks) until the last /mnt/tmpf
n has zero size, sync
, then delete the tmpf
* files again and umount the filesystemsFrom the above-linked StackExchange post, virt-sparsify
might work if installing libguestfs-tools
and libguestfs-zfs
. Best try this locally first, I suppose?
The cleaning the caches can be done in the VM (e.g. in postBuild.sh
) or after the VM was shut down and its filesystems mounted in the Linux builder host (which has the benefits that you can also clear out /tmp
, /var/tmp
, perhaps reduce the size of syslog files to 0 while keeping them (with uid/gid/perms intact) with e.g. : >/mnt/var/log/messages
, etc.).
Everything else (zeroïng, compacting) must be done with the VM shat off. For the final compacting step, it must also be umounted and otherwise completely disengaged from the builder host again.
can anyone figure out which of the packages is the largest?
VM_PRE_INSTALL_PKGS="autoconf automake gcc5core gcc5g++ libtool rsync"
I tried qemu-img convert -O
, it has little effect.
neil dixit:
I tried
qemu-img convert -O
, it has little effect.
You need to zero out the empty blocks in the filesystem first, of course. Only these can be made sparse.
bye, //mirabilos -- Gestern Nacht ist mein IRC-Netzwerk explodiert. Ich hatte nicht damit gerechnet, darum bin ich blutverschmiert… wer konnte ahnen, daß SIE so reagier’n… gestern Nacht ist mein IRC-Netzwerk explodiert~~~ (as of 2021-06-15 The MirOS Project temporarily reconvenes on OFTC)
Dixi quod…
You need to zero out the empty blocks in the filesystem first,
And before that, delete not-needed files so the blocks become empty/unused of course ☻
bye, //mirabilos -- Gestern Nacht ist mein IRC-Netzwerk explodiert. Ich hatte nicht damit gerechnet, darum bin ich blutverschmiert… wer konnte ahnen, daß SIE so reagier’n… gestern Nacht ist mein IRC-Netzwerk explodiert~~~ (as of 2021-06-15 The MirOS Project temporarily reconvenes on OFTC)
@Neilpang I think that the biggest packages are gcc and g++, I will try without installing them to see if there is a significant change on the VM image size.
@RICCIARDI-Adrien great. please just fork the repo to run the build actions, it will be easier for you. just one lick.
Adrien RICCIARDI dixit:
@Neilpang I think that the biggest packages are gcc and g++, I will try @without installing them to see if there is a significant change on the @VM image size.
Without them, the whole thing makes no sense, though. You cannot even reasonably use autoconf and friends without.
@mirabilos Indeed, but it would be a trade-off : you would still need to install less dependencies, like rsync in your case.
Adrien RICCIARDI dixit:
@mirabilos Indeed, but it would be a trade-off : you would still need @to install less dependencies, like rsync in your case.
Yes, that saves 30 seconds off 12 minutes or so…
Without gcc and g++ :
Changing to init state 5 - please wait
Exporting solaris-11.4.qcow2
total 9.4G
drwxr-xr-x 6 runner docker 4.0K Dec 8 12:03 .
drwxr-xr-x 3 runner docker 4.0K Dec 8 11:13 ..
drwxr-xr-x 8 runner docker 4.0K Dec 8 11:13 .git
drwxr-xr-x 3 runner docker 4.0K Dec 8 11:13 .github
-rw-r--r-- 1 runner docker 1.1K Dec 8 11:13 LICENSE
-rw-r--r-- 1 runner docker 17 Dec 8 11:13 README.md
-rw-r--r-- 1 runner docker 4.0K Dec 8 11:13 build.sh
drwxr-xr-x 2 runner docker 4.0K Dec 8 11:13 conf
-rw-r--r-- 1 runner docker 1.9K Dec 8 11:51 enablessh.local
-rw-r--r-- 1 runner docker 396 Dec 8 11:13 enablessh.txt
drwxr-xr-x 2 runner docker 4.0K Dec 8 11:13 hooks
-rw-r--r-- 1 runner docker 2.1K Dec 8 11:57 index.html
-rw-rw-rw- 1 runner docker 44K Dec 8 11:57 screen.png
-rw-r--r-- 1 runner docker 1.9K Dec 8 11:57 screen.txt
-rw------- 1 runner docker 2.6K Dec 8 12:03 solaris-11.4-host.id_rsa
-rw-r--r-- 1 runner docker 394 Dec 8 11:55 solaris-11.4-id_rsa.pub
-rw-r--r-- 1 root root 3.9G Dec 8 11:58 solaris-11.4.qcow2
-rw-r--r-- 1 root root 947M Dec 8 11:58 solaris-11.4.qcow2.xz
-rw-r--r-- 1 libvirt-qemu kvm 708M Dec 8 11:14 solaris.iso
-rw-r--r-- 1 runner docker 3.9G Dec 8 11:57 solaris.qcow2
-rwxr-xr-x 1 runner docker 13K Dec 8 11:13 vbox.sh
how about g++ only or gcc only?
I agree with @mirabilos that the most important packages are indeed gcc and g++. So there are two choices left : 1) Do not install gcc and g++ and just install the "light" packages, this will save a really little percentage of CI execution time. 2) Try to make room in the VM image after gcc and g++ have been installed. Do we investigate the second solution with the clues given by @mirabilos ? I don't know Solaris, @mirabilos would you like to try to make space in the image ?
@Neilpang I do another test with gcc only, which might be more often used than g++.
With gcc only :
Exporting solaris-11.4.qcow2
total 16G
drwxr-xr-x 6 runner docker 4.0K Dec 8 14:08 .
drwxr-xr-x 3 runner docker 4.0K Dec 8 13:10 ..
drwxr-xr-x 8 runner docker 4.0K Dec 8 13:10 .git
drwxr-xr-x 3 runner docker 4.0K Dec 8 13:10 .github
-rw-r--r-- 1 runner docker 1.1K Dec 8 13:10 LICENSE
-rw-r--r-- 1 runner docker 17 Dec 8 13:10 README.md
-rw-r--r-- 1 runner docker 4.0K Dec 8 13:10 build.sh
drwxr-xr-x 2 runner docker 4.0K Dec 8 13:10 conf
-rw-r--r-- 1 runner docker 1.9K Dec 8 13:46 enablessh.local
-rw-r--r-- 1 runner docker 396 Dec 8 13:10 enablessh.txt
drwxr-xr-x 2 runner docker 4.0K Dec 8 13:10 hooks
-rw-r--r-- 1 runner docker 2.2K Dec 8 13:56 index.html
-rw-rw-rw- 1 runner docker 40K Dec 8 13:56 screen.png
-rw-r--r-- 1 runner docker 1.9K Dec 8 13:56 screen.txt
-rw------- 1 runner docker 2.6K Dec 8 14:08 solaris-11.4-host.id_rsa
-rw-r--r-- 1 runner docker 394 Dec 8 13:50 solaris-11.4-id_rsa.pub
-rw-r--r-- 1 root root 6.6G Dec 8 13:57 solaris-11.4.qcow2
-rw-r--r-- 1 root root 1.7G Dec 8 13:57 solaris-11.4.qcow2.xz
-rw-r--r-- 1 libvirt-qemu kvm 708M Dec 8 13:12 solaris.iso
-rw-r--r-- 1 runner docker 6.6G Dec 8 13:56 solaris.qcow2
-rwxr-xr-x 1 runner docker 13K Dec 8 13:11 vbox.sh
As expected, gcc is quite fat.
did you try to remove the temp files?
neil dixit:
how about g++ only or gcc only?
g++ is just a small addon package to gcc and depends on the latter, of course.
Adrien RICCIARDI dixit:
Do we investigate the second solution with the clues given by @mirabilos ? I don't know Solaris, @mirabilos would you like to try to make space in the image ?
Please do.
I don’t know Solaris either, I just shared my usual methods for cleaning up disc images.
Adrien RICCIARDI dixit:
As expected, gcc is quite fat.
No, it has many dependencies.
tbh, I’d be happy with SUNWcc instead of GCC, but that’s apparently not likely to happen. ☹
neil dixit:
did you try to remove the temp files?
And package downloads and probably even package indicēs and so on…
bye,
//mirabilos
--
21:12⎜
I just added a new tag for gcc. please use:
with:
release: "11.4-gcc"
thanks.
@Neilpang This is a nice solution to create a dedicated tag for the gcc version. I just tested it and it works well, decreasing the action execution time from about 6min 40sec to about 5min 15sec. Thank you !
neil dixit:
I just added a new tag for gcc. please use:
with: release: "11.4-gcc"
thanks.
Thank you❣ This works.
Is the OCR stuff (tesseract, vnc…) still needed? (If not, don’t even install it when preparing host.)
Can the VM image download reasonably be cached (by the action)? (I don’t know GHA’s size limits for these things, and when I have a workflow for each VM, the sizes would accumulate, so I don’t know if that’s possible/sensible.)
bye, //mirabilos -- «MyISAM tables -will- get corrupted eventually. This is a fact of life. » “mysql is about as much database as ms access” – “MSSQL at least descends from a database” “it's a rebranded SyBase” “MySQL however was born from a flatfile and went downhill from there” – “at least jetDB doesn’t claim to be a database” ‣‣‣ Please, http://deb.li/mysql and MariaDB, finally die!
Hi,
I’d love if you could pre-install rsync on the image at image creation time, so that it isn’t needed to be done every time the job is run. This also saves electricity, which is a plus in these times!
This really affects all vmactions, but the Solaris one is the slowest.
You could at the same time fix #33 by also pre-installing
gcc5core
andgcc5g++
so it’s not the only VM without a compiler, and installing that takes ¼h or so already (ouch).This currently precludes me from just letting GHA run the vmactions jobs CI-style instead of running them manually occasionally.