Move Developer files to separate download

[probonopd]

Starting with 0.7.0, helloSystem will no longer contain developer-centric files and non-localized files on the ISO. Those files will be moved out to a separate download.

Apparently some "developer stuff" is required for the operation of FreeBSD by "mere mortals" who do not even know what a compiler is:

• [x] libLLVM-12.so needed for swrast_dri.so

libEGL warning: DRI2: failed to authenticate
libEGL warning: MESA-LOADER: failed to open swrast: Shared object "libLLVM-12.so" not found, required by "swrast_dri.so" (search paths /usr/local/lib/dri)

Unfortunately, this is a 87 MB file. Is there any way around this?

[probonopd]

Let's see if things improve with https://github.com/helloSystem/ISO/commit/e26134cfef0f416cdba456b756cf53edfdd8b109

ISO size before: 965 MB, after: 992 MB

[probonopd]

One idea to create the separate user and developer filesystem trees:

1. Write a developer.exclude list that contains fnmatch(3) search patterns for the developer files
2. Use mtree(8) to generate a spec file
3. Use the spec file when generating the filesystem image with makefs(8)

There we have it - no copying around of trees, no deleting of files from the trees.

Seems like the BSD people have thought of everything you might need when putting together systems. :+1:

[probonopd]

Say this is our user.exclude:

# If the pattern contains a `/' character,
# it will be matched against entire pathnames (relative
# to the starting directory); otherwise, it will be
# matched against basenames only

doc
docs
*.la
man
/usr/include
/usr/local/include
*.h
.cache
debug
*.a
*.o
src
git-core
git
devhelp
*-doc
examples
/usr/bin/svn*
/usr/bin/clang*
/usr/bin/c++
/usr/bin/cpp
/usr/bin/cc
/usr//bin/lldb*
/usr/local/bin/ccxxmake
/usr/bin/llvm*
/usr/bin/ld.lld
/usr/bin/ex
/usr/bin/nex
/usr/bin/nvi
/usr/bin/vi
/usr/bin/view

# TODO: Delete /usr/local/llvm* EXCEPT for 'libLLVM-*.so*'
# Must not delete libLLVM-12.so which is needed for swrast_dri.so
# TODO: Translate the following to a fnmatch(3) pattern
# find /usr/local/llvm* -not -name "libLLVM-*.so*"

Then we can create the spec file with:

 mtree -p /usr/local/furybsd/uzip/ -c -X user.exclude >  user.spec

With that spec file we could generate the user filesystem image.

Now, to generate the developer filesystem image, we need the exact reverse of the excluded files. In other words, everything that is in the user.spec file should be excluded this time. Hence, we would need to convert the content of the user.spec file into an excludelist. Can it be converted into that format?

mtree -C -R all -f spec > developer.exclude
sed -i '' -e 's|^\.||g' developer.exclude

With that, we can generate the spec file for the developer filesystem image:

mtree -p /usr/local/furybsd/uzip/ -c -X developer.exclude >  developer.spec

This takes forever. There must be a better way!

In the end, we need to compare the number of files to see whether it has worked.

[probonopd]

The following should work:

1. create a manifest and convert it to 1-liner format
2. add a comment to the developer files in the manifest using regex (sed)
3. copy manifest 2x
4. leave only the marked lines in the developer manifest
5. in the user manifest leave only the unmarked lines

[probonopd]

cd  "${uzip}"

mtree -p  . -c > "${livecd}"/spec
mtree -C -f "${livecd}"/spec > "${livecd}"/spec.user
sed -i '' -e 's|^\./usr/include/.*|& # developer|' "${livecd}"/spec.user
sed -i '' -e 's|^\./usr/lib/clang/.*/include/.*|& # developer|' "${livecd}"/spec.user
# ...
cp "${livecd}"/spec.user "${livecd}"/spec.developer
sed -i '' -e '/# developer/d' "${livecd}"/spec.user
sed -i '' -e '/# developer/!d' "${livecd}"/spec.developer
makefs "${cdroot}/rootfs.ufs" "${livecd}"/spec.user
makefs "${iso}/developer.ufs" "${livecd}"/spec.developer

developerimagename=$(basename $(echo ${isopath} | sed -e 's|.iso$|.developer.img|g'))

if [ $MAJOR -lt 13 ] ; then
  mkuzip -o "${iso}/${developerimagename}" "${iso}/${iso}.ufs"
else
  # Use zstd when possible, which is available in FreeBSD beginning with 13
  mkuzip -A zstd -C 15 -o "${iso}/${developerimagename}" "${iso}/developer.ufs"
fi

rm "${iso}/developer.ufs"
cd -

[probonopd]

This has been implemented in https://github.com/helloSystem/ISO/commit/bee4d65479bafbabb7b3266a6b864d3e71ace37d