pkgin binary is not included when building via x86_hdimage.sh

[richi235]

This gave me a pretty hard time when compiling and trying to/experiment with minix. Since there were a few packages I definitely neeeded.

• It looks like pkgin source isn't part of this repo either
• Anyway looks like release.sh includes pkgin via:

# Packages we have to pre-install, and url to use
: ${PACKAGEURL="http://www.minix3.org/pkgsrc/packages/$version_pretty/`uname -m`/All/"}
: ${PREINSTALLED_PACKAGES:="pkg_install pkgin"}

and

for p in $PREINSTALLED_PACKAGES
do  echo " * Pre-installing: $p from $PACKAGEURL"
    $PKG_ADD -f -P $RELEASEDIR $PACKAGEURL/$p
done

I think it would be good to do this in in the x86_(hd|cd|ram|*)image.sh scripts too.

If nothing speaks against I could do a PR, perhaps as one common preinstalled_packages.sh script sourced by all these. Thoughts on this?

[sambuc]

Hello,

There is already a facility in place to pre-download packages:

$ ./releasetools/x86_hdimage.sh --help
./releasetools/x86_hdimage.sh: illegal option -- -
Usage: ./releasetools/x86_hdimage.sh [options]
  -X xsrc       Build with X11 located in "xsrc"
  -x            Add X11 sets to extraction list
  -b            Add ASR service binaries to the image
                (said binaries must be built beforehand)

Environment variables:
  CREATE_IMAGE_ONLY     If set to 1, skip invocation of build.sh (default: 0)
  JOBS                  Number of CPUs to use for build.sh to use (default: 1)
  SETS                  Sets to extract for image (default: depends on script)
  BUILDVARS             Extra options passed to build.sh (default: none)

  PACKAGE_DIR           Path to packages to bundle (default: none)
  BUNDLE_PACKAGES       List of packages to bundle (default: none)
  PKG_INFO              Path to 'pkg_info' for bundling (default: pkg_info)

  BUNDLE_SETS           If set to 1, bundle sets for setup (default: only for CD)

if the following variables are set properly, packages will be available on the image, ready to be installed with pkg_add:

  PACKAGE_DIR           Path to packages to bundle (default: none)
  BUNDLE_PACKAGES       List of packages to bundle (default: none)

This requires you to first download the packages you want somewhere in your local system.

The reason packages are not pre-installed on bootable (hd|sd) images is it requires pkg_add to be built as a tool.

Kind regards

[boricj]

Pre-installing won't work when cross-compiling. It's been quite some time, but I recall pkg_add not being happy about installing MINIX packages on a Linux host.

While pre-installing should be possible when doing a straight compilation under MINIX, the current image framework generation always cross-compile. Perhaps that should be improved to also support straight compilation under MINIX to allow pre-installation.

This framework hasn't reached feature-parity yet with release.sh. I should probably do something about that before the next release is due...

[boricj]

I tried to make make release work, but the tarballs can't be created because of the target do-hdboot. Boot modules are expected by the sets lists to be in /boot/minix/.temp but that target relocates them in /boot/minix/3.4.0r4. This target doesn't seems to be invoked when cross-compiling, but I'm not sure if my conclusion is correct since I don't know much about the NetBSD build system.

I guess the best course of action would be to remove the .temp hack and put these files directly where they should be in the first place, but I don't know how these files ended up in /boot/minix/.temp in the first place...

[antoineL]

On 07/08/2016 12:00Z, Jean-Baptiste Boric wrote:

I tried to make make release work, but the tarballs can't be created because of the target do-hdboot. Boot modules are expected by the sets lists to be in /boot/minix/.temp but that target relocates them in /boot/minix/3.4.0r4.

Only because the source tree is presently for version [will be] 3.4.0 and that execution of mkboot/do-hdboot has built revision 4.

Also, that script updated the symlink /boot/minix_latest, which is the real one used by MINIX booting process.

The revision is mostly a feature used to be able to identify the N previous versions which are kept, particularly in case you cannot boot with your new revision 4, then you manually boot revision 3 and will get back the present system, where the system should boot normally, at least until it tries to mount the on-disk root system.

I do not believe this is pertinent when building a new system. Neither is /boot/minix/.temp, by the way.

This target doesn't seems to be invoked when cross-compiling,

Agreed; my understanding is that the mkboot script (which is invoked by *hdboot targets) has never been adapted to cross-compiling situations.

There are traces of attempts at it in form of a knob named MKINSTALLBOOT, which when =no, prevents the operation of the script and also the building of updating of boot.cfg; earlier, it also caused the renaming of /boot/minix/.temp into /multiboot, but this was dropped in 2014 (84bb300); unfortunately, IMHO.

The actual value of MIKINSTALLBOOT is a bit capricious: it is (almost) forced to =no by build.sh (which I understand was thought to be the correct use for any cross-compiling cases), defaults to =yes in bsd.own.mk for i386 while being defaulted to =no in the Makefile itself (which is OK for ARM); and is forced to =yes when release.sh script is invoked, while also forcing the invocation of the do-hdboot target, to cause the correct setup for booting that particular way to build the product.

I guess the best course of action would be to remove the .temp hack and put these files directly where they should be in the first place, but I don't know how these files ended up in /boot/minix/.temp in the first place...

Since you asked for history...

With the previous boot manager (1993-2012), the MINIX system was a single image file, which was built with a program named installboot off the various pieces (kernel and the boot-time modules, and sometimes also the ramdisk image as a separate piece, before being merged into memory); that program shared about no relationship with NetBSD's installboot(8); by the way, the name of this now-disappeared program is why the NetBSD installboot(8) has a special name (currently installboot_nbsd), and it also the reason for the name MKINSTALLBOOT.

Those images were given name like 1.7.0r1, 2.0.2r150, or 3.2.0r653 (to cope with former 14-character name restriction) and were stored inside the /minix/ directory (before 1995, there was only one /minix file and no selection possibility); that directory was renamed into /boot/image/ for MINIX 3 (2006).

Then as GSoC 2010, MINIX gained multiboot support; to deal more or less with that moving name scheme, a symlink was added as part of the hdboot target, which made /boot/image/image_latest being a fixed name pointing always at the ready to use system.

When the system was built also as ELF (2011), new programs (mkimage) were introduced to adapt the boot monitor, using previous multiboot work; at the beginning of it, for ELF there were no rotation or similar, only one system was available, the just compiled one; when it happened to be broken and you did have not saved the previous working one, you lose.

As part of his work to port the NetBSD boot monitor, Evgeniy restored a feature similar to the previous scheme (commit 8979450, 2012), using subdirectory rather than single file, and shell instructions instead of C code. This is when he used the .temp hack, as a intermediary directory name to store the new system before invoking mkboot on it to give it the new correct name. At the same time, it added the MINIX-specific load_mods command in NetBSD boot monitor.

The bad point here, retrospectively, is that while the for+cp dance done to copy the ${PROGRAMS} into the new directory is pretty standard shell code and can be used also in cross-compiling environment, what does the mkboot script is full of MINIX arcane invocations. And after 2012, the Makefile was modified several times to prevent mkboot to be run in cases it will break (probably in order to fix the broken build.) This lefts the .temp hack to show its ugly head for much more time than it ought to: for example, it is listed in the sets list, or in the image.functions.

I do not know what is the correct thing to do at this point; it seems to me that nowadays, everything is done to simplify the cross-compilation case, since it is the most natural for the embedded world; also the NetBSD build.sh system tries hard to make cross-compilation the standard; unfortunately pkgsrc is not that advanced, which this issue is at least partly about.

Anyway, if it should be decided whether or how to modify the naming scheme for images (perhaps removing the whole system to have several ready-to-boot systems at the same time) and _solvingthe.temphack, I believe this warrants its own issue.