Extract common code, and cleanup various directory paths.

[cmacq2]

This change introduces the new brickstrap-argv.sh module.

Changes:

• Change the -d option to be 'destdir' as opposed to 'rootfs' directory. All output of brickstrap should now be generated as files or subdirectorys located within the given 'destdir'.
• Extract common option parsing code into utility functions
• Re-organise brickstrap artefacts inside rootfs/chroot. Artefacts like tar-only directory or tar-exclude file are now confined to a 'brickstrap' directory hierarchy. This way there is a lot less arbitrary poking inside the rootfs going on.
• Introduce 'images' directory for storing rootfs image files. This paves the way for generating multiple images with arbitrary partition layouts.
• Introduce -I option to specify image base name. By default the basename of 'destdir' is used with an appended timestamp.
• Introduce getter function to lookup various paths, and base paths.
• Remove numerous globally namespaced variables, notably ROOTDIR. (There are 'getter' functions which provide same information more safely now).
• Introduce -P option for blacklisting individual packages. This is a more convenient alternative to BLACKLIST_PACKAGES
• Introduce -B option for blacklisting entire package files. This is a more convenient alternative to BLACKLIST_PACKAGE_FILES
• Update brickstrap-utils.sh module to take advantage of the changes.
• Update brickstrap.sh to take advantage the changes.
• Fix image generation logic to not require a temporary file + mv step, to avoid name clashes.

This change resolves issue #26 (through destdir): https://github.com/ev3dev/brickstrap/issues/26

See also: https://github.com/ev3dev/brickstrap/pull/23#discussion_r49371653

[cmacq2]

Note that this change removes a number of variables from the global namespace, notably ROOTDIR. Corresponding updates to the example projects are not part of this PR and must be incorporated separately:

• ROOTDIR -> $(br_rootfs_dir)
• IMAGE -> $BR_IMAGE_BASE_NAME (probably what you want)
• Any others? See the 'getter' functions down in brickstrap-argv.sh.

Likely ROOTDIR is the big one, use:

grep -R ROOTDIR <project>

To find out which files need updating.

[cmacq2]

Any comments?

[dlech]

Sorry, I've been deep into other things. I'll try to have a look soon.

[dlech]

Introduce 'images' directory for storing rootfs image files. This paves the way for generating multiple images with arbitrary partition layouts.

What is the use case for multiple images from a single rootfs? It seems to me that if you want a different image, you would have to build a new rootfs. I would just as soon have the (single) image not in a directory by itself.

[dlech]

Introduce -P option for blacklisting individual packages. This is a more convenient alternative to BLACKLIST_PACKAGES

If we are going to get into this sort of fine-grained options, I would like to have long option names. Of course, that is out of the scope of this pull request.

Also, I would like to choose more obscure letters for the short form of blacklisting so that we can save the more common letters (P and B) for future options where they might make more sense.

[dlech]

The rest of this looks good (other than my couple comments).

[cmacq2]

What is the use case for multiple images from a single rootfs? It seems to me that if you want a different image, you would have to build a new rootfs. I would just as soon have the (single) image not in a directory by itself.

That depends. If you have a single project which supports multiple devices you may want to generate multiple types of rootfs in a single build because (for example):

• The Raspberry partitioning layout is quite specific to work around quirks of the R.Pi bootloader.
• Devices like the BeagleBone have a dedicated flash. You may want to use flash friendly filesystems like UBIFS or JFFS2 for those devices that support it, but have SD images for those that don't.
• You might want to generate both a device rootfs and a VM (e.g.: kvm+qemu) image.

Also having an images/ directory is slightly more robust in the face of arbitrary -I values (edge case: -I rootfs, (custom) driver that simply copies the tar file, use case: deployment to NFS later on -- NFS being fairly useful during development to avoid the tedium/time-wasting of reflashing on small changes).

[cmacq2]

If we are going to get into this sort of fine-grained options, I would like to have long option names. Of course, that is out of the scope of this pull request.

Also, I would like to choose more obscure letters for the short form of blacklisting so that we can save the more common letters (P and B) for future options where they might make more sense.

Any particular letters in mind?

I'm not sure that I like this option. For example, a package could still be installed even if it is listed here because it is a dependency of another package.

Likewise, rather than blacklisting full package lists, it seems like we should just advocate making components more granular.

Also, if you blacklist a package and there is a hook that depends on it, you will have to blacklist the hook as well. I just see these options causing more problems than they fix and I am not looking forward to providing technical support for them.

The issue that is addressed by this blacklisting functionality stems from the fact that there's a distinction between the end-user of a project configuration and its developer. The blacklisting mechanism is a tool for the end-user to selectively override some of the authors decisions: in particular if a user wishes to trim down an image somewhat. It's not a developer tool. It's also not a way for users to 'reshape' the configuration beyond trimming some cosmetic stuff.

Essentially it is a 80/20 solution to the problems of configuration authors having a somewhat one sided focus on the needs of their own pet project, historical baggage and/or cutting corners in getting a board up and running quickly. The alternative is for users to fork the configuration or develop their own configuration but that is a rather steeper learning curve altogether.

For example your ev3dev projects pull in a lot of runtimes but the user might decide they don't really need/want a nodejs and don't want their time and bandwidth wasted on downloading it.

As for hooks: we could of course add a mechanism to blacklist hooks (easy!) but that is where your argument actually rings very true:

it seems like we should just advocate

I.e.: hooks should be grouped inside a component together with their corresponding packages. That way the user simply picks their component selection appropriately and the problem is avoided entirely as well.

---

Tangent:

We should advocate also that developers not use hooks at all whenever feasible because they amount to interference with the package management system - i.e. the proper thing to do with hooks is to package your integration bits and then specify it among the packages to install. Then you don't need hooks and you automatically have this compatibility sorted for you by the package manager. For example hooks which touch policy-rc.d shouldn't really be run at all when your init system is systemd (and vice versa: hooks that touch systemd configs shouldn't run for a sysv init system).

---

Again the emphasis should be on the fact that it is an 80/20 solution to trim down some subjective bloat, but not a general mechanism for users to override the configuration developer's choices. The documentation should reflect and emphasise that.

[dlech]

Any particular letters in mind?

How about -x to exclude a single package and -X to exclude a package list file?

[dlech]

This is all much fancier than I really have a need for but your reasoning is sound. I hope that you will do some advertising to encourage people to use brickstrap to benefit from all these features that you are adding. :wink:

I'm going to have to take this for a test drive, but I think you've talked me out of most of my descents here.

[cmacq2]

How about -x to exclude a single package and -X to exclude a package list file?

Sounds good, I'll update the PR accordingly.

[cmacq2]

I adjusted the usage/help output to incorporate your feedback, and renamed the blacklisting short options as per your suggestion.

[dlech]

I made the required variable name changes and consolidated the *-ev3dev-jessie projects into a single project. I'm very happy (and a little surprised) that I was able to do it without a single problem. It worked as expected on the first try. So great job and keep up the good work!