Add more flexibility to the configuration overlays.

[ghost]

"-A arm" is fixed, but shoud be: switch (edi_bootstrap_architecture) { case "armhf": "arm" case "arm64": "arm64" default: fail }

[lueschem]

Good point! Just curious - did this cause any issue? Or does mkimage just use this parameter to "document" the stuff? Instead of writing two tasks we could also do a dictionary lookup (the dictionary would map the Debian architecture to the U-Boot architecture). This would make it easy to extend it to new architectures. Something like:

debian_uboot_arch_map:
    armel: arm
    armhf: arm
    arm64: arm64

[ghost]

Did this cause any issue: No, just noticed that iMX6 and iMX8 would have these values different and hence needed to be moved into configuration. In edi-pi there is some hacks which translates it; it might as well be moved into the configuration files in edi-pi as well. After iMX6 support in edi-imx, I might looking at Raspberry Pi Zero in edi-pi. I also noted that it runs armel (same "-A arm" as for armhf).

The less tricks we have in plugins, the easier porting and merging upstream changes into each instance; hence the dream to clean out hacks from ./plugins/*

But I think I'm missing a layer of indirection; it would be awesome to have one file in between ./configuration/base/common.yml and ./configuration/overlay/....yml with things that is common for a device; say all pi3 files. I.e. each of the pi3-...-{dev,build,test,cross-compile}.global.yml etc that share the same prefix.

[ghost]

I agree, a dictionary / map lookup might be good and make it automagically work without having to do even more config; I like that Debian ARCH autoconfig.

[lueschem]

There are also other places where we need to map the Debian architecture: Example 1, qemu:

def _get_qemu_binary_name(self):
        arch_dict = {'amd64': 'x86_64',
                     'arm64': 'aarch64',
                     'armel': 'arm',
                     'armhf': 'arm',
                     'i386': 'i386',
                     'mips': 'mips',
                     'mipsel': 'mipsel',
                     'powerpc': 'ppc',
                     'ppc64el': 'ppc64le',
                     's390x': 's390x'}
        debian_arch = self.config.get_bootstrap_architecture()
        qemu_arch = arch_dict.get(debian_arch)
        if not qemu_arch:
            raise FatalError('Unable to derive QEMU architecture form Debian architecture ({}).'.format(debian_arch))

        return 'qemu-{}-static'.format(qemu_arch)

Example 2, u-boot/edi-boot-shim (incomplete - more cases will be needed):

case "${DEBIAN_ARCHITECTURE}" in
    "armhf" | "armel")
        U_BOOT_ARCHITECTURE="arm"
    ;;
    *)
        U_BOOT_ARCHITECTURE="${DEBIAN_ARCHITECTURE}"
    ;;
esac

IMHO this mapping should be bullet proof and just automagically work for all the mapped entries.

[lueschem]

I agree that an additional layer might make sense. Currently we have:

base -> global -> HOST -> USER

What about extending it:

base -> somegoodname -> global -> HOST -> USER

Candidates for somegoodname:

• target
• device
• local

If the "somegoodname" configuration shall be applied to multiple configurations then we can use symlinks to apply it to selected configurations:

Example:

a.yml --> configuration/base/common.yml
b.yml --> configuration/base/common.yml
c.yml --> configuration/base/common.yml
configuration/base/common.yml
configuration/overlay/a.somegoodname.yml
configuration/overlay/b.somegoodname.yml --> a.somegoodname.yml

[ghost]

Agreed. I like the simple look-up and I also think it is robust.

PS: Maybe it could later be moved to edi: plugin/debian/uboot-something-something when we have factored all the parameters out.

[lueschem]

At the moment I prefer "local". Then the user can decide what he wants to put into the overlay.

[ghost]

The strategic question is; is it "customize here with whatever" or is it target towards a specific task. That will both guide naming, the documentation, and what goes into "it".

Depending on the decision, other names might also make sense: family, group, class.

[lueschem]

Then something like this might make sense:

base -> global -> group -> HOST -> USER

(Please note that I switched the order - the group overlay will overrule the global overlay.)

[lueschem]

In fact the HOST and USER overlays are rarely used AFAIK.

[ghost]

I haven't used HOST/USER either, but I do think it's a cool feature. Especially the USER overlay when multiple people, say a classroom, wants to keep track of many RPI's. I saw your school project thingy, I might use it with a group of Scouts / Pfadfinderen, and in those case the USER is very neat.

[ghost]

Currently, my most pressing annoyance is about consistency between all RPI3, or all imx6, or all iot-gate-... and so on. That is my reason to mention it in the first place. Like all RPI3 is armhf, all imx6 is armhf, all imx8 is arm64 etc. In the default EDI project I don't care too much, because the example overlay are superbly clean and short and in fact I think the base/to-be-called-common-maybe.yml already serves this purpose.

The issue happens, as I see it, because edi-pi and edi-imx is a collection of merged edi instances.

[lueschem]

Over time we had the consistency issue also with different releases (e.g. additional packages that got installed for a certain release). Therefore the question is whether one additional level is sufficient. Another possibility would be:

base -> global -> global.0 -> global.1 -> global.2 -> ... -> HOST -> USER

Maybe this would even be more future proof.

[ghost]

I'm not certain whether we will actually fix any complexity issue, or whether we are just increasing combinatorial complexity exponentially.

Maybe we can not handle it by linear addition, maybe we actually need a split:

a.yml ---> configuration/base/common.yml
./configuration/pi3.yml //pi3 specifics not put in a.yml, nor its overlay.
./custom_app/release2.2.yml
./custom_app/release3.1.yml // Major version might have a very different playbook etc.
edi image create ./a.yml --recipe ./configuration/pi3.yml --recipe ./custom_app/release2.2.yml
edi image create ./a.yml --recipe ./configuration/pi4.yml --recipe ./custom_app/release3.1.yml

And then the algorithm could be as before and then adding on top, each --recipe in order. Special case, without any --recipe parameter, we actually only have status quo.

[ghost]

or maybe like this:

edi image create pi3.yml --recipe ./profile/run.yml --recipe ./custom_app/release2.2.yml
edi image create pi4.yml --recipe ./profile/developer.yml --recipe ./custom_app/release3.1.yml

[lueschem]

In the beginning I did the design decision that the configuration yml together with the overlays should define the entire setup (reproducibility is often key for embedded systems). The configuration name is also important for the whole naming of (intermediate) artifacts. If we would allow command line options then this would require a rework of the entire logic. So far, the setup worked well for even big projects but sometimes an additional level would reduce the configuration duplication. Given the current experience I would rather stick to the overlay approach we have. If we allow more overlays we can use them like a matrix:

config/overlay  global     global.0  global.1
---------------------------------------------
pi3             pi3          pi        -
pi4             pi4          pi        -
pi4-dev         pi4          pi        dev
iot-gate        iot-gate     imx       -
iot-gate-dev    iot-gate     imx       dev

[ghost]

How would you estimate global.0?

[lueschem]

The global.0, global.1, ... implementation should be straight forward. If you also believe that this could be the way to go then I can take a look at it.