Add support for a search path mechanism operating on a directory hierarchy.

[cmacq2]

Add support for a search path mechanism operating on a directory hierarchy.

Variables are introduced to parameterise relative paths based on arch, board, distro and variant values. (See: br_find_path, br_locate_path, br_locate_directory)

Validation of these variables is added (see: br_validate_search_path_vars). This validation of such variables may be 'circumvented' by setting a corresponding variable named BR_IGNOREINVALID* (with * being the name of variable to trust blindly).

An override mechanism is introduced based on conceptual layers corresponding to the variables. (See br_find_path, br_consume_path_list_iterate_directories.)

Additional convenience functions are introduced for producing lists of all paths that match a relative base path. (See br_list_paths, br_list_directories)

These lists may be iterated over using br_consume_path_list and br_consume_path_list_iterate_directories. The latter may be used to 'merge' the configuration files in directories.

A few brprint* functions are added, which could be exposed as debug tools for users of brickstrap (and were used to lightly test the brickstrap-paths.sh code).

N.B.: the code hasn't been extensively tested to see how well it copes with 'special' characters (such as: \, $, @, ', ") in various paths.

[dlech]

This looks good to me. Since it is so many functions, I am thinking it might be prudent to setup some sort of unit testing to make sure they work as advertised.

[dlech]

FYI, we have a bash testing package that might be useful here. https://github.com/ev3dev/bats

[dlech]

Do you have a sample directory structure so that I can actually try this out?

[cmacq2]

I've 'tightened' the namespacing of various variables and functions because most of them should be strictly 'private' to brickstrap itself -- also with a view towards documenting it properly. I've added some preliminary documentation to the path mechanism (currently just introduction paragraphs) but eventually this should grow to cover the salient points of discussion #17

---

I've not yet looked into unit testing or prepared dummy directory trees. You can set these up yourself also, of course, and just let me know from which remote + branch to pull them into my branch so we can fold your work into the PR itself ... As it's a lot of work and many hands make light work... ;-)

[dlech]

I have read over docs/path.md and to be honest, it is not clear at all how I am supposed to setup my directories and it seems way too complicated. By complicated, I mostly mean way too many options and variables. I'm going to go back to #17 and propose a flatter way of doing this rather than nested directories.

[cmacq2]

By complicated, I mostly mean way too many options and variables.

Well the documentation isn't quite finished, yet -- right now there's a quick overview/summary and a introduction. That's not really documentation on how to use feature and what each variable is supposed to mean.

However to address you real question:

how I am supposed to setup my directories

That's down to the question: does the "previous" version of brickstrap suit you fine? If so, great, no changes to your directory layout are required. If all you want to do is generate a single image for a single specific board according to a single specific configuration then the variables don't matter and you can ignore them and continue as before.

It's only when you wish to support multiple build 'targets' off the same configuration that you want to start introducing these variables to your directory layout. Then you pick the variable which most precisely describes your partitioning like, this:

• Do you want to do different things for different architectures? -> $BR_ARCH
• Do you want to do different things for different boards? -> $BR_BOARD
• Do you want to do different things for different distros? -> $BR_DISTRO
• Do you want to build different generic 'variants' of otherwise the same config? -> $BR_VARIANT

Please note carefully: the operative question is "do you want to do different things for different configs?", which is not all the same as "is there a difference between various builds?" There is no point in introducing artificial partitions that yield the same output anyway -- it is only when different output is needed that these variables become useful.

Once different output is required the mechanism offers two features that we can use inside brickstrap itself:

• Shadowing/overriding paths. This can be used for "more specific" description of the configuration to override the "more generic" version. That's also where chaining of variables come in, essentially each variable acts as a filter and the most specific case 'wins'. This is what br_locate_path and br_locate_directory offer: picking the most 'specific' version of a given file from the configuration tree that can apply.
• Merging subtrees, that is adding "more specific" details like additional package files to a "more generic" base configuration/setup. This is what the br_list_paths and br_list_directories do.

These two primitive features can be combined (as e.g. br_consume_path_list_iterate_directories does) to give you things like: "include all applicable package files for the given variables, but, make sure that for each package file only the most specific version is picked". The same with hooks (so you run all applicable hooks, but if there are multiple versions of the 'same' hook you run the most specifically applicable one.)

[dlech]

right now there's a quick overview/summary and a introduction

That's kind of my point. If 80 lines of text is a "quick" introduction and hasn't event got into the "how to" part yet, then this is way too complicated.

Also, I have realized that are a couple things about the directories that I don't like. You have to be aware of priorities, which makes more effort in setting things up. Also, the directories get many levels deep, which I would rather not have.

I don't have a need for shadowing. Just mixing/overlaying the hooks, packages and root directories is good enough. I think most of the other special files can be handled the same way where they can be mixed together.

[cmacq2]

That's kind of my point. If 80 lines of text is a "quick" introduction and hasn't event got into the "how to" part yet, then this is way too complicated.

Okay, please don't take this the wrong way but have you compared your description of the alternative with the summary section? You'll note that in 21 lines, ~12 of which are due to formatting (7 for a table and couple of blanks) the entirety of this scheme is formally laid out.

So I don't really buy your argument about the 80 lines.

EDIT: What my remark about the "how to" is about is for guiding the novice user of the feature, specifically:

• What variables would be used for which purpose, and what kind of config you'd want to associated with that. (Something you'll have to explain about your alternative as well: when to use -F vs. when to use -B?)
• How you can introspect your config tree for a given set of variables so you can see what the 'result' is that brickstrap uses.

I don't have a need for shadowing. Just mixing/overlaying the hooks,

What exactly is the difference between 'shadowing' and 'overlaying' here? Or are they the same and do we push complexity elsewhere by special casing things that therefore, special cases being so special, should be documented as such...?

[cmacq2]

Also, I have realized that are a couple things about the directories that I don't like. You have to be aware of priorities, which makes more effort in setting things up.

Not really. At least I don't see how. Can you give an example so I can either learn how I am wrong or point out where you are?

What I do see is that at some point you're going to need a way to resolve ambiguity. That's what a priority scheme happens to do rather easily and straightforwardly. :)

Also, the directories get many levels deep, which I would rather not have.

Well this can be addressed, up to a point. It's mostly a cosmetic thing so I'd rather not do that until I know we'd actually want to use it.

[dlech]

Okay, please don't take this the wrong way but have you compared your description of the alternative with the summary section?

Touché. :smiley:

What exactly is the difference between 'shadowing' and 'overlaying' here?

"shadowing" is when directories have files with the same name and the highest priority directory wins. "overlaying" is when directories do not have any files with the same name, so the contents of the directories are just mixed together.

Can you give an example so I can either learn how I am wrong or point out where you are?

I can't really think of a real-life example where I would use priorities off hand, so I'll just drop this argument. At the same time though, I think I would find myself namespacing file names for hooks and packages so that they would not accidentally replace another one.

[cmacq2]

Closed for now. Based on further discussion in #17 it looks like we'll go with a different scheme based around the notion of abstract components.