Support repository "overlays"

scymtym commented 5 years ago

The generator should support composing multiple recipe repositories in the typical "overlay" way.

Syntax

A recipe repository directory may contain in its root directory a parents file. The contents of this file is a YAML document, whose root node is a sequence of length at least one. Each element of the sequence is a scalar node that is one of

A path designating a directory
A URL designating a GIT repository
A URL and reference name (branch name, tag name, commit hash) designating a commit in a GIT repository. The reference name is specified as the fragment part of the URL(?)

Example:

- /opt/rdtk/recipes-core
- https://github.com/rdtk/recipes-unibi
- https://github.com/me/my-recipes#experiments
                                  #^^^^^^^^^^^ branch

Semantics

Repository traversal and ordering

When accessing a recipe repository for the first time,

The generator looks for a parents file and reads it, if it exists.
For each entry in the file
- If the entry designates a local file, the generator accesses the repository, processing it recursively in the same fashion (i.e. starting at 1., looking for another parents file)
- If the entry designates a remote repository, the generator clones that repository using its ordinary cache mechanism and checks out the specified reference (or defaults to the master branch). After that, the generator accesses the repository, processing it recursively in the same fashion (i.e. starting at 1., looking for another parents file)
All traversed recipe repositories are linearized according to the following rules:
- For entries in a given parents file, an entry and its ancestors (that is, transitively referenced recipe repositories) go before the entries textually following it in the file and all their ancestors
- Children go before their ancestors
- For repeated entries, all but the one at the smallest index are discarded

Example:

The constellation

repository-1

repository-2
  parents:
    - repository-1

repository-3

repository-4
  parents:
    - repository-3
    - repository-1

repository-5
  parents:
    - repository-2
    - repository-4

when processing starts at repository-5, would result in the linearized order repository-5 repository-2, repository-1, repository-4, repository-3.

Recipe file lookup

When looking for a recipe file, that is a given relative filename

The generator simply checks in each repository in the order given by the linearized repository list whether the given relative filename exists in that repository
Once the relative filename exists in a repository, the lookup terminates and that file is used

LeroyR commented 5 years ago

repository-1
  project B

repository-2
  project A
  parents:
    - repository-1

repository-3

repository-4
 project A
 project B
  parents:
    - repository-3
    - repository-1

repository-5
  parents:
    - repository-2
    - repository-4
when processing starts at repository-5, would result in the linearized order repository-5 repository-2, repository-1, repository-4, repository-3.

The processing should go Breadth-first. This would ensure using A2 and B4 while still being able to use A2 B1 by simply adding repository-1 to the parent list.

scymtym commented 5 years ago

The processing should go Breadth-first. This would ensure using A2 and B4 while still being able to use A2 B1 by simply adding repository-1 to the parent list.

I think that would be problematic in terms of modularity. Consider

jan-recipes:
  distributions:
    test:
      versions:
        - A
        - B
  parents:
    - cse-recipes
    - leroy-recipes
    - base-recipes

cse-recipes:
  projects:
    - A
  parents:
    - citec-recipes

citec-recipes:
  projects:
    - B

leroy-recipes:
  projects:
    - A
    - B

base-recipes:

For the test distribution in jan-recipes, this would result in

Depth-first:
```
cse-recipes:A
citec-recipes:B
```
Breadth-first:
```
cse-recipes:A
leroy-recipes:B
```

The problem I see with the breadth-first linearization is that neither of the "consistent" results, i.e. cse-recipes:A, citec-recipes:B and leroy-recipes:A, leroy-recipes:B, is produced. By "consistent" I mean that the respective authors of cse-recipes and leroy-recipes each defined a particular combination of A and B recipes that works in their respective "scope".

while still being able to use A2 B1 by simply adding repository-1 to the parent list.

That is a good point in favor of breadth-first processing, but all examples I can come up with involve the kind of modularity violation described above.

Maybe depth-first with an option for non-recursive processing would be good? That would allow forcing a certain linearization if needed, but default to "safe" semantics.

LeroyR commented 5 years ago

With a "big" base-recipes depth-first processing could really hinder the ability to include overwrites as the base is used over the other user-defined parents. Breadth-first lets the user reach the "consistent" results explicit by using:

    - cse-recipes
    - citec-recipes
    - leroy-recipes
    - base-recipes

or

   - leroy-recipes
   - cse-recipes
   - base-recipes

As parents as per the example.

Maybe depth-first with an option for non-recursive processing would be good? That would allow forcing a certain linearization if needed, but default to "safe" semantics.

How about "no default" e.g. first line in parent file has to be either option

LeroyR commented 5 years ago

Ping @rhaschke @semeyerz

scymtym commented 5 years ago

Since we only need a single parent to realize the rdtk/recipes-core + rdtk/recipes-unibi structure, we could restrict parent count and depth both to one and avoid the issue for now.

RDTK / generator