refactor: Support the move to having build files located in `./files/` in all modules

[gmpinder]

This would be the changes required for supporting https://github.com/blue-build/cli/issues/138.

[xynydev]

Modules requiring refactor

• files
• gschema-overrides
• script
• systemd

General refactor plan

With the setup in config/, we require that each module use a specific subdirectory of config/ for its files. In my view, with a files/ directory in the root of the repository, it should be possible for any module to read from any subdirectory of files/. Each module should check for the existence of /tmp/files/. If that directory exists, /tmp/config/ should be entirely ignored.

(Note: examples below use un-implemented files module syntax from #203)

Without such a refactor, a repo might look like this:

files/
   files/
      usr/
         ...
   gschema-overrides/
      ...
recipes/
   recipe.yml

With the following recipe:

# recipe.yml
...
modules:
  - type: files
    files:
      - source: usr/
        destination: /usr
  - type: gschema-overrides
    include: 
      - zz1-myoverride.gschema.override

With such a refactor, a repo might look like this:

files/
   system/
      usr/
         ...
   gschema/
      ...
recipes/
   recipe.yml

With the following recipe:

# recipe.yml
...
modules:
  - type: files
    files:
      - source: system/
        destination: /
  - type: gschema-overrides
    include: 
      - gschema/zz1-myoverride.gschema.override

Specific refactor details

systemd

The systemd module currently doesn't take in configuration for specifying the units to copy, rather it always copies them from config/systemd/system/ and config/systemd/user/. I am unsure how to refactor this module. It also does something easily doable with the files module.

[fiftydinar]

@xynydev I think that we should keep gschema-overrides name of the folder, to reflect the name of the module.

files module is the exception here with files/system/, but not a rule, because files/files/ can be confusing to the users.

Everything else looks good.

The systemd module currently doesn't take in configuration for specifying the units to copy, rather it always copies them from config/systemd/system/ and config/systemd/user/. I am unsure how to refactor this module. It also does something easily doable with the files module.

Well we can copy systemd units from files/systemd/system/ & files/systemd/user/ if files/systemd/ exists. If not, use legacy location to copy files from config/systemd/ location.

[xynydev]

@xynydev I think that we should keep gschema-overrides name of the folder, to reflect the name of the module.

files module is the exception here with files/system/, but not a rule, because files/files/ can be confusing to the users.

My point was that those folder names shouldn't be hardcoded IMO. So it could be files/gschema-overrides/and files/files/ if a user wants that.

[fiftydinar]

My point was that those folder names shouldn't be hardcoded IMO. So it could be files/gschema-overrides/and files/files/ if a user wants that.

In that case, I agree.

[gmpinder]

So this idea I had might be too much to implement in the scope of this issue. However, I'm working on the Stages feature (https://github.com/blue-build/cli/pull/173) at the moment and ran into a particular issue of needing to copy between stages. I created the copy module that is pretty much a 1-1 translation of a COPY instruction. It looks something like this:

type: copy
from: ghcr.io/blue-build/cli:latest-installer
src: /out/bluebuild
dest: /usr/bin/

It would end up creating the instruction:

COPY --from=ghcr.io/blue-build/cli:latest-installer /out/bluebuild /usr/bin/

So I essentially re-implemented the "built-in" files module. However, I chose to make a new module cause I though of a potential extension of the files module. How about making the files module a way to perform regular file operations like mv, rm, chmod, chown, etc. Obviously we can adjust the scope to what we feel is appropriate. Just wanted to drop this idea here while you were in the process of refactoring.

[xynydev]

IMV some of those file operations might be possible to commit into git, while most would be more straightforward to do with shell snippets. I wouldn't complicate the files module with them.

Your copy module also looks pretty good, but it has one crucial difference; the bash files module is one layer, but can perform multiple copy operations, while the copy module has to be repeated, and creates a layer for each copy operation. This usecase is something that the containerfile snippets already adequately fills, though, so creating a wrapper for it is really more of a convenience thing.

[gmpinder]

the bash files module is one layer, but can perform multiple copy operations, while the copy module has to be repeated, and creates a layer for each copy operation.

That's a good distinction. Alright fair enough

[prydom]

Being able to use a container image as a file source is something I had to resort to using the containerfile module for. You can see that Containerfile here. It would be convenient if the files module supported a from: argument which templated arbitrary --mount=type=bind,from= arguments.

I have a Github action which builds .srpms and .specs into RPMs and needed to get those RPMs into my overlay (that build container is here).

Thanks @gmpinder for mentioning that stage feature because it could potentially simplify things as long as caching works out. One of the spec files is Mesa and I don't want to be rebuilding that if I don't have to.