Custom extra files for rootfs

[bboehmke]

This changes adds support to add extra files to the rootfs. The definition is done similar to the flags, buildflags and env.

All files in the directory

./extrafiles/github.com/test/package/

will be added to the rootfs in

/

Currently it is not supported to overwrite files!

Here an example output for some extra files in extrafiles/github.com/gokrazy/breakglass:

2021/06/22 22:16:05 packer.go:670: package github.com/gokrazy/breakglass:
2021/06/22 22:16:05 packer.go:672:   will be started with command-line flags
2021/06/22 22:16:05 packer.go:674:     from flags/github.com/gokrazy/breakglass/flags.txt
2021/06/22 22:16:05 packer.go:676:     last modified: 2021-06-20T11:29:39+02:00 (58h46m26s ago)
2021/06/22 22:16:05 packer.go:672:   will include extra files in the root file system
2021/06/22 22:16:05 packer.go:674:     from extrafiles/github.com/gokrazy/breakglass
2021/06/22 22:16:05 packer.go:676:     last modified: 2021-06-22T16:55:17+02:00 (5h20m48s ago)
2021/06/22 22:16:05 packer.go:680: 
2021/06/22 22:16:06 packer.go:1201: extra files of package github.com/gokrazy/breakglass collides with rootfs: [etc/http-port.txt]

[stapelberg]

What are you using this feature for? Wouldn’t config files be more convenient to keep on /perm so that you can modify them without building a new image?

[bboehmke]

I want to use this to build a image that includes a complete configured application that I can deploy and update with single call of gokr-packer. The device is basically a dumb IoT device (MQTT & some GPIO stuff) which does not need much updates for the config.

By keeping the config in the rootfs I can simple update it by deploying a new image.

An other way would be an automatically initialization of /perm and an custom update mechanism which seems to be much more work for this use case.

Maybe I am on the wrong path but everything on a read-only file system that can simply be swapped sound good to me for this.

[stapelberg]

I want to use this to build a image that includes a complete configured application that I can deploy and update with single call of gokr-packer. The device is basically a dumb IoT device (MQTT & some GPIO stuff) which does not need much updates for the config.

Gotcha. What’s the application you’re building? Is it under your control or third-party? If it’s built by you, you could also just adjust the default config to do what you need, no?

[bboehmke]

The main tool would be a simple self written go app (has to be converted from a python script) that does a bridge between the raspberry GPIO and MQTT. Because I am not sure if I will only have this single device in the future, I would already start to make it generic and configured via a config file (which GPIO maps to which MQTT topic, defining MQTT connection, ...).

So the idea is to have a simple set of configurations per device and simply deploy and update them via gokr-packer. Maybe comparable as an extreme reduced version of https://esphome.io/ with the different that it is configured by a file instead of generated code.

[stapelberg]

Ah, so this is for host-specific configuration for simple-enough devices that they don’t need a /perm partition. Thanks for the explanation.

I wonder if we could make this one step more generic and allow bundling files not just for /etc, but for any path in the resulting root file system? That way, you could e.g. also install extra files under /usr/share/myapp.

Also, what happens when the extra files collide with the gokrazy system files, or each other? Both cases should probably result in an error message to avoid surprises down the line.

[bboehmke]

Currently the config is simply placed in a subdirectory in etc based on the package name. So there should not be so much collision.

I also like the idea to make it more generic. In this case I see 2 simple way to handle collisions: 1. Simply overwrite everything or 2. Stop on a collision. Maybe it is also an option to add both and add a flag to enable overwrite.

What do you think?

[stapelberg]

I think stopping on collision is the better choice for now.

We can always add an overwrite flag later when we know more about the desired semantics.

[bboehmke]

Ok I will rework the change and update the PR

[bboehmke]

I have now added a new flag custom_root_files that can be used to define a directory that contains the additional files and directories for the rootfs. If a file exists an error should be shown.

[stapelberg]

Sorry, I should have been more clear: I think it would be good to keep the per-package option pattern we use for flag and environment values.

I.e., in this case, we’d have an additional directory, say extrafiles, with a Go package underneath, and then the actual extra files. E.g.:

extrafiles/github.com/gokrazy/breakglass/usr/share/mime/magic

That way, the mechanism works similarly to the existing one (→ easier mental model), we can display more helpful error messages (you’re adding file X for package A, but package B already ships with it), and correctly account disk space usage in the future.

Thanks,

[bboehmke]

I thought about that but because this affects more or less the complete rootfs it is not even package related. For example this could be used to copy complete applications into the rootfs that are not related to a package (even if this is maybe not the idea behind gokrazy).

But I can also add it like a per package option but it looks more complicated to me for this use case.

[stapelberg]

Yes, I understand that it affects the complete root file system.

As you suspect, gokrazy is built around the notion of Go packages. Every resource should be tied to a Go package, including files that end up in the root file system.

It can seem more complicated for specific use-cases, but I think overall adhering to the mental model of “everything is a Go package” is the better choice :)

[bboehmke]

I can rework it but for the simplest implementation I think we get some limitations:

1. because multiple files are possible there is no useful lastModified for to the packageConfigFile
2. to not iterate multiple times through the list I would add them directly to the rootfs after the other this causes:
   • that the function is called after the package option print out
   • that it would not be possible to detect where the conflict file comes from (only thing known is base system or some package before)

For 1. it could be possible to rework the packageConfigFiles to accept multiple files per type.

For 2. it would also be possible to first build one rootfs per package and check/merge them later. This would require a more complex implementation.

What do you think is the right approach?

[stapelberg]

• because multiple files are possible there is no useful lastModified for to the packageConfigFile

We can just use the most recent lastModified of all files that were considered.

it would also be possible to first build one rootfs per package and check/merge them later. This would require a more complex implementation.

I’m not 100% sure I understand the problem, but first discovering all packages and their files (so that we can produce good error messages) sounds like the better option to me.

[bboehmke]

I reworked it again and also updated the description and title of the PR

[bboehmke]

rebase is also done