Feature Proposal: Docker "packages"

[doanac]

This is closely related to PR #1046 by @cajun-rat. I've been playing around some in code trying to get a feel for things and think I may have a more holistic approach for this. From a high-level UX it could simply be a user in app.atsgarage.com doing an "add package" with a targetFormat=DOCKER_APP. And wow - tufreposerver can store files ... I had no idea. I think for my production usage, I'd probably use my credentials.zip to push things to TUF by hand that I've "released", but the GUI option seems cool.

The package could then be added to a device(s). Currently it will fail because SotaUptaneClient will say "Cannot install a non-OSTree package on an OSTree system". At this point I think we could implement a new DockerApp package manager. I'm not sure how we'd wire things up in code, ie do we create something like a CompositePackageManager that has pointers to both the OstreeManager and the DockerAppManager or do we conditionally compile in some Docker logic into the OstreeManager, or maybe SotaUptaneClient could change to construct the proper PackageManager implementation on-the-fly based on the targetFormat? Either way, you'd probably want a new value for config.pacman.type = kOstreeAndDocker.

I'm totally open to trying to make proof-of-concept if this isn't crazy. However, I'm worried about the answer to question 1 below.

Open Questions/Concerns

1. Would having 2 or really $N "packages" (you could choose multiple docker-apps) on target work with Uptane? PackageManagerInterface::getManifest might not be capable of expressing this? However, maybe its fine to just report the installed image?

2. Docker-App is still pretty experimental Seems like their DockerCon Preview is enough for us. We coud download the app bundle via the ota-reposerver and then do something like docker-app-linux render helloworld.dockerapp | docker-compose -p <package name> -f - up

[pattivacek]

I'm just a bit confused about the big picture here. You want to have both docker apps that you can update as well as an OSTree system? These will all live on the same device? Where will the docker apps live such that an OSTree update won't wipe them away?

You are correct that if you want to use docker as a package manager you will probably need to create a new class that inherits from PackageManagerInterface. That shouldn't be too hard. I don't understand how it should interface with OSTree, but in theory you could still just make calls into the OSTree package manager from there if you really wanted to. That might be tricky, but seems possible.

1. Would having 2 or really $N "packages" (you could choose multiple docker-apps) on target work with Uptane?

Why not just model this as multiple targets? Uptane as well as our implementation are not very flexible about this, and it seems like the going logic is just to treat these as multiple targets, even if they live on the same device in reality.

[doanac]

I'm just a bit confused about the big picture here.

The idea is that devices have really minimal base images. You can then choose their "personality" by choosing which containers you want to run. eg "this is a google assistant" or "this is an alexa" or "this is mqtt-broker". The containers are managed outside of ostree.

Why not just model this as multiple targets?

Could you elaborate a little more on this? I think you might be implying to basically run a couple of aktualizr instances. One for the ostree and one for docker?

Another idea that might be more clean would be having a new OstreeWithDocker manager. It would look just like the current ostree targets. However, the "custom" data could have an optional attribute:

            "custom" : {
               "targetFormat" : "OSTREE",
               "hardwareIds" : [
                  "raspberrypi0-wifi"
               ],
           "docker-apps": {
                 "alexa": "<not quite sure the value here, maybe reposerver target url that would be the docker app file to download>",
                 "google-assistant": "...",
                 "mqtt-broker": "..."
           },
               "version" : "osf-39",
               "name" : "raspberrypi0-wifi-lmp"
            }

Then a device could update its pacman configuration section to have an optional value: docker-apps = alexa, mqtt-broker

[pattivacek]

Could you elaborate a little more on this? I think you might be implying to basically run a couple of aktualizr instances. One for the ostree and one for docker?

Sorry, I think I didn't express myself very clearly. I hope there is no need for multiple aktualizr instances! I was trying to point out the main hurdle to implementing any sort of system in which one device has multiple packages that can each get independently updated: there is not specific support for that in Uptane or libaktualizr. The easiest way to work around it is to treat each package as a "secondary". But if the primary filesystem is OSTree, that seems like it might not play very well with any other package management system, since OSTree usually controls the entire filesystem except for /var. So would all of the docker packages live in /var, or how do you envision the two package managers interoperating?

[doanac]

So would all of the docker packages live in /var,

docker doesn't exactly have "packages", but it stores its persistent stuff under /var/lib/docker

or how do you envision the two package managers interoperating?

If we were to embed the "docker-apps" list in custom data as I described above, then I think we could create a new package manager, say "DockerAppManager", that extends OstreeManager. It would first pass-through all calls to OstreeManager to let the Ostree upgrade happen. And then do its docker-stuff. So "pull" would first pull Ostree and then pull down the docker images (i see there's some weirdness in the current fetcher logic we might need to tidy up). "install" would do the OstreeManager::Install and then start the docker containers.

[pattivacek]

it stores its persistent stuff under /var/lib/docker

In that case, this probably could work. There would be some weirdness about balancing the two package management systems, but that can be sorted out somehow.

[doanac]

thanks for the feedback @patrickvacek. I'll start on a minimal proof-of-concept skeleton so everyone can get a feel for what this will look like.

One area things already feel a little awkward in is in fetcher.cc. The OstreeManager::pull ifdef creates a little friction. For now, I'll just keep hacking. However, once PR #1122 is merged, it might worth figuring out something nicer there. My thought was adding two new methods to the PackageManagerInterface:

  virtual bool shouldFetch(const Uptane::Target &target) const {
    (void)target;
    return false;
  }
  virtual data::InstallationResult fetch(const KeyManager &keys, const Uptane::Target &target,
                                         const std::function<void()> &pause_cb = {}, ProgressCb progress_cb = nullptr) { throw std::runtime_error("Unimplemented");  }

And then sotauptaneclient could defer to the package manager for fetching when appropriate. This is probably worth its own issue to discuss though.

[pattivacek]

One area things already feel a little awkward in is in fetcher.cc. The OstreeManager::pull ifdef creates a little friction.

Yes, I've never liked that BUILD_OSTREE. There used to be more and we got hid all of them behind the package manager interface except for that one. I would gladly accept ideas of how to improve that!

[doanac]

I've created a non-functional branch that kind help visualize what this could look like. I thought that might help drive things forward:

https://github.com/advancedtelematic/aktualizr/compare/master...doanac:docker-pkgmgr?expand=1

With a branch like that, you could build aktualizr (or aktualizr-lite) with -DBUILD_OSTREE -DBUILD_DOCKERAPP and then create a sota.toml with a section like:

[pacman]
type = "ostree+docker-app"
docker_apps = "alexa, lwm2m-gateway"

[pattivacek]

@doanac I think that's a reasonable start! Thanks for putting that together.

I wonder, would it make sense to make the docker_apps config option more generic? This concept of multiple apps/targets running within one manager seems common and likely to come up again elsewhere, and it won't hurt to make it generic now. Just naming it "apps" might suffice. That would also let us get rid of some of those ifdefs.

I'm also curious if @lbonn or @eu-smirnov have any thoughts.

[doanac]

Just a quick update to track the work. Here's another test branch I have: https://github.com/advancedtelematic/aktualizr/compare/master...doanac:docker-appmgr-2?expand=1

There's still one big hack: https://github.com/advancedtelematic/aktualizr/commit/888c509ee5ddd2ee458e85efa6cc3894487f88c8

I think this can probably be resolved by moving some of the updateMetaImages logic from SotaUptaneClient into Uptane::ImagesRepository. And then you wouldn't need the SotaUptaneClient for iterating targets. More thought still needed there.