Proposal: buildctl daemon command

[tonistiigi]

BuildKit uses daemon model for many benefits like parallel job processing, easier extendability, precise privileged execution control, remote control etc. The downside of it is that it makes the setup harder for new users and people who want to run it occasionally.

The proposal is to hide the presence of daemon unless some more advanced use cases are needed.

Automatic bootstrapping

If the user runs a buildctl command that requests API access and there is no response.

• If connecting address has been changed from the default, a standard dialing error is shown.

• If the connecting address is default buildctl tries to start the daemon automatically and shows a log message about that.

buildctl first finds if there is a buildd binary(or other dependencies for macOS) and gives up if it can't locate it.

Then it tries to check if it has enough privileges to start the daemon. In Linux, this means root with needed capabilities enabled. On macOS, the daemon would be launched in a vm without requiring extra privileges.

• If there are enough permissions daemon will start on background and user command executes. A log message will be shown about it. Otherwise, the user shouldn't even notice it except for a small delay. The daemon continues running after the user command has finished. If the daemon doesn't come up(or no workers can be started), then an error will be shown.

  • If there are NOT enough permissions, buildctl will show a helpful message explaining that daemon needs to be started and the command that user needs to run to do it.

The command would look something like:

sudo buildctl daemon start --root=/home/myuser/buildkitstate --allow-user=myuser --entitlements=security.confined

This command will:

• Check if it has enough permission to run daemon. Even with sudo, this check may fail if the caps are bound.

• Check that user calling sudo has write permission to root dir. This is to avoid buildkit taking disk space from partitions the user doesn't have access to.

• next, it uses the sudo privileges to set a helper binary buildkitd-launcher that comes with buildkit to suid and makes sure root owns it.

• It writes a configuration file that lists the allowed user values and entitlements they are allowed to use. The default entitlements that don't have a security list can be added automatically. Makes sure the file is owned by root and only modifiable by root.

• Executes buildkitd-launcher that starts the daemon. Only unix socket requests are allowed against this daemon. Every connection checks the dialing user is in the configuration file and make sure the build requests don't use any entitlements that have not been granted.

• Next time after daemon needs to be started; the user doesn't need to type the start command again as it can directly execute the suid binary now.

An unprivileged user using the buildkit daemon this way should be considered as secure as containers. There shouldn't be any way for the user to issue a command against buildkit daemon that has access to anything outside of the state directory they already have access to.

Extra commands

When the daemon is running user can use some helper command to control it.

buildctl daemon state - shows info about current config(maybe logs as well)

buildctl daemon stop - stop the server. restarted automatically by buildctl when needed. There could also be a --timeout for the daemon start command for automatically shutting down server on inactivity.

buildctl daemon reset - start a fresh daemon with empty state dir

[AkihiroSuda]

Rather than tackling sudo and hyperkit (and hyperv for windows), can we just use Docker for Linux/Mac/Windows?

[tonistiigi]

@AkihiroSuda We could show the Docker example as well(especially if we can detect Docker), but BuildKit shouldn't have a (recommended) dependency on Docker and it doesn't really solve the security problem because having access to Docker API is not secure while BuildKit API should be secure(unless configured with insecure entitlements).

[AkihiroSuda]

BuildKit shouldn't have a (recommended) dependency on Docker

Accurately, it will just have a dependency on github.com/moby/moby/client, not Docker. But we can recommend Docker (for Mac/Windows) for ease of installation.

it doesn't really solve the security problem

If buildctl can call sudo, isn't it already unsecure?

[tonistiigi]

If buildctl can call sudo, isn't it already unsecure?

buildctl can't randomly call sudo. User calls sudo one time to grant access to the user in a limited form. If this command doesn't explicitly allow the build jobs to use insecure entitlements then a regular user can keep using buildkit without having a way to do anything that they couldn't do as a regular user (contrary to Docker API, where if you have access you are effectively root on the host).

[kunalkushwaha]

If no one is working on this, I would like to work on this.

[tonistiigi]

@AkihiroSuda What was your take on this? While I'm all for the rootless when it is enough I think this still has use cases and when we can detect that rootless is supported without downsides we can use the rootless mode instead. I'd also like a better mac experience.

Lets try to break it into multiple smaller chunks with separate PRs. Maybe adding something that would support https://github.com/genuinetools/img/issues/92#issuecomment-393733812 would be a good start(or even that can be in multiple phases)

[AkihiroSuda]

Maybe we can revisit this when we have support for distributed daemons?

[tonistiigi]

@AkihiroSuda The distributed case would always manual setup for defining the cluster so I'm not sure how they are related. Unless you're already thinking about automatically spawning the cluster. Priority wise, I agree that I'd like to see the distributed mode more.

[AkihiroSuda]

@jessfraz any thoughts?

[jessfraz]

no opinions here :)

[kunalkushwaha]

The points suggested in https://github.com/genuinetools/img/issues/92#issuecomment-393733812 are required for if has to be run as single process(daemon-less) like img.

Whereas this proposal seems starting daemon process if its not running (depends on various condition), so cache keys issue may not be problem here.

• Do rootless build is planned to run as daemon-less mode or buildkit daemon will be always there?
• What will be other parts to problems that may require to solve this. capabilities/entitlement https://github.com/moby/buildkit/issues/238 ?

[tonistiigi]

@kunalkushwaha

This proposal would start the daemon automatically as well if there are enough permissions for that, multiple buildctl instances would share it and if it is automatic it should be shut down as well if unused. That part is similar to img requirements. In addition this defines how the user can grant the set of privileges one time with sudo if the automatic path doesn't work.

Do rootless build is planned to run as daemon-less mode or buildkit daemon will be always there?

I don't think there is such a thing as fully daemonless mode if we want multiple clients. The question is only if we expose to the user that there is a deamon process and do they need to do anything special for it. If we can detect that we can avoid it and the user does not want to customize their experience then there is no need to expose the daemon.

What will be other parts to problems that may require to solve this. capabilities/entitlement

Entitlements are not strictly required for this but this proposal defines how to handle entitlements for the unprivileged user. That being said, I think entitlements feature itself would be a higher priority than this in case you would like to work on that.

[kunalkushwaha]

@tonistiigi thanks for explaining in detail. I would like to work on both.

So more comments I will add on entitlement proposal.

[AkihiroSuda]

daemonless UX: #979