Easier support for debugging container pebble

[jameinel]

So we have support for "export PEBBLE=..." when you want to play with pebble as an application manager and interact with it. (you can set PEBBLE for both the client and the server and it will work).

However, if you want to juju ssh into a workload and debug against pebble, you actually have to use the undocumented PEBBLE_SOCKET environment variable. PEBBLE always appends .pebble.socket while in containers the actual path is, /charm/containers/$CONTAINERNAME/pebble.socket

Ideally it would be something like pebble -c container get-plan

and 'juju ssh' could set an environment variable pointing at /charm or somesort.

At a minimum, we need to officially document PEBBLE_SOCKET and how to interact with sidecars and their pebble.

[benhoyt]

@jameinel This is a good idea, and I do think a shortcut here would be helpful.

Note that PEBBLE_SOCKET is not exactly undocumented -- when you type pebble with no args or pebble --help it prints:

Pebble lets you control services and perform management actions on
the system that is running them.

Usage: pebble <command> [<options>...]
...
Set the PEBBLE environment variable to override the configuration directory 
(which defaults to /var/lib/pebble/default). Set PEBBLE_SOCKET to override 
the Unix socket used for the API (defaults to $PEBBLE/.pebble.socket).
...

However, it's not super-easy to discover when using Juju sidecar charms.

I don't particularly like pebble -c <container> ..., because it's adding a very Juju/K8s-specific option to pebble itself, which has so far stayed a generalized service manager that happens to be used as PID 1 in the context of Juju sidecar container charms.

Possible alternatives:

1) A new Juju command juju pebble --container <container> <unit> <pebble-command>, for example juju pebble --container redis snappass-test/0 services. Might be a bit much adding a new juju client command, though. 2) A wrapper/alias installed on the charm container, something like pebble-do -c <container> <cmd>, so you'd say for example juju ssh snappass-test/0 pebble-do -c redis services. I guess you could even call the wrapper just pebble, but it might be a bit confusing to call it the same thing and add -c. (pebble is currently not in the path on the charm, you have to explicitly say /charm/bin/pebble.)

Just for reference, here's the current incantation (which shows why this feature would be useful):

$ juju ssh snappass-test/0 
# PEBBLE_SOCKET=/charm/containers/redis/pebble.socket /charm/bin/pebble services
Service  Startup  Current
redis    enabled  active

Thoughts @hpidcock?

[jnsgruk]

I agree that we should probably make this a little more convenient -- even just adding pebble to the $PATH would be an improvement to start with :)

[mthaddon]

One other thing to mention here is that it would be great to be able to easily run a command via pebble with all the environment variables of a particular layer. Something like /charm/bin/pebble exec --env=services.app.environment command or /charm/bin/pebble exec --service-environment=app command. Currently you have to output the value of the pebble plan and then manually construct all the environment variables yourself and pass those to pebble exec, which can be very time consuming and error prone at a time when you might least need it to be so.

[benhoyt]

@mthaddon I think you'll be pleased to hear about https://github.com/canonical/pebble/pull/246 then. It adds the concept of "service context" to pebble exec and exec health checks. For exec, you specify the service context with pebble exec --context=<service> and it inherits the environment vars, working directory, and user/group from that service. The --context option is overridden by -w, --env, --uid/user, and --gid/group -- that is, local options win.

This is included in the Pebble version baked into Juju 3.2.2, and will be in Juju 3.1.6 (coming out in the next couple of weeks I think).

[benhoyt]

@mthaddon Where do you think would be the best place to socialize these minor Pebble improvements so that you and other charmers see them?

[benhoyt]

I think it would be nice to do something small here in 25.04, so adding that label. It's not too bad right now, but you have to remember the /charm/bin/... prefix. Perhaps we can talk to the Juju team and create a small spec and implement this.

For reference, the ways I usually do this now are:

# Using "juju ssh":
$ juju ssh --container snappass snappass-test/0 /charm/bin/pebble services
Service   Startup  Current  Since
snappass  enabled  active   today at 04:14 UTC

# Using "kubectl" ("t" is the model):
$ microk8s.kubectl -n t exec snappass-test-0 -c snappass -- /charm/bin/pebble services
Service   Startup  Current  Since
snappass  enabled  active   today at 04:14 UTC