Ability to run commands on multiple containers

[benaryorg]

I would like to be able to run commands on every container, ideally using wildcards.

This would be useful in the following cases:

• if you have "pairs" of containers each building some sort of system (web+db) and your naming is like - you could restart *-database or alpha-*
• migrating a group of containers
• running updates

As using * in a shell us awful, how about % (SQL-like).

I have a setup where the host system builds all the packages (gentoo ⇒ source distribution) and saves it via it's internal system so the containers can install the binaries. That all is handled by the package manager already. But if I want the host to "ping" the containers to start the updates after the host has finished I would have to build a workaround. greping and then running a loop is not very cool in a cronjob. Updates would take an endless time.

It would be great to provide this functionality out of the box.

[stgraber]

Sounds useful and should be reasonably easy for the commands that already do support multiple containers like stop/start/delete/... but I suspect will be much harder to define for things like exec where user interaction may be expected.

[benaryorg]

I suspected that already so I had the idea to abort automatically if the program asks for input or to read input in the lxc routine from stdin and write that to every single commands input, depending on command line parameters.

[stgraber]

so technically the API supports a "fire and forget" mode for exec which would quite probably work for this. You won't know whether things succeeded though.

[benaryorg]

That could be given as a command line option, but I think that at least stderr should be accessible.

Multiple processes using the same stderr should be perfectly fine (just a bit confusing).

Also waiting for the processes to finish should be implemented, again as an option.

[stgraber]

I'm not sure that stderr will be fine since we're switching into raw mode. Characters will likely end up mixed as the protocol isn't line-by-line if I remember correctly.

[benaryorg]

I guess one would need to have the error handling inside the command itself then.

[srkunze]

@benaryorg About * and shell: one distinct benefit of * is that it doesn't need some sort of explanation. Saltstack is using it as well for targeting minions. And it just explains naturally. :)

[jfgibbins]

I looked, but didn't see this one. You need a some time off if you're remembering these from two years ago at 1 in the morning, lol. But I was thinking, file push would also be useful for regexp, as well, as things like network attach, profile attach, etc. Exclude interactive, config edit, or any procedure that would cause many to one, like a file pull.

[benaryorg]

Seems I've learned a lot over the past year:

lxc list --format json | jq '.[].name'

and similar things are your friend. Yes, of course filtering on LXD's side would be nice, but completely unnecessary in regard to the Unix philosophy as there's jq.

I vote for closing this unless anyone else still has the urge to incorporate any such features into LXD.

[jfgibbins]

Well, wasn't really looking for filtering to get a list back. What the request was for, was to use regexp to type in one argument rather than typing out 50 servers as arguments to a command, ie; lxc delete container* -f, rather than lxc delete container1 container2 ... container50 -f. But thanks for trying benaryorg.

[benaryorg]

Ah, yeah, sorry, apparently I totally missed the point.…

Here's some code for that too:

lxc list --format json | jq -r '.[].name | select(test("^po"))' | xargs echo lxc delete

Which yields lxc delete postfix postgresql for me. Feel free to simply remove the echo after xargs and replace ^po by the regex you want to delete servers.

Although if that isn't what you're looking for, sorry for bothering.

[srkunze]

lxc list --format json | jq -r '.[].name | select(test("^po"))' | xargs echo lxc delete

vs

lxc delete po*

I think I know my favorite. Once this feature is in place, people will start structuring their naming scheme accordingly.

Not saying, your solution wouldn't work, but it's much more error-prone (especially for commands such as delete).

[jfgibbins]

Interesting method, and certainly old school unix, but I see kybd sales going through the roof from all the pounding of the keys. :)

[srkunze]

@stgraber What do you think about this feature? Would you like a pull request?

[stgraber]

I'm fine with it so long as it's pretty self contained (doesn't cause a huge amount of change to every command) and it doesn't cause regressions to existing behavior.

So yeah, if you want to work on this, send us a PR and we can then look into all the tiny details :)

[srkunze]

While setting up an IDE again for this, I've learned that list already has this kind of mechanism using regexes. So, it already seems to be decided: regex instead of globbing.

So it seems it's already there, just needs some refactoring out and applying to other commands. Particular interesting would be exec. I really would love to see that working for exec but it would mean some more work on a complete different matter: make exec work for more than 1 container in the first place.

[srkunze]

@stgraber Is there a particular reason why the filtering of containers is done on the client side?

[jfgibbins]

So the base is there to do it, just not available across the board. That seems like a good thing. Appreciate you putting the time into it. I would, but I just don't know the language. Can't we just all go back to Fortran. :) Picking a programming language nowadays is like going to baskin-robbins.

[stgraber]

@srkunze it's just more flexible that way. "lxc list" gets the "cheap" API answer with the container list, then after that does the more expensive calls only for the containers that will be displayed.

Doing the filtering on the server side wouldn't really makes thing much faster since the same data would still need to be retrieved in the first place and designing a good and user friendly way to filter that data with API requests isn't particularly easy (we have #3364 for that).

[stgraber]

Things we should figure out:

• Container selection (wildcard, regexp, container properties, ...)
• Output handling (mixed, mixed with name prefix, sorted post-exec, live in separate files, ...)

I'd like to keep the lxc exec options somewhat manageable so we'll need to think about a clean way of handling this.

[benaryorg]

Regarding the output format, we've got similar tooling at work, and for output from multiple machines simultaneously, there is not perfect solution (unless you actually split the output into several (e.g. file) streams. If there are several interleaved output-streams there needs to be some sort of chunking. Done line-wise you have problems when output is not a full line, e.g. status output that (assuming sane, control-character-free output) appends dots (e.g. the openssl genpkey commands). If done in chunks with timeouts you have to decide for some lesser human readable and even difficult to parse on a shell format, since you need to indicate whether there was a line-ending or not.

That said, when I wrote similar tooling I decided for a switch that chose an output format. Two modes are currently implemented; TSV and JSON. TSV because it is somewhat human readable and very easy to use on a command line. However, to preserve the nature of being TS (tab-seperated) I had to escape the actual line of output, in which case you need to at least implement \\ and \t, to make it machine readable again. The output is structured as follows:

field	description
execution time	such that the output can later be sorted by that to get the chronological order, otherwise it's hard to tell e.g.sort to leave the internal order untouched when sorting by only one other key
output type	whether the following is a connection error to the server, whether the machine is turned off, or whether actual output follows, here I also implemented exit, to determine exit status and duration for each exec
server	on which server/host it's being executed, great for when you get connection errors
container	which container for obvious reasons
output	either just the output when type is out or err (for stderr), or success for exit code 0 or error with another field for the actual exitcode

The JSON output was also line buffered, but didn't need escaping since JSON handles that for us. The output was just several newline delimited JSON objects, great for parsing with e.g. jq. It's useful to also put the ID of the server in here, e.g. make the container an object having an id, name and server. (Question at this point, is it possible the server changes mid-execution thanks to live-migration?)