Easier management of remote hosts for lxd connection plugin; consistency with lxd_container?

ISSUE TYPE

Feature Idea

COMPONENT NAME

lxd_container module and lxd connection plugin

ANSIBLE VERSION

ansible 2.2.0.0
  config file = /home/brian/ansible-configs/auth-servers/ansible.cfg
  configured module search path = Default w/o overrides

CONFIGURATION

N/A

OS / ENVIRONMENT

N/A

SUMMARY

I want to do these two things:

Create containers on a remote host. This is what the lxd_container module does.
Manage containers on a remote host. This is what the lxd connection plugin does. [^1]

By "remote" I mean that the host running the lxd container is not the host where ansible is running. However, referring to a remote target host is done in two completely different ways:

lxd_container requires a url for the remote host
the lxd connection plugin requires named remotes (<remote>:<container>). e.g. if you create a remote called hostb then you can add to your inventory:
```
foo ansible_connection=lxd ansible_host=hostb:foo
```

But there is no ansible module for managing named remotes.

[^1] Of course, you can also manage remote containers by ssh'ing into them directly just like a normal host. But then the container must already have IP networking fully configured, and ssh authorized_keys set up. If we want ansible to bootstrap the container to this state then we need to use lxd native mechanisms.

Relates to: #2208, #2743, #2744

EXPECTED RESULTS

I can configure a remote lxd server once and use it with both the lxd_container module and the lxd connection plugin.

DETAIL

(Sorry, this may become a bit of a ramble)

Let's say ansible runs on hosta, the lxd server is hostb, and the container on hostb is called foo

If you think about it, there are actually four different ways you can manage a remote lxd instance. I'll call these "modes" for want of a better name.

Use the REST API to talk directly to lxd daemon on host B
ssh to host B, and use the REST API to talk to lxd on localhost
Shell out to the lxc command line on host A using named remotes, e.g. lxc exec hostb:foo ...
ssh to host B, and shell out to the lxc command line there, e.g. lxc exec foo ...

Unfortunately, the current ansible code is a mixture of these.

The lxd_container module uses the REST API. You can use it in mode 1:

- hosts: foo
  gather_facts: no
  tasks:
    - name: create container
      delegate_to: localhost
      lxd_container:
        url: https://192.168.5.82:8443
        name: '{{inventory_hostname}}'
        state: started
        source: ...

For this to work the trust relationships must be set up between the hosts. It is sufficient if you've previously created a remote (with lxd remote add ...) which prompts interactively for the trust password; or you can provide the trust_password as an argument to the module.

Oddly it doesn't appear to verify the server certificate, even when the remote has been removed. I think this is a bug and I'll report it separately.

You can also use it in mode 2:

- hosts: foo
  gather_facts: no
  tasks:
    - name: create container
      delegate_to: hostb
      lxd_container:
        name: '{{inventory_hostname}}'
        state: started
        source: ...

This assumes you're already using ansible to manage hostb (i.e. ssh is setup and working). This is convenient because you don't have to worry about certificate setup, as it uses a Unix domain socket on hostB to talk to the local lxd instance.

All this is fine so far.

Now onto the lxd connection plugin. This currently works by shelling out to the lxc command line tool, i.e. lxc exec ... to run commands, and lxc file push/pull to copy files.

My first attempt to do this on a remote host was with delegate_to (mode 4), but it doesn't work. It's not surprising really: delegate_to delegates the whole action to some other host, not just the lxc exec or lxc file push/pull part of that action [^2]

So we are left with mode 3, i.e. using container names of the form <remote>:<container>. This only works if <remote> has already have been created as a named remote on the local host - and unfortunately, ansible doesn't have a module for creating remotes, so this needs to be done manually.

Once that's setup, you can do this in the inventory:

foo ansible_connection=lxd ansible_host=hostb:foo

And now this works:

- hosts: foo
  gather_facts: no
  tasks:
    - name: do something
      raw: hostname

Aside: gather_facts: no and raw are used here because the fresh container doesn't have python2 yet

PROPOSAL

So we have two completely different mechanisms for referring to remote hosts. How can this be made consistent?

Option 1: change the lxd connection plugin to use the REST API. There would have to be some way to give it a url parameter (and/or trust_password), as with the lxd_container module. Maybe:

foo ansible_connection=lxd ansible_host=foo ansible_lxd_url=https://192.168.5.52:8443

Replacing lxc exec this way is not trivial; it requires using a websocket to redirect stdin/stdout/stderr for the REST API

Option 2: implement an ansible module to manage named remotes, in the same way as there is lxd_profile to manage named profiles.

Then at least you can create the remotes that the lxd connection plugin currently requires.

Option 3: like option 2, but also change the lxd_container module to use named remotes instead of the rest API. That is: either lxd_connection would read the remotes configuration, or it would shell out to the lxc command line tool and not directly use the REST API at all. But that would be a rewrite of the lxd_container module; extensive discussion took place at the time (#2208) and it was decided the REST API was preferred.

[^2] Example: suppose in the inventory we have

foo ansible_connection=lxd
hostb ansible_host=192.168.5.82

The following doesn't do what we want:

- hosts: foo
  gather_facts: no
  tasks:
    - name: do something
      delegate_to: hostb
      shell: hostname

The hostname command here runs on hostb, not inside the container 'foo'. That's fair enough, since we are delegating the hostname command directly to hostb.

And the following doesn't work either:

- hosts: foo
  gather_facts: no
  connection: lxd
  tasks:
    - name: do something
      delegate_to: hostb
      shell: hostname

This gives:

fatal: [foo]: UNREACHABLE! => {"changed": false, "msg": "container not found: 192.168.5.82", "unreachable": true}

i.e. now it's trying to treat hostb itself as an lxd container.

@hnakamur ping, this issue is waiting for your response. click here for bot help

Thanks for your proposal. I would like the option 2.

As for option 1, I agree replacing lxc exec is not trivial. Also it is an incompatible change since it does not support the form <remote>:<container> anymore. Although it is not documented that the form <remote>:<container> is supported in the lxd connection plugin, probably there are people who already use this form to specify a container. https://github.com/ansible/ansible/blob/v2.2.0.0-1/lib/ansible/plugins/connection/lxd.py

Option 3 is also an incompatible change of the lxd_container module.

So, option 2 is my choice.

There are no APIs in LXD REST API to manage remotes. So the new module lxd_remote have to shell out to the lxc remote command.

$ lxc remote --help
Usage: Manage remote LXD servers.

lxc remote add <name> <url> [--accept-certificate] [--password=PASSWORD]
                            [--public] [--protocol=PROTOCOL]                Add the remote <name> at <url>.
lxc remote remove <name>                                                    Remove the remote <name>.
lxc remote list                                                             List all remotes.
lxc remote rename <old> <new>                                               Rename remote <old> to <new>.
lxc remote set-url <name> <url>                                             Update <name>'s url to <url>.
lxc remote set-default <name>                                               Set the default remote.
lxc remote get-default                                                      Print the default remote.

Options:

    --accept-certificate  (= false)
        Accept certificate
    --debug  (= false)
        Enables debug mode.
    --force-local  (= false)
        Force using the local unix socket.
    --no-alias  (= false)
        Ignore aliases when determining what command to run.
    --password (= "")
        Remote admin password
    --protocol (= "")
        Server protocol (lxd or simplestreams)
    --public  (= false)
        Public image server
    --verbose  (= false)
        Enables verbose mode.

To check whether a remote exists or not, we need to look into the output of lxc remote list.

$ lxc remote list
+-----------------+------------------------------------------+---------------+--------+--------+
|      NAME       |                   URL                    |   PROTOCOL    | PUBLIC | STATIC |
+-----------------+------------------------------------------+---------------+--------+--------+
| images          | https://images.linuxcontainers.org       | simplestreams | YES    | NO     |
+-----------------+------------------------------------------+---------------+--------+--------+
| local (default) | unix://                                  | lxd           | NO     | YES    |
+-----------------+------------------------------------------+---------------+--------+--------+
| ubuntu          | https://cloud-images.ubuntu.com/releases | simplestreams | YES    | YES    |
+-----------------+------------------------------------------+---------------+--------+--------+
| ubuntu-daily    | https://cloud-images.ubuntu.com/daily    | simplestreams | YES    | YES    |
+-----------------+------------------------------------------+---------------+--------+--------+

I looked the source code of lxc remote list and found the output format is fixed to table. https://github.com/lxc/lxd/blob/lxd-2.6.2/lxc/remote.go#L376-L388

The lxc command seems to read the config file ~/.config/lxc/config.yml or $LXD_CONF/config.yml if it exists. https://github.com/lxc/lxd/blob/lxd-2.6.2/lxc/main.go#L45-L49 In my environment, this file does not exist.

The default remote list is hardcoded at https://github.com/lxc/lxd/blob/lxd-2.6.2/config.go#L71-L80

So, I think the read the table format output from lxc remote list is the easiest way after all.

Could you tell me what do you think?

By the way, I use lxd on localhost only, so I myself don't need the new lxd_remote module. I'm afraid I have no time to create it since I have other things to do. So, could someone volunteer to create the new lxd_module? Thanks!

There are no APIs in LXD REST API to manage remotes.

That's because "remotes" are an entirely local concept - a configuration of the lxc command line tool itself.

lxc list supports a --format flag to give machine-readable output, e.g. lxc list --format json; but sadly, as you've found, lxc remote list does not.

The lxc command seems to read the config file ~/.config/lxc/config.yml or $LXD_CONF/config.yml if it exists. ... In my environment, this file does not exist.

It does for me:

$ cat ~/.config/lxc/config.yml
default-remote: local
remotes:
  images:
    addr: https://images.linuxcontainers.org
    public: true
    protocol: simplestreams
  wrn-dns1:
    addr: https://192.168.5.82:8443
    public: false
  wrn-dns2:
    addr: https://192.168.5.83:8443
    public: false
aliases: {}

You can see I've added two remotes (using lxc remote add - I did not touch this file directly).

However lxc remote list also shows ubuntu and ubuntu-daily which are not listed in the yml file; I don't know where they come from.

As for option 1, I agree replacing lxc exec is not trivial. Also it is an incompatible change since it does not support the form : anymore.

But it could be made backwards compatible. If a container name contains a colon, you would look up the first part in config.yml to find the url to to use for the REST API.

The existing lxd_container module could also be enhanced to work this way quite easily. That is:

lxd_container:
  name: 'wrn-dns1:foo'

could expand to:

lxd_container:
  name: foo
  url: https://192.168.5.82:8443

Aside: after the expansion you've lost the remote name. This relates to #3663 because the remote name is also used as a label to store the server certificate under ~/.config/lxc/servercerts/<remote>.crt

That's because "remotes" are an entirely local concept - a configuration of the lxc command line tool itself.

Yes. I didn't recognize this fact when I implemented the lxd_container module. And this made me think that maybe I should have use the lxc command instead of the LXD REST API to implement the lxd_container module, since the lxc command covers the whole functions needed and the LXD REST API covers only the subset of them.

So, maybe we should go for the option 3 after all. Even though this is an incompatible change and users have to change their playbooks, maybe switching from the LXD REST API the lxc command is simpler in the long run.

However lxc remote list also shows ubuntu and ubuntu-daily which are not listed in the yml file; I don't know where they come from.

ubuntu and ubuntu daily are hardcoded in the lxc command.

https://github.com/lxc/lxd/blob/lxd-2.6.2/config.go#L71-L74

var StaticRemotes = map[string]RemoteConfig{
    "local":        LocalRemote,
    "ubuntu":       UbuntuRemote,
    "ubuntu-daily": UbuntuDailyRemote}

and set with the following code. https://github.com/lxc/lxd/blob/lxd-2.6.2/config.go#L110-L112

    for k, v := range StaticRemotes {
        c.Remotes[k] = v
    }

This repository has been locked. All new issues and pull requests should be filed in https://github.com/ansible/ansible

Please read through the repomerge page in the dev guide. The guide contains links to tools which automatically move your issue or pull request to the ansible/ansible repo.

This issue was moved to ansible/ansible#18881

ansible / ansible-modules-extras