search

atc0005 / ansible-role-lxd-testenv

Ansible role used to generate a LXD container test environment
MIT License
0 stars 0 forks source link

FAILED | LXD 3.0.3 with ZFS storage: "Common start logic: no such file or directory" #13

Closed atc0005 closed 5 years ago

atc0005 commented 5 years ago

ansible-playbook -i inventories/testing site.yml -K

SUDO password: 

PLAY [Configure base LXD host] *****************************************************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************************************************************************
ok: [localhost]

TASK [atc0005.lxd-host : Placeholder] **********************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "TODO: Flesh out this role"
}

PLAY [Setup LXD test environment] **************************************************************************************************************************************************************************

TASK [atc0005.lxd-testenv : Create LXD profile to enable Docker use in containers] *************************************************************************************************************************
ok: [localhost -> localhost]

TASK [atc0005.lxd-testenv : DEBUG | Display ssh-keygen command used to generate key for container use] *****************************************************************************************************
skipping: [localhost]

TASK [atc0005.lxd-testenv : Generate key for test environment container use] *******************************************************************************************************************************
ok: [localhost -> localhost]

TASK [atc0005.lxd-testenv : Create containers] *************************************************************************************************************************************************************
changed: [localhost -> localhost] => (item=ansible-centos-5)
changed: [localhost -> localhost] => (item=ansible-ubuntu-5)
changed: [localhost -> localhost] => (item=ansible-centos-1)
changed: [localhost -> localhost] => (item=ansible-centos-2)
changed: [localhost -> localhost] => (item=ansible-centos-3)
changed: [localhost -> localhost] => (item=ansible-centos-4)
changed: [localhost -> localhost] => (item=ansible-ubuntu-docker)
changed: [localhost -> localhost] => (item=ansible-ubuntu-1)
changed: [localhost -> localhost] => (item=ansible-ubuntu-2)
changed: [localhost -> localhost] => (item=ansible-ubuntu-3)
failed: [localhost -> localhost] (item=ansible-ubuntu-4) => {"actions": [], "changed": false, "item": "ansible-ubuntu-4", "msg": "Common start logic: no such file or directory"}

TASK [atc0005.lxd-testenv : Display container creation results] ********************************************************************************************************************************************
skipping: [localhost]
    to retry, use: --limit @/home/ubuntu/Desktop/ansible-playbook-lxd-testenv/site.retry

PLAY RECAP *************************************************************************************************************************************************************************************************
localhost                  : ok=4    changed=0    unreachable=0    failed=1

ubuntu@ubuntu-1604-virtual-machine:~/Desktop/ansible-playbook-lxd-testenv$ lxc list

l+-----------------------+---------+---------------------+------+------------+-----------+
|         NAME          |  STATE  |        IPV4         | IPV6 |    TYPE    | SNAPSHOTS |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-centos-1      | RUNNING | 10.27.16.111 (eth0) |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-centos-2      | RUNNING | 10.27.16.55 (eth0)  |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-centos-3      | RUNNING | 10.27.16.112 (eth0) |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-centos-4      | RUNNING | 10.27.16.147 (eth0) |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-centos-5      | RUNNING | 10.27.16.213 (eth0) |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-ubuntu-1      | RUNNING | 10.27.16.104 (eth0) |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-ubuntu-2      | RUNNING | 10.27.16.191 (eth0) |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-ubuntu-3      | RUNNING | 10.27.16.103 (eth0) |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-ubuntu-4      | STOPPED |                     |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-ubuntu-5      | RUNNING | 10.27.16.87 (eth0)  |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-ubuntu-docker | RUNNING | 10.27.16.18 (eth0)  |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+

ubuntu@ubuntu-1604-virtual-machine:~/Desktop/ansible-playbook-lxd-testenv$ lxc rm --force ansible-ubuntu-4

Error: remove /var/lib/lxd/storage-pools/lxd/containers/ansible-ubuntu-4: directory not empty

A quick Google search suggests this has something to do with ZFS.

Potentially related:

atc0005 commented 5 years ago

$ lxc storage show lxd

config:
  source: lxd
  zfs.pool_name: lxd
description: ""
name: lxd
driver: zfs
used_by:
- /1.0/containers/ansible-centos-1
- /1.0/containers/ansible-centos-2
- /1.0/containers/ansible-centos-3
- /1.0/containers/ansible-centos-4
- /1.0/containers/ansible-centos-5
- /1.0/containers/ansible-ubuntu-1
- /1.0/containers/ansible-ubuntu-2
- /1.0/containers/ansible-ubuntu-3
- /1.0/containers/ansible-ubuntu-4
- /1.0/containers/ansible-ubuntu-5
- /1.0/containers/ansible-ubuntu-docker
- /1.0/images/115495eb719cb7edd974970b25f9633fe0169eb85a824b9bb8f90a724076ae85
- /1.0/images/a52f5118717ebd8c4c0fadf9c8925e2373c7c1a7120a7a76b4e698786f2cbd5c
- /1.0/images/beb29682c9b598c824592166ee619c8478f5ccba226f525a7d7c0e8810bf3874
- /1.0/profiles/default
status: Created
locations:
- none

$ lxc delete --force ansible-centos-{1..5} $ lxc list

+-----------------------+---------+---------------------+------+------------+-----------+
|         NAME          |  STATE  |        IPV4         | IPV6 |    TYPE    | SNAPSHOTS |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-ubuntu-1      | RUNNING | 10.27.16.104 (eth0) |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-ubuntu-2      | RUNNING | 10.27.16.191 (eth0) |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-ubuntu-3      | RUNNING | 10.27.16.103 (eth0) |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-ubuntu-4      | STOPPED |                     |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-ubuntu-5      | RUNNING | 10.27.16.87 (eth0)  |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-ubuntu-docker | RUNNING | 10.27.16.18 (eth0)  |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+

$ lxc delete --force ansible-ubuntu-{1..3} $ lxc delete --force ansible-ubuntu-5 $ lxc delete --force ansible-ubuntu-docker $ lxc list

+------------------+---------+------+------+------------+-----------+
|       NAME       |  STATE  | IPV4 | IPV6 |    TYPE    | SNAPSHOTS |
+------------------+---------+------+------+------------+-----------+
| ansible-ubuntu-4 | STOPPED |      |      | PERSISTENT | 0         |
+------------------+---------+------+------+------------+-----------+

$ lxc delete ansible-ubuntu-4

Error: remove /var/lib/lxd/storage-pools/lxd/containers/ansible-ubuntu-4: directory not empty

$ lxc list

+------------------+---------+------+------+------------+-----------+
|       NAME       |  STATE  | IPV4 | IPV6 |    TYPE    | SNAPSHOTS |
+------------------+---------+------+------+------------+-----------+
| ansible-ubuntu-4 | STOPPED |      |      | PERSISTENT | 0         |
+------------------+---------+------+------+------------+-----------+

$ lxc storage show lxd

config:
  source: lxd
  zfs.pool_name: lxd
description: ""
name: lxd
driver: zfs
used_by:
- /1.0/containers/ansible-ubuntu-4
- /1.0/images/115495eb719cb7edd974970b25f9633fe0169eb85a824b9bb8f90a724076ae85
- /1.0/images/a52f5118717ebd8c4c0fadf9c8925e2373c7c1a7120a7a76b4e698786f2cbd5c
- /1.0/images/beb29682c9b598c824592166ee619c8478f5ccba226f525a7d7c0e8810bf3874
- /1.0/profiles/default
status: Created
locations:
- none

$ lxc image list

+-------+--------------+--------+-----------------------------------------------+--------+----------+------------------------------+
| ALIAS | FINGERPRINT  | PUBLIC |                  DESCRIPTION                  |  ARCH  |   SIZE   |         UPLOAD DATE          |
+-------+--------------+--------+-----------------------------------------------+--------+----------+------------------------------+
|       | 115495eb719c | no     | Centos 7 amd64 (20190209_07:36)               | x86_64 | 124.86MB | Feb 9, 2019 at 11:32am (UTC) |
+-------+--------------+--------+-----------------------------------------------+--------+----------+------------------------------+
|       | a52f5118717e | no     | Ubuntu xenial amd64 (20190209_07:43)          | x86_64 | 105.81MB | Feb 9, 2019 at 11:32am (UTC) |
+-------+--------------+--------+-----------------------------------------------+--------+----------+------------------------------+
|       | beb29682c9b5 | no     | ubuntu 16.04 LTS amd64 (release) (20190204.3) | x86_64 | 158.26MB | Feb 4, 2019 at 9:20pm (UTC)  |
+-------+--------------+--------+-----------------------------------------------+--------+----------+------------------------------+

$ lxc image delete 115495eb719c $ lxc image delete a52f5118717e $ lxc image delete beb29682c9b5

$ lxc storage show lxd

config:
  source: lxd
  zfs.pool_name: lxd
description: ""
name: lxd
driver: zfs
used_by:
- /1.0/containers/ansible-ubuntu-4
- /1.0/profiles/default
status: Created
locations:
- none

$ lxc delete --force ansible-ubuntu-4

Error: remove /var/lib/lxd/storage-pools/lxd/containers/ansible-ubuntu-4: directory not empty

$ sudo zpool list

NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
lxd   39.8G  4.69M  39.7G         -     6%     0%  1.00x  ONLINE  -

$ sudo zfs list

NAME                 USED  AVAIL  REFER  MOUNTPOINT
lxd                 4.48M  38.5G    19K  none
lxd/containers        19K  38.5G    19K  none
lxd/deleted           38K  38.5G    19K  none
lxd/deleted/images    19K  38.5G    19K  none
lxd/images            19K  38.5G    19K  none

$ uname -a

Linux ubuntu-1604-virtual-machine 4.4.0-142-generic #168-Ubuntu SMP Wed Jan 16 21:00:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Worth noting: There were recent kernel (shown above) and zfs updates installed.

atc0005 commented 5 years ago

$ sudo ls -R /var/lib/lxd/storage-pools/lxd/containers/ansible-ubuntu-4

/var/lib/lxd/storage-pools/lxd/containers/ansible-ubuntu-4:
backup.yaml

root@ubuntu-1604-virtual-machine:/var/lib/lxd/storage-pools/lxd/containers/ansible-ubuntu-4# ls -l

total 4
-r-------- 1 root root 2903 Feb  8 08:55 backup.yaml

root@ubuntu-1604-virtual-machine:/var/lib/lxd/storage-pools/lxd/containers/ansible-ubuntu-4# cat backup.yml

cat: backup.yml: No such file or directory

This is odd.

atc0005 commented 5 years ago

root@ubuntu-1604-virtual-machine:/var/lib/lxd/storage-pools/lxd/containers# ls -larth ansible-ubuntu-4/

total 12K
-r-------- 1 root root 2.9K Feb  8 08:55 backup.yaml
drwx--x--x 2 root root 4.0K Feb  8 08:55 .
drwx--x--x 3 root root 4.0K Feb  9 22:39 ..
atc0005 commented 5 years ago

# grep ansible-ubuntu-4 /var/log/syslog | tail

Feb  9 22:19:16 ubuntu-1604-virtual-machine ansible-lxd_container: Invoked with description=None source={'alias': 'ubuntu/xenial/amd64', 'type': 'image', 'mode': 'pull', 'protocol': 'lxd', 'server': 'https://images.linuxcontainers.org'} url=unix:/var/lib/lxd/unix.socket config={'environment.http_proxy': '', 'environment.ftp_proxy': '', 'environment.https_proxy': ''} cert_file=/home/ubuntu/.config/lxc/client.crt ephemeral=None architecture=None devices=None wait_for_ipv4_addresses=True state=started profiles=['default'] timeout=600 key_file=/home/ubuntu/.config/lxc/client.key trust_password=NOT_LOGGING_PARAMETER force_stop=False name=ansible-ubuntu-4
Feb  9 22:19:26 ubuntu-1604-virtual-machine lxd[4885]: t=2019-02-09T22:19:26-0600 lvl=eror msg="Failed to mount ZFS dataset \"lxd/containers/ansible-ubuntu-4\" onto \"/var/lib/lxd/storage-pools/lxd/containers/ansible-ubuntu-4\""
Feb  9 22:29:08 ubuntu-1604-virtual-machine ansible-lxd_container: Invoked with description=None source={'alias': 'ubuntu/xenial/amd64', 'type': 'image', 'mode': 'pull', 'protocol': 'lxd', 'server': 'https://images.linuxcontainers.org'} url=unix:/var/lib/lxd/unix.socket config={'environment.http_proxy': '', 'environment.ftp_proxy': '', 'environment.https_proxy': ''} cert_file=/home/ubuntu/.config/lxc/client.crt ephemeral=None architecture=None devices=None wait_for_ipv4_addresses=True state=started profiles=['default'] timeout=600 key_file=/home/ubuntu/.config/lxc/client.key trust_password=NOT_LOGGING_PARAMETER force_stop=False name=ansible-ubuntu-4
Feb  9 22:29:18 ubuntu-1604-virtual-machine lxd[4885]: t=2019-02-09T22:29:18-0600 lvl=eror msg="Failed to mount ZFS dataset \"lxd/containers/ansible-ubuntu-4\" onto \"/var/lib/lxd/storage-pools/lxd/containers/ansible-ubuntu-4\""
Feb  9 22:31:18 ubuntu-1604-virtual-machine lxd[4885]: t=2019-02-09T22:31:18-0600 lvl=eror msg="Failed deleting container storage" err="remove /var/lib/lxd/storage-pools/lxd/containers/ansible-ubuntu-4: directory not empty" name=ansible-ubuntu-4
Feb  9 22:39:16 ubuntu-1604-virtual-machine lxd[4885]: t=2019-02-09T22:39:16-0600 lvl=eror msg="Failed deleting container storage" err="remove /var/lib/lxd/storage-pools/lxd/containers/ansible-ubuntu-4: directory not empty" name=ansible-ubuntu-4
Feb  9 22:41:18 ubuntu-1604-virtual-machine lxd[4885]: t=2019-02-09T22:41:18-0600 lvl=eror msg="Failed deleting container storage" err="remove /var/lib/lxd/storage-pools/lxd/containers/ansible-ubuntu-4: directory not empty" name=ansible-ubuntu-4
atc0005 commented 5 years ago

# grep zfs /var/log/apt'history.log | sed -e 's/Upgrade: //' -e 's/), /),\n/g' | sort

code:amd64 (1.30.2-1546901646, 1.31.0-1549443364),
curl:amd64 (7.47.0-1ubuntu2.11, 7.47.0-1ubuntu2.12),
fonts-opensymbol:amd64 (2:102.7+LibO5.1.6~rc2-0ubuntu1~xenial4, 2:102.7+LibO5.1.6~rc2-0ubuntu1~xenial6),
google-chrome-stable:amd64 (72.0.3626.81-1, 72.0.3626.96-1),
libarchive13:amd64 (3.1.2-11ubuntu0.16.04.5, 3.1.2-11ubuntu0.16.04.6),
libcurl3:amd64 (7.47.0-1ubuntu2.11, 7.47.0-1ubuntu2.12),
libcurl3-gnutls:amd64 (7.47.0-1ubuntu2.11, 7.47.0-1ubuntu2.12),
libnvpair1linux:amd64 (0.6.5.6-0ubuntu26, 0.6.5.6-0ubuntu27),
libreoffice-avmedia-backend-gstreamer:amd64 (1:5.1.6~rc2-0ubuntu1~xenial4, 1:5.1.6~rc2-0ubuntu1~xenial6)
libreoffice-base-core:amd64 (1:5.1.6~rc2-0ubuntu1~xenial4, 1:5.1.6~rc2-0ubuntu1~xenial6),
libreoffice-calc:amd64 (1:5.1.6~rc2-0ubuntu1~xenial4, 1:5.1.6~rc2-0ubuntu1~xenial6),
libreoffice-common:amd64 (1:5.1.6~rc2-0ubuntu1~xenial4, 1:5.1.6~rc2-0ubuntu1~xenial6),
libreoffice-core:amd64 (1:5.1.6~rc2-0ubuntu1~xenial4, 1:5.1.6~rc2-0ubuntu1~xenial6),
libreoffice-draw:amd64 (1:5.1.6~rc2-0ubuntu1~xenial4, 1:5.1.6~rc2-0ubuntu1~xenial6),
libreoffice-gnome:amd64 (1:5.1.6~rc2-0ubuntu1~xenial4, 1:5.1.6~rc2-0ubuntu1~xenial6),
libreoffice-gtk:amd64 (1:5.1.6~rc2-0ubuntu1~xenial4, 1:5.1.6~rc2-0ubuntu1~xenial6),
libreoffice-impress:amd64 (1:5.1.6~rc2-0ubuntu1~xenial4, 1:5.1.6~rc2-0ubuntu1~xenial6),
libreoffice-math:amd64 (1:5.1.6~rc2-0ubuntu1~xenial4, 1:5.1.6~rc2-0ubuntu1~xenial6),
libreoffice-ogltrans:amd64 (1:5.1.6~rc2-0ubuntu1~xenial4, 1:5.1.6~rc2-0ubuntu1~xenial6),
libreoffice-pdfimport:amd64 (1:5.1.6~rc2-0ubuntu1~xenial4, 1:5.1.6~rc2-0ubuntu1~xenial6),
libreoffice-style-breeze:amd64 (1:5.1.6~rc2-0ubuntu1~xenial4, 1:5.1.6~rc2-0ubuntu1~xenial6),
libreoffice-style-galaxy:amd64 (1:5.1.6~rc2-0ubuntu1~xenial4, 1:5.1.6~rc2-0ubuntu1~xenial6),
libreoffice-style-human:amd64 (1:5.1.6~rc2-0ubuntu1~xenial4, 1:5.1.6~rc2-0ubuntu1~xenial6),
libreoffice-writer:amd64 (1:5.1.6~rc2-0ubuntu1~xenial4, 1:5.1.6~rc2-0ubuntu1~xenial6),
libuutil1linux:amd64 (0.6.5.6-0ubuntu26, 0.6.5.6-0ubuntu27),
libzfs2linux:amd64 (0.6.5.6-0ubuntu26, 0.6.5.6-0ubuntu27),
libzpool2linux:amd64 (0.6.5.6-0ubuntu26, 0.6.5.6-0ubuntu27),
openssh-client:amd64 (1:7.2p2-4ubuntu2.6, 1:7.2p2-4ubuntu2.7),
openssh-server:amd64 (1:7.2p2-4ubuntu2.6, 1:7.2p2-4ubuntu2.7),
openssh-sftp-server:amd64 (1:7.2p2-4ubuntu2.6, 1:7.2p2-4ubuntu2.7),
python3-uno:amd64 (1:5.1.6~rc2-0ubuntu1~xenial4, 1:5.1.6~rc2-0ubuntu1~xenial6),
uno-libs3:amd64 (5.1.6~rc2-0ubuntu1~xenial4, 5.1.6~rc2-0ubuntu1~xenial6),
ure:amd64 (5.1.6~rc2-0ubuntu1~xenial4, 5.1.6~rc2-0ubuntu1~xenial6),
zfs-doc:amd64 (0.6.5.6-0ubuntu26, 0.6.5.6-0ubuntu27),
zfsutils-linux:amd64 (0.6.5.6-0ubuntu26, 0.6.5.6-0ubuntu27),
zfs-zed:amd64 (0.6.5.6-0ubuntu26, 0.6.5.6-0ubuntu27),
atc0005 commented 5 years ago

# dmesg | grep ubuntu-4

[  399.880787] audit: type=1400 audit(1549582185.166:44): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-ansible-ubuntu-4_</var/lib/lxd>" pid=9075 comm="apparmor_parser"
[  710.038717] audit: type=1400 audit(1549582496.141:62): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-ansible-ubuntu-4_</var/lib/lxd>" pid=14325 comm="apparmor_parser"
[  883.467688] audit: type=1400 audit(1549582670.271:84): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-ansible-ubuntu-4_</var/lib/lxd>" pid=22380 comm="apparmor_parser"
[  957.453490] audit: type=1400 audit(1549582744.288:95): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-ansible-ubuntu-4_</var/lib/lxd>" pid=26694 comm="apparmor_parser"
[ 1138.465359] audit: type=1400 audit(1549582925.966:106): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-ansible-ubuntu-4_</var/lib/lxd>" pid=32439 comm="apparmor_parser"
[ 3781.445678] audit: type=1400 audit(1549585573.977:109): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-ansible-ubuntu-4_</var/lib/lxd>" pid=51911 comm="apparmor_parser"
[ 3923.556917] audit: type=1400 audit(1549585717.224:129): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-ansible-ubuntu-4_</var/lib/lxd>" pid=59654 comm="apparmor_parser"
[49125.580599] audit: type=1400 audit(1549631024.090:132): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-ansible-ubuntu-4_</var/lib/lxd>" pid=9456 comm="apparmor_parser"
[49310.012614] audit: type=1400 audit(1549631208.611:151): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-ansible-ubuntu-4_</var/lib/lxd>" pid=16694 comm="apparmor_parser"
[50581.526996] audit: type=1400 audit(1549632483.591:155): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-ansible-ubuntu-4_</var/lib/lxd>" pid=40461 comm="apparmor_parser"
[50734.207533] audit: type=1400 audit(1549632636.347:174): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-ansible-ubuntu-4_</var/lib/lxd>" pid=47792 comm="apparmor_parser"
[55498.894302] audit: type=1400 audit(1549637412.507:177): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-ansible-ubuntu-4_</var/lib/lxd>" pid=7790 comm="apparmor_parser"
[55841.415234] audit: type=1400 audit(1549637755.496:195): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-ansible-ubuntu-4_</var/lib/lxd>" pid=16408 comm="apparmor_parser"
[56911.473198] audit: type=1400 audit(1549638827.505:201): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-ansible-ubuntu-4_</var/lib/lxd>" pid=41627 comm="apparmor_parser"
atc0005 commented 5 years ago

root@ubuntu-1604-virtual-machine:/var/lib/lxd/storage-pools/lxd/containers# ls -l

total 4
drwx--x--x 2 root root 4096 Feb  8 08:55 ansible-ubuntu-4

root@ubuntu-1604-virtual-machine:/var/lib/lxd/storage-pools/lxd/containers# rm -rf ansible-ubuntu-4/

root@ubuntu-1604-virtual-machine:/var/lib/lxd/storage-pools/lxd/containers# ls -l

total 0
atc0005 commented 5 years ago

ansible-playbook -i inventories/testing site.yml -K

SUDO password: 

PLAY [Configure base LXD host] *****************************************************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************************************************************************
ok: [localhost]

TASK [atc0005.lxd-host : Placeholder] **********************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "TODO: Flesh out this role"
}

PLAY [Setup LXD test environment] **************************************************************************************************************************************************************************

TASK [atc0005.lxd-testenv : Create LXD profile to enable Docker use in containers] *************************************************************************************************************************
ok: [localhost -> localhost]

TASK [atc0005.lxd-testenv : DEBUG | Display ssh-keygen command used to generate key for container use] *****************************************************************************************************
skipping: [localhost]

TASK [atc0005.lxd-testenv : Generate key for test environment container use] *******************************************************************************************************************************
ok: [localhost -> localhost]

TASK [atc0005.lxd-testenv : Create containers] *************************************************************************************************************************************************************
changed: [localhost -> localhost] => (item=ansible-ubuntu-docker)
changed: [localhost -> localhost] => (item=ansible-centos-5)
changed: [localhost -> localhost] => (item=ansible-ubuntu-5)
changed: [localhost -> localhost] => (item=ansible-centos-1)
changed: [localhost -> localhost] => (item=ansible-centos-2)
changed: [localhost -> localhost] => (item=ansible-centos-3)
changed: [localhost -> localhost] => (item=ansible-centos-4)
changed: [localhost -> localhost] => (item=ansible-ubuntu-1)
changed: [localhost -> localhost] => (item=ansible-ubuntu-2)
changed: [localhost -> localhost] => (item=ansible-ubuntu-3)
failed: [localhost -> localhost] (item=ansible-ubuntu-4) => {"actions": [], "changed": false, "item": "ansible-ubuntu-4", "msg": "Common start logic: no such file or directory"}

TASK [atc0005.lxd-testenv : Display container creation results] ********************************************************************************************************************************************
skipping: [localhost]
    to retry, use: --limit @/home/ubuntu/Desktop/ansible-playbook-lxd-testenv/site.retry

PLAY RECAP *************************************************************************************************************************************************************************************************
localhost                  : ok=4    changed=0    unreachable=0    failed=1
atc0005 commented 5 years ago

$ sudo zfs list

[sudo] password for ubuntu: 
NAME                                                                          USED  AVAIL  REFER  MOUNTPOINT
lxd                                                                           421M  38.1G    19K  none
lxd/containers                                                               34.6M  38.1G    19K  none
lxd/containers/ansible-centos-1                                              4.33M  38.1G   214M  /var/lib/lxd/storage-pools/lxd/containers/ansible-centos-1
lxd/containers/ansible-centos-2                                              4.34M  38.1G   214M  /var/lib/lxd/storage-pools/lxd/containers/ansible-centos-2
lxd/containers/ansible-centos-3                                              4.34M  38.1G   214M  /var/lib/lxd/storage-pools/lxd/containers/ansible-centos-3
lxd/containers/ansible-centos-4                                              4.34M  38.1G   214M  /var/lib/lxd/storage-pools/lxd/containers/ansible-centos-4
lxd/containers/ansible-centos-5                                              4.34M  38.1G   214M  /var/lib/lxd/storage-pools/lxd/containers/ansible-centos-5
lxd/containers/ansible-ubuntu-1                                              2.57M  38.1G   167M  /var/lib/lxd/storage-pools/lxd/containers/ansible-ubuntu-1
lxd/containers/ansible-ubuntu-2                                              2.57M  38.1G   167M  /var/lib/lxd/storage-pools/lxd/containers/ansible-ubuntu-2
lxd/containers/ansible-ubuntu-3                                              2.57M  38.1G   167M  /var/lib/lxd/storage-pools/lxd/containers/ansible-ubuntu-3
lxd/containers/ansible-ubuntu-5                                              2.58M  38.1G   167M  /var/lib/lxd/storage-pools/lxd/containers/ansible-ubuntu-5
lxd/containers/ansible-ubuntu-docker                                         2.57M  38.1G   167M  /var/lib/lxd/storage-pools/lxd/containers/ansible-ubuntu-docker
lxd/deleted                                                                    38K  38.1G    19K  none
lxd/deleted/images                                                             19K  38.1G    19K  none
lxd/images                                                                    382M  38.1G    19K  none
lxd/images/115495eb719cb7edd974970b25f9633fe0169eb85a824b9bb8f90a724076ae85   215M  38.1G   215M  none
lxd/images/a52f5118717ebd8c4c0fadf9c8925e2373c7c1a7120a7a76b4e698786f2cbd5c   167M  38.1G   167M  none
atc0005 commented 5 years ago

$ lxc storage show lxd

config:
  source: lxd
  zfs.pool_name: lxd
description: ""
name: lxd
driver: zfs
used_by:
- /1.0/containers/ansible-centos-1
- /1.0/containers/ansible-centos-2
- /1.0/containers/ansible-centos-3
- /1.0/containers/ansible-centos-4
- /1.0/containers/ansible-centos-5
- /1.0/containers/ansible-ubuntu-1
- /1.0/containers/ansible-ubuntu-2
- /1.0/containers/ansible-ubuntu-3
- /1.0/containers/ansible-ubuntu-4
- /1.0/containers/ansible-ubuntu-5
- /1.0/containers/ansible-ubuntu-docker
- /1.0/images/115495eb719cb7edd974970b25f9633fe0169eb85a824b9bb8f90a724076ae85
- /1.0/images/a52f5118717ebd8c4c0fadf9c8925e2373c7c1a7120a7a76b4e698786f2cbd5c
- /1.0/profiles/default
status: Created
locations:
- none

$ lxc delete ansible-ubuntu-4 $ lxc delete ansible-ubuntu-4

Error: not found

First time (silent) success, second time expected "not found" message.

atc0005 commented 5 years ago

Relevant /var/log/syslog entries:

Feb  9 23:12:53 ubuntu-1604-virtual-machine ansible-lxd_container: Invoked with description=None source={'alias': 'ubuntu/xenial/amd64', 'type': 'image', 'mode': 'pull', 'protocol': 'lxd', 'server': 'https://images.linuxcontainers.org'} url=unix:/var/lib/lxd/unix.socket config={'environment.http_proxy': '', 'environment.ftp_proxy': '', 'environment.https_proxy': ''} cert_file=/home/ubuntu/.config/lxc/client.crt ephemeral=None architecture=None devices=None wait_for_ipv4_addresses=True state=started profiles=['default'] timeout=600 key_file=/home/ubuntu/.config/lxc/client.key trust_password=NOT_LOGGING_PARAMETER force_stop=False name=ansible-ubuntu-4
Feb  9 23:13:03 ubuntu-1604-virtual-machine lxd[4885]: t=2019-02-09T23:13:03-0600 lvl=eror msg="Failed to mount ZFS dataset \"lxd/containers/ansible-ubuntu-4\" onto \"/var/lib/lxd/storage-pools/lxd/containers/ansible-ubuntu-4\""

root@ubuntu-1604-virtual-machine:/var/lib/lxd/storage-pools/lxd/containers# ls -l

total 15
drwx--x--x 4 165536 165536 6 Feb  9 23:11 ansible-centos-1
drwx--x--x 4 165536 165536 6 Feb  9 23:12 ansible-centos-2
drwx--x--x 4 165536 165536 6 Feb  9 23:12 ansible-centos-3
drwx--x--x 4 165536 165536 6 Feb  9 23:12 ansible-centos-4
drwx--x--x 4 165536 165536 6 Feb  9 23:11 ansible-centos-5
drwx--x--x 4 165536 165536 6 Feb  9 23:12 ansible-ubuntu-1
drwx--x--x 4 165536 165536 6 Feb  9 23:12 ansible-ubuntu-2
drwx--x--x 4 165536 165536 6 Feb  9 23:12 ansible-ubuntu-3
drwx--x--x 4 165536 165536 6 Feb  9 23:11 ansible-ubuntu-5
drwx--x--x 4 165536 165536 6 Feb  9 23:11 ansible-ubuntu-docker

root@ubuntu-1604-virtual-machine:/var/lib/lxd/storage-pools/lxd/containers# lxc list

+-----------------------+---------+---------------------+------+------------+-----------+
|         NAME          |  STATE  |        IPV4         | IPV6 |    TYPE    | SNAPSHOTS |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-centos-1      | RUNNING | 10.27.16.204 (eth0) |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-centos-2      | RUNNING | 10.27.16.68 (eth0)  |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-centos-3      | RUNNING | 10.27.16.127 (eth0) |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-centos-4      | RUNNING | 10.27.16.205 (eth0) |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-centos-5      | RUNNING | 10.27.16.208 (eth0) |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-ubuntu-1      | RUNNING | 10.27.16.106 (eth0) |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-ubuntu-2      | RUNNING | 10.27.16.229 (eth0) |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-ubuntu-3      | RUNNING | 10.27.16.80 (eth0)  |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-ubuntu-5      | RUNNING | 10.27.16.112 (eth0) |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+
| ansible-ubuntu-docker | RUNNING | 10.27.16.107 (eth0) |      | PERSISTENT | 0         |
+-----------------------+---------+---------------------+------+------------+-----------+
atc0005 commented 5 years ago

$ lxc init ubuntu:16.04 ansible-ubuntu-4

Creating ansible-ubuntu-4

$ lxc start ansible-ubuntu-4 $ lxc info ansible-ubuntu-4

Name: ansible-ubuntu-4
Remote: unix://
Architecture: x86_64
Created: 2019/02/10 05:24 UTC
Status: Running
Type: persistent
Profiles: default
Pid: 31008
Ips:
  eth0: inet    10.27.16.138    vethG9E1LH
  eth0: inet6   fe80::216:3eff:fe19:4e35    vethG9E1LH
  lo:   inet    127.0.0.1
  lo:   inet6   ::1
Resources:
  Processes: 38
  Disk usage:
    root: 5.47MB
  CPU usage:
    CPU usage (in seconds): 5
  Memory usage:
    Memory (current): 217.45MB
    Memory (peak): 217.51MB
  Network usage:
    lo:
      Bytes received: 0B
      Bytes sent: 0B
      Packets received: 0
      Packets sent: 0
    eth0:
      Bytes received: 404.86kB
      Bytes sent: 6.48kB
      Packets received: 100
      Packets sent: 79
atc0005 commented 5 years ago

ansible-playbook -i inventories/testing site.yml -K

SUDO password: 

PLAY [Configure base LXD host] *****************************************************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************************************************************************
ok: [localhost]

TASK [atc0005.lxd-host : Placeholder] **********************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "TODO: Flesh out this role"
}

PLAY [Setup LXD test environment] **************************************************************************************************************************************************************************

TASK [atc0005.lxd-testenv : Create LXD profile to enable Docker use in containers] *************************************************************************************************************************
ok: [localhost -> localhost]

TASK [atc0005.lxd-testenv : DEBUG | Display ssh-keygen command used to generate key for container use] *****************************************************************************************************
skipping: [localhost]

TASK [atc0005.lxd-testenv : Generate key for test environment container use] *******************************************************************************************************************************
ok: [localhost -> localhost]

TASK [atc0005.lxd-testenv : Create containers] *************************************************************************************************************************************************************
ok: [localhost -> localhost] => (item=ansible-centos-5)
ok: [localhost -> localhost] => (item=ansible-ubuntu-5)
ok: [localhost -> localhost] => (item=ansible-centos-1)
ok: [localhost -> localhost] => (item=ansible-centos-2)
ok: [localhost -> localhost] => (item=ansible-centos-3)
ok: [localhost -> localhost] => (item=ansible-centos-4)
ok: [localhost -> localhost] => (item=ansible-ubuntu-docker)
ok: [localhost -> localhost] => (item=ansible-ubuntu-2)
ok: [localhost -> localhost] => (item=ansible-ubuntu-3)
ok: [localhost -> localhost] => (item=ansible-ubuntu-1)
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'environment.http_proxy'
failed: [localhost -> localhost] (item=ansible-ubuntu-4) => {"changed": false, "item": "ansible-ubuntu-4", "module_stderr": "Traceback (most recent call last):\n  File \"/home/ubuntu/.ansible/tmp/ansible-tmp-1549776536.47-160132870368507/AnsiballZ_lxd_container.py\", line 113, in <module>\n    _ansiballz_main()\n  File \"/home/ubuntu/.ansible/tmp/ansible-tmp-1549776536.47-160132870368507/AnsiballZ_lxd_container.py\", line 105, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/ubuntu/.ansible/tmp/ansible-tmp-1549776536.47-160132870368507/AnsiballZ_lxd_container.py\", line 48, in invoke_module\n    imp.load_module('__main__', mod, module, MOD_DESC)\n  File \"/tmp/ansible_lxd_container_payload_UT_lLO/__main__.py\", line 603, in <module>\n  File \"/tmp/ansible_lxd_container_payload_UT_lLO/__main__.py\", line 599, in main\n  File \"/tmp/ansible_lxd_container_payload_UT_lLO/__main__.py\", line 509, in run\n  File \"/tmp/ansible_lxd_container_payload_UT_lLO/__main__.py\", line 401, in _started\n  File \"/tmp/ansible_lxd_container_payload_UT_lLO/__main__.py\", line 471, in _needs_to_apply_container_configs\n  File \"/tmp/ansible_lxd_container_payload_UT_lLO/__main__.py\", line 461, in _needs_to_change_container_config\nKeyError: 'environment.http_proxy'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

TASK [atc0005.lxd-testenv : Display container creation results] ********************************************************************************************************************************************
skipping: [localhost]
    to retry, use: --limit @/home/ubuntu/Desktop/ansible-playbook-lxd-testenv/site.retry

PLAY RECAP *************************************************************************************************************************************************************************************************
localhost                  : ok=4    changed=0    unreachable=0    failed=1
atc0005 commented 5 years ago

Rebooted VM just to rule that out.

atc0005 commented 5 years ago

The playbook completed (including container creation) without issue. Going to run the removal playbook to see how things go.

atc0005 commented 5 years ago

One thing worth noting is that the removal playbook changed in the last few days from stopping containers first to just outright deleting them, which presumably is the equivalent of lxc delete --force.

$ time ansible-playbook -i inventories/testing lxd-remove.yml -K

SUDO password: 

PLAY [Tear down our LXD test environment] ******************************************************************************************************************************************************************

TASK [atc0005.lxd-testenv : Remove container entries from hosts file on LXD host] **************************************************************************************************************************
changed: [localhost -> localhost]

TASK [atc0005.lxd-testenv : Remove container host entries from SSH client user config on host] *************************************************************************************************************
skipping: [localhost]

TASK [atc0005.lxd-testenv : Remove container host entries from SSH client system config on host] ***********************************************************************************************************
changed: [localhost -> localhost]

TASK [atc0005.lxd-testenv : Remove container host keys from SSH client system config on host] **************************************************************************************************************
skipping: [localhost]

TASK [atc0005.lxd-testenv : Remove container host keys from SSH client user config on host] ****************************************************************************************************************
skipping: [localhost]

TASK [atc0005.lxd-testenv : Stop containers] ***************************************************************************************************************************************************************
skipping: [localhost] => (item=ansible-centos-4) 
skipping: [localhost] => (item=ansible-ubuntu-4) 
skipping: [localhost] => (item=ansible-centos-5) 
skipping: [localhost] => (item=ansible-ubuntu-5) 
skipping: [localhost] => (item=ansible-ubuntu-docker) 
skipping: [localhost] => (item=ansible-centos-2) 
skipping: [localhost] => (item=ansible-ubuntu-2) 
skipping: [localhost] => (item=ansible-ubuntu-1) 
skipping: [localhost] => (item=ansible-ubuntu-3) 
skipping: [localhost] => (item=ansible-centos-3) 
skipping: [localhost] => (item=ansible-centos-1) 

TASK [atc0005.lxd-testenv : Delete containers] *************************************************************************************************************************************************************
changed: [localhost -> localhost] => (item=ansible-centos-4)
changed: [localhost -> localhost] => (item=ansible-ubuntu-4)
changed: [localhost -> localhost] => (item=ansible-centos-5)
changed: [localhost -> localhost] => (item=ansible-ubuntu-5)
changed: [localhost -> localhost] => (item=ansible-ubuntu-docker)
changed: [localhost -> localhost] => (item=ansible-centos-2)
changed: [localhost -> localhost] => (item=ansible-ubuntu-2)
changed: [localhost -> localhost] => (item=ansible-ubuntu-1)
changed: [localhost -> localhost] => (item=ansible-ubuntu-3)
changed: [localhost -> localhost] => (item=ansible-centos-3)
changed: [localhost -> localhost] => (item=ansible-centos-1)

PLAY RECAP *************************************************************************************************************************************************************************************************
localhost                  : ok=3    changed=3    unreachable=0    failed=0   

real    0m34.194s
user    0m3.744s
sys 0m0.912s
atc0005 commented 5 years ago

At this point I can't reproduce anymore and troubleshooting ZFS and LXD 3.0.3 issues are outside the scope of what I'm trying to accomplish. Will update further if the issue reoccurs.