dokime7 commented 6 years ago

Required information

Distribution: Ubuntu
Distribution version: 18.04 LTS

The output of "lxc info" or if that fails:

config: {}
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
addresses: []
architectures:
- x86_64
- i686
certificate: |
-----BEGIN CERTIFICATE-----
######
-----END CERTIFICATE-----
certificate_fingerprint: d2964d844ooir0f3dc965b6a48ad7fd3b8e01372f28ea6eb676bb6289087b0b1
driver: lxc
driver_version: 3.0.0
kernel: Linux
kernel_architecture: x86_64
kernel_version: 4.15.0-20-generic
server: lxd
server_pid: 14676
server_version: 3.0.0
storage: btrfs
storage_version: 4.15.1
server_clustered: false
server_name: emos1

Kernel version: 4.15.0-20-generic #21-Ubuntu SMP x86_64
LXC version: 3.0.0
LXD version: 3.0.0
Storage backend in use: btrfs

Issue description

I'm trying to run https://mailcow.email/ docker in lxd container and I get this error : ERROR: for netfilter-mailcow Cannot start service netfilter-mailcow: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:70: creating device nodes caused \\\"open /var/lib/docker/btrfs/subvolumes/77cc0dd8362cc94816957585d98e7ca50dfa2ebcfeceea2f7cb47ea79a7ee7e7/dev/tty: no such device or address\\\"\"": unknown

Steps to reproduce

lxc launch ubuntu:18.04 docker1 -c security.nesting=true Into docker1 container :
apt-get install docker.io (or install docker-ce from docker)
git clone https://github.com/mailcow/mailcow-dockerized
./generate_config.sh
docker-compose up

Information to attach

[ ] Container configuration (lxc config show NAME --expanded)

architecture: x86_64
config:
image.architecture: amd64
image.description: ubuntu 18.04 LTS amd64 (release) (20180426.2)
image.label: release
image.os: ubuntu
image.release: bionic
image.serial: "20180426.2"
image.version: "18.04"
linux.kernel_modules: bridge,br_netfilter,ip_tables,ip6_tables,ip_vs,netlink_diag,nf_nat,overlay,xt_conntrack
security.nesting: "true"
volatile.base_image: b36ec647e374da4816104a98807633a2cc387488083d3776557081c4d0333618
volatile.eth0.hwaddr: 00:16:3e:1b:5d:93
volatile.idmap.base: "0"
volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
volatile.last_state.power: RUNNING
devices:
eth0:
name: eth0
nictype: bridged
parent: lxdbr0
type: nic
root:
path: /
pool: default
type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

Output of docker-compose up : $ docker-compose up


Removing mailcowdockerized_dockerapi-mailcow_1
mailcowdockerized_clamd-mailcow_1 is up-to-date
mailcowdockerized_memcached-mailcow_1 is up-to-date
mailcowdockerized_sogo-mailcow_1 is up-to-date
Starting mailcowdockerized_ipv6nat_1 ... 
mailcowdockerized_mysql-mailcow_1 is up-to-date
mailcowdockerized_unbound-mailcow_1 is up-to-date
Recreating a80fe0eeb0b4_mailcowdockerized_dockerapi-mailcow_1 ... 
mailcowdockerized_watchdog-mailcow_1 is up-to-date
mailcowdockerized_redis-mailcow_1 is up-to-date
mailcowdockerized_postfix-mailcow_1 is up-to-date
mailcowdockerized_dovecot-mailcow_1 is up-to-date
mailcowdockerized_php-fpm-mailcow_1 is up-to-date
mailcowdockerized_nginx-mailcow_1 is up-to-date
Starting mailcowdockerized_ipv6nat_1                          ... error
mailcowdockerized_rspamd-mailcow_1 is up-to-date
mailcowdockerized_acme-mailcow_1 is up-to-date

ERROR: for mailcowdockerized_ipv6nat_1 Cannot start service ipv6nat: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:70: creating device nodes caused \\"open /var/lib/docStarting 86142ef4fcb0_mailcowdockerized_netfilter-mailcow_1 ... error

Recreating a80fe0eeb0b4_mailcowdockerized_dockerapi-mailcow_1 ... error odes caused \\"open /var/lib/docker/btrfs/subvolumes/77cc0dd8362cc94816957585d98e7ca50dfa2ebcfeceea2f7cb47ea79a7ee7e7/dev/tty: no such device or address\\"\"": unknown

ERROR: for a80fe0eeb0b4_mailcowdockerized_dockerapi-mailcow_1 Cannot start service dockerapi-mailcow: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:301: running exec setns process for init caused \"exit status 10\"": unknown

ERROR: for ipv6nat Cannot start service ipv6nat: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:70: creating device nodes caused \\"open /var/lib/docker/btrfs/subvolumes/7d47eb7631eb7d4fa8a9e675107d6c54ed4a03852c4ce988c354e73d48023afd/dev/tty: no such device or address\\"\"": unknown

ERROR: for netfilter-mailcow Cannot start service netfilter-mailcow: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:70: creating device nodes caused \\"open /var/lib/docker/btrfs/subvolumes/77cc0dd8362cc94816957585d98e7ca50dfa2ebcfeceea2f7cb47ea79a7ee7e7/dev/tty: no such device or address\\"\"": unknown

ERROR: for dockerapi-mailcow Cannot start service dockerapi-mailcow: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:301: running exec setns process for init caused \"exit status 10\"": unknown ERROR: Encountered errors while bringing up the project.

stgraber commented 6 years ago

That's most likely because of the usual kernel pts weirdness. Try ssh-ing into your container or get a separate pts using "script /dev/null /bin/bash", that may help there.

dokime7 commented 6 years ago

I don't well understand, but I'm already connecting by ssh on LXD container when I run docker-compose up. From the LXD container docker1 there is no tty file in /var/lib/docker/btrfs/subvolumes/77cc0dd8362cc94816957585d98e7ca50dfa2ebcfeceea2f7cb47ea79a7ee7e7/dev/

root@docker1:~# ls /var/lib/docker/btrfs/subvolumes/77cc0dd8362cc94816957585d98e7ca50dfa2ebcfeceea2f7cb47ea79a7ee7e7/dev
console  pts  shm

But if I'm entering on a running docker container there is the tty file...

root@docker1:~/mailcow-dockerized# docker-compose exec clamd-mailcow bash 
bash-4.3# ls /dev
console  fd       mqueue   ptmx     random   stderr   stdout   urandom
core     full     null     pts      shm      stdin    tty      zero
bash-4.3#

Obviously the mailcow-dockerized work well on docker outside of LXD container.

stgraber commented 6 years ago

@brauner thoughts?

brauner commented 6 years ago

Will take a look in a few.

dokime7 commented 6 years ago

Thank you very much!

brauner commented 6 years ago

I'm trying to reproduce here.

brauner commented 6 years ago

I don't know what this docker-compose thing is doing but it's building stuff for ages now.

dokime7 commented 6 years ago

Yes it takes a while...

brauner commented 6 years ago

@dokime7 so said that it works on the host. Does your host run on btrfs as well?

dokime7 commented 6 years ago

It's on another server that is on ext4

dokime7 commented 6 years ago

I could maybe create a ZFS partition in order to test with...

brauner commented 6 years ago

@dokime7 would be fantastic if you could!

dokime7 commented 6 years ago

OK I will do it. Do you think there is a connection with btrfs?

brauner commented 6 years ago

Not completely sure but it could be.

brauner commented 6 years ago

The fact that you get ENODEV sounds very much like btrfs weirdness because what runC itself is trying to do is to simply create a dummy file which will serve as the target of a bind-mount of /dev/tty but it somehow can't create that file. This makes me suspect that there's some weirdness going on with btrfs itself.

dokime7 commented 6 years ago

It's curious because other docker containers start well (on mailcow) and we found /dev/tty inside.

dokime7 commented 6 years ago

So with ZFS, same error :

root@docker1:~/mailcow-dockerized# docker-compose up -d
WARNING: The WATCHDOG_NOTIFY_EMAIL variable is not set. Defaulting to a blank string.
mailcowdockerized_mysql-mailcow_1 is up-to-date
mailcowdockerized_sogo-mailcow_1 is up-to-date
Starting mailcowdockerized_ipv6nat_1 ... 
mailcowdockerized_unbound-mailcow_1 is up-to-date
Starting mailcowdockerized_dockerapi-mailcow_1 ... 
mailcowdockerized_clamd-mailcow_1 is up-to-date
mailcowdockerized_redis-mailcow_1 is up-to-date
mailcowdockerized_watchdog-mailcow_1 is up-to-date
mailcowdockerized_dovecot-mailcow_1 is up-to-date
mailcowdockerized_memcached-mailcow_1 is up-to-date
mailcowdockerized_postfix-mailcow_1 is up-to-date
mailcowdockerized_php-fpm-mailcow_1 is up-to-date
mailcowdockerized_nginx-mailcow_1 is up-to-date
Starting mailcowdockerized_ipv6nat_1           ... error
mailcowdockerized_rspamd-mailcow_1 is up-to-date
Creating mailcowdockerized_netfilter-mailcow_1 ... 

Starting mailcowdockerized_dockerapi-mailcow_1 ... error
ker/vfs/dir/e651ce79444c1ed6acd3148a76bc157187d69f432ae9517218382e1e6c5323de/dev/tty: no such device or address\\\"\"": unknown

Creating mailcowdockerized_netfilter-mailcow_1 ... error

ERROR: for mailcowdockerized_netfilter-mailcow_1  Cannot start service netfilter-mailcow: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:70: creating device nodes caused \\\"open /var/lib/docker/vfs/dir/11c54a8152e8e632ad4c767768a4dbad3cc601cba3895a7b2f42eedd2c82e23b/dev/tty: no such device or address\\\"\"": unknown

ERROR: for ipv6nat  Cannot start service ipv6nat: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:70: creating device nodes caused \\\"open /var/lib/docker/vfs/dir/e651ce79444c1ed6acd3148a76bc157187d69f432ae9517218382e1e6c5323de/dev/tty: no such device or address\\\"\"": unknown

ERROR: for dockerapi-mailcow  Cannot start service dockerapi-mailcow: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:301: running exec setns process for init caused \"exit status 10\"": unknown

ERROR: for netfilter-mailcow  Cannot start service netfilter-mailcow: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:70: creating device nodes caused \\\"open /var/lib/docker/vfs/dir/11c54a8152e8e632ad4c767768a4dbad3cc601cba3895a7b2f42eedd2c82e23b/dev/tty: no such device or address\\\"\"": unknown
ERROR: Encountered errors while bringing up the project.

brauner commented 6 years ago

Hm, but works fine in a standard docker container so this is mighty weird.

brauner commented 6 years ago

What's confusing me is that the ENODEV would also only happen if there'd be no controlling terminal.

brauner commented 6 years ago

Please attach to the container and show me the output of:

ls -al /proc/self/fd/

dokime7 commented 6 years ago

From LXD container :

root@docker1:~# ls -al /proc/self/fd/
total 0
dr-x------ 2 root root  0 May  4 15:04 .
dr-xr-xr-x 9 root root  0 May  4 15:04 ..
lrwx------ 1 root root 64 May  4 15:04 0 -> /dev/pts/0
lrwx------ 1 root root 64 May  4 15:04 1 -> /dev/pts/0
lrwx------ 1 root root 64 May  4 15:04 2 -> /dev/pts/0
lr-x------ 1 root root 64 May  4 15:04 3 -> /proc/12289/fd

dokime7 commented 6 years ago

I don't know why a mailcow docker container like clamd starts normaly and have /dev/tty

root@docker1:~/mailcow-dockerized# docker-compose exec clamd-mailcow ls -l /dev
WARNING: The WATCHDOG_NOTIFY_EMAIL variable is not set. Defaulting to a blank string.
total 0
crw-rw----    1 root     tty       136,   0 May  4 13:01 console
lrwxrwxrwx    1 root     root            11 May  3 22:53 core -> /proc/kcore
lrwxrwxrwx    1 root     root            13 May  3 22:53 fd -> /proc/self/fd
crw-rw-rw-    1 nobody   nobody      1,   7 May  3 22:51 full
drwxrwxrwt    2 root     root            40 May  3 22:53 mqueue
crw-rw-rw-    1 nobody   nobody      1,   3 May  3 22:51 null
lrwxrwxrwx    1 root     root             8 May  3 22:53 ptmx -> pts/ptmx
drwxr-xr-x    2 root     root             0 May  3 22:53 pts
crw-rw-rw-    1 nobody   nobody      1,   8 May  3 22:51 random
drwxrwxrwt    2 root     root            40 May  3 22:53 shm
lrwxrwxrwx    1 root     root            15 May  3 22:53 stderr -> /proc/self/fd/2
lrwxrwxrwx    1 root     root            15 May  3 22:53 stdin -> /proc/self/fd/0
lrwxrwxrwx    1 root     root            15 May  3 22:53 stdout -> /proc/self/fd/1
crw-rw-rw-    1 nobody   nobody      5,   0 May  4 13:04 tty
crw-rw-rw-    1 nobody   nobody      1,   9 May  3 22:51 urandom
crw-rw-rw-    1 nobody   nobody      1,   5 May  3 22:51 zero

but netfilter container does not...

dokime7 commented 6 years ago

Two of the three containers that won't start have the "privileged: true" on docker-compose.yml file.

dokime7 commented 6 years ago

OK, so when I comment out the "privileged: true" from netfilter service on docker-compose.yml file, it starts!

brauner commented 6 years ago

Two of the three containers that won't start have the "privileged: true" on docker-compose.yml file.

Yeah, that explains it! I suspect that in this case they might try to create actual device nodes instead of bind-mounting them. It's a very misleading error though.

dokime7 commented 6 years ago

Yes, you can reproduce easily by run: docker run --privileged hello-world

brauner commented 6 years ago

Yeah, they should either just refuse to run privileged when in user namespace or adhere to the user namespace restrictions. In any case, nothing that LXD itself can do so closing if you don't mind. :)

dokime7 commented 6 years ago

Hmm so it's not possible to run privileged docker container in a not privileged LXD container, it seems obvious!

dokime7 commented 6 years ago

And for the last one container that fail to start with this error : ERROR: for dockerapi-mailcow Cannot start service dockerapi-mailcow: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:301: running exec setns process for init caused \"exit status 10\"": unknown Do you think that is the same problem?

brauner commented 6 years ago

On Fri, May 04, 2018 at 06:55:08AM -0700, Jeremie Dokime wrote:

And for the last one container that fail to start with this error : ERROR: for dockerapi-mailcow Cannot start service dockerapi-mailcow: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:301: running exec setns process for init caused \"exit status 10\"": unknown Do you think that is the same problem?

Is it a privileged container too?

dokime7 commented 6 years ago

No privileged

brauner commented 6 years ago

Can you show me the configuration for that specific container?

canonical / lxd

/dev/tty: no such device or address #4531

Required information

Issue description

Steps to reproduce

Information to attach