search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.7k stars 2.41k forks source link

failed to mount runtime directory for rootless netns: no such file or directory #13671

Closed ThanatosDi closed 2 years ago

ThanatosDi commented 2 years ago

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

bug

Description

From docker to Podman to develop with Laradock after build, want up containers will get error failed to mount runtime directory for rootless netns: no such file or directory

Steps to reproduce the issue:

  1. build nginx, php-fpm(8.0), workspace with Laradock

  2. podman-compose up -d nginx php-fpm workspace

Describe the results you received:

podman-compose version: 1.0.4
['podman', '--version', '']
using podman version: 4.0.2
** excluding:  {'minio', 'mailu-antivirus', 'proxy2', 'thumbor', 'ipython-engine', 'docker-registry', 'traefik', 'docker-web-ui', 'mailhog', 'hhvm', 'portainer', 'beanstalkd', 'certbot', 'rethinkdb', 'confluence', 'meilisearch', 'percona', 'proxy', 'haproxy', 'mailu-webdav', 'pgadmin', 'gitlab-runner', 'ide-webide', 'manticore', 'redis', 'metabase', 'couchdb', 'sonarqube', 'elasticsearch', 'graylog', 'apache2', 'gearman', 'laravel-echo-server', 'mosquitto', 'clickhouse', 'swagger-ui', 'mailcatcher', 'netdata', 'ide-icecoder', 'adminer', 'mongo', 'mailu-antispam', 'react', 'weaver', 'zookeeper', 'caddy', 'jenkins', 'swagger-editor', 'mercure', 'postgres', 'memcached', 'jupyterhub-user', 'blackfire', 'gitlab', 'ide-codiad', 'aerospike', 'postgres-postgis', 'logstash', 'aws', 'redis-webui', 'phpmyadmin', 'mailu-imap', 'mailu-front', 'redis-cluster', 'mailu-fetchmail', 'kibana', 'maildev', 'mailu', 'mailu-webmail', 'dejavu', 'grafana', 'mysql', 'ide-theia', 'mariadb', 'tomcat', 'laravel-horizon', 'jupyterhub', 'mailu-smtp', 'php-worker', 'cassandra', 'mssql', 'kafka-manager', 'sqs', 'solr', 'mailu-admin', 'rabbitmq', 'mongo-webui', 'neo4j', 'kafka', 'ipython-controller', 'beanstalkd-console', 'selenium'}
['podman', 'inspect', '-t', 'image', '-f', '{{.Id}}', 'laradock_nginx']
['podman', 'ps', '--filter', 'label=io.podman.compose.project=laradock', '-a', '--format', '{{ index .Labels "io.podman.compose.config-hash"}}']
podman pod create --name=pod_laradock --infra=false --share=
1d02a18cf61bb710a50a09a0fef2816e925fda1ea908eccee8fda0371482fc63
exit code: 0
** skipping:  laradock_minio_1
** skipping:  laradock_mysql_1
** skipping:  laradock_percona_1
** skipping:  laradock_mssql_1
** skipping:  laradock_mariadb_1
** skipping:  laradock_postgres_1
** skipping:  laradock_postgres-postgis_1
** skipping:  laradock_neo4j_1
** skipping:  laradock_mongo_1
** skipping:  laradock_rethinkdb_1
** skipping:  laradock_redis_1
** skipping:  laradock_redis-cluster_1
** skipping:  laradock_zookeeper_1
** skipping:  laradock_aerospike_1
** skipping:  laradock_sqs_1
** skipping:  laradock_mercure_1
** skipping:  laradock_meilisearch_1
** skipping:  laradock_certbot_1
** skipping:  laradock_mailcatcher_1
** skipping:  laradock_mailhog_1
** skipping:  laradock_maildev_1
** skipping:  laradock_selenium_1
** skipping:  laradock_jenkins_1
** skipping:  laradock_grafana_1
** skipping:  laradock_solr_1
** skipping:  laradock_thumbor_1
** skipping:  laradock_portainer_1
** skipping:  laradock_gitlab-runner_1
** skipping:  laradock_jupyterhub-user_1
** skipping:  laradock_ipython-controller_1
** skipping:  laradock_ipython-engine_1
podman volume inspect laradock_docker-in-docker || podman volume create laradock_docker-in-docker
['podman', 'volume', 'inspect', 'laradock_docker-in-docker']
['podman', 'network', 'exists', 'laradock_backend']
podman run --name=laradock_docker-in-docker_1 -d --pod=pod_laradock --label io.podman.compose.config-hash=966d890b9e136d0938d206925da7a196d3fc6a0d584623752a50d4683c4d5af5 --label io.podman.compose.project=laradock --label io.podman.compose.version=1.0.4 --label com.docker.compose.project=laradock --label com.docker.compose.project.working_dir=/home/thanatosdi/Desktop/task/laradock --label com.docker.compose.project.config_files=docker-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=docker-in-docker -e DOCKER_TLS_SAN=DNS:docker-in-docker -v /home/thanatosdi/Desktop/task:/var/www -v laradock_docker-in-docker:/certs/client --net laradock_backend --network-alias docker-in-docker --expose 2375 --privileged docker:19.03-dind
Error: failed to mount runtime directory for rootless netns: no such file or directory
exit code: 127
podman start laradock_docker-in-docker_1
Error: unable to start container "7f9a6a15ad861e33dc7115430906c7bd83bd0b31ded6733bc902dd6eea54fbaf": failed to mount runtime directory for rootless netns: no such file or directory
exit code: 125
** skipping:  laradock_netdata_1
** skipping:  laradock_metabase_1
** skipping:  laradock_ide-theia_1
** skipping:  laradock_ide-webide_1
** skipping:  laradock_ide-codiad_1
** skipping:  laradock_ide-icecoder_1
** skipping:  laradock_docker-registry_1
** skipping:  laradock_docker-web-ui_1
** skipping:  laradock_mailu-front_1
** skipping:  laradock_mailu-antivirus_1
** skipping:  laradock_mailu-webdav_1
** skipping:  laradock_mailu-webmail_1
** skipping:  laradock_mailu-fetchmail_1
** skipping:  laradock_traefik_1
** skipping:  laradock_mosquitto_1
** skipping:  laradock_couchdb_1
** skipping:  laradock_manticore_1
** skipping:  laradock_swagger-editor_1
** skipping:  laradock_swagger-ui_1
** skipping:  tomcat
** skipping:  react
** skipping:  laradock_kafka_1
** skipping:  laradock_kafka-manager_1
podman volume inspect laradock_docker-in-docker || podman volume create laradock_docker-in-docker
['podman', 'volume', 'inspect', 'laradock_docker-in-docker']
['podman', 'network', 'exists', 'laradock_frontend']
['podman', 'network', 'exists', 'laradock_backend']
podman run --name=laradock_workspace_1 -d --pod=pod_laradock --requires=laradock_docker-in-docker_1 --label io.podman.compose.config-hash=966d890b9e136d0938d206925da7a196d3fc6a0d584623752a50d4683c4d5af5 --label io.podman.compose.project=laradock --label io.podman.compose.version=1.0.4 --label com.docker.compose.project=laradock --label com.docker.compose.project.working_dir=/home/thanatosdi/Desktop/task/laradock --label com.docker.compose.project.config_files=docker-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=workspace -e PHP_IDE_CONFIG=serverName=laradock -e DOCKER_HOST=tcp://docker-in-docker:2376 -e DOCKER_TLS_VERIFY=1 -e DOCKER_TLS_CERTDIR=/certs -e DOCKER_CERT_PATH=/certs/client -v /home/thanatosdi/Desktop/task:/var/www -v laradock_docker-in-docker:/certs/client -v /home/thanatosdi/Desktop/task/laradock/php-worker/supervisord.d:/etc/supervisord.d --net laradock_backend,laradock_frontend --network-alias workspace --add-host dockerhost:10.0.75.1 -p 50010:22 -p 50009:3000 -p 50008:3001 -p 50005:8080 -p 50006:8000 -p 50007:4200 --tty laradock_workspace
ERRO[0001] Starting some container dependencies         
ERRO[0001] "failed to mount runtime directory for rootless netns: no such file or directory" 
Error: error starting some containers: internal libpod error
exit code: 126
podman start laradock_workspace_1
ERRO[0000] Starting some container dependencies         
ERRO[0000] "failed to mount runtime directory for rootless netns: no such file or directory" 
Error: unable to start container "384411aa06cbac44832ab8d658ba78b81031d66a13a692407b203eb7750fa6be": error starting some containers: internal libpod error
exit code: 125
** skipping:  laradock_phpmyadmin_1
** skipping:  laradock_pgadmin_1
** skipping:  laradock_laravel-echo-server_1
** skipping:  laradock_redis-webui_1
** skipping:  laradock_mongo-webui_1
** skipping:  laradock_mailu-imap_1
** skipping:  laradock_mailu-smtp_1
** skipping:  laradock_mailu-antispam_1
** skipping:  laradock_mailu-admin_1
** skipping:  laradock_sonarqube_1
** skipping:  Confluence
podman volume inspect laradock_docker-in-docker || podman volume create laradock_docker-in-docker
['podman', 'volume', 'inspect', 'laradock_docker-in-docker']
['podman', 'network', 'exists', 'laradock_backend']
podman run --name=laradock_php-fpm_1 -d --pod=pod_laradock --requires=laradock_docker-in-docker_1,laradock_workspace_1 --label io.podman.compose.config-hash=966d890b9e136d0938d206925da7a196d3fc6a0d584623752a50d4683c4d5af5 --label io.podman.compose.project=laradock --label io.podman.compose.version=1.0.4 --label com.docker.compose.project=laradock --label com.docker.compose.project.working_dir=/home/thanatosdi/Desktop/task/laradock --label com.docker.compose.project.config_files=docker-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=php-fpm -e PHP_IDE_CONFIG=serverName=laradock -e DOCKER_HOST=tcp://docker-in-docker:2376 -e DOCKER_TLS_VERIFY=1 -e DOCKER_TLS_CERTDIR=/certs -e DOCKER_CERT_PATH=/certs/client -e FAKETIME=-0 -v /home/thanatosdi/Desktop/task/laradock/php-fpm/php8.0.ini:/usr/local/etc/php/php.ini -v /home/thanatosdi/Desktop/task:/var/www -v laradock_docker-in-docker:/certs/client --net laradock_backend --network-alias php-fpm --add-host dockerhost:10.0.75.1 --expose 9000 -p 50012:9003 laradock_php-fpm
ERRO[0000] Starting some container dependencies         
ERRO[0000] "failed to mount runtime directory for rootless netns: no such file or directory" 
ERRO[0000] "a dependency of container 384411aa06cbac44832ab8d658ba78b81031d66a13a692407b203eb7750fa6be failed to start: container state improper" 
Error: error starting some containers: internal libpod error
exit code: 126
podman start laradock_php-fpm_1
ERRO[0000] Starting some container dependencies         
ERRO[0000] "failed to mount runtime directory for rootless netns: no such file or directory" 
ERRO[0000] "a dependency of container 384411aa06cbac44832ab8d658ba78b81031d66a13a692407b203eb7750fa6be failed to start: container state improper" 
Error: unable to start container "b7ff3eaad68c31e912c4d00521016971509c69779ed12e76c6c3a69af786926e": error starting some containers: internal libpod error
exit code: 125
** skipping:  laradock_php-worker_1
** skipping:  laradock_laravel-horizon_1
** skipping:  laradock_hhvm_1
** skipping:  laradock_clickhouse_1
** skipping:  proxy
** skipping:  proxy2
** skipping:  laradock_aws_1
** skipping:  laradock_gitlab_1
** skipping:  laradock_jupyterhub_1
** skipping:  laradock_weaver_1
['podman', 'network', 'exists', 'laradock_frontend']
['podman', 'network', 'exists', 'laradock_backend']
podman run --name=laradock_nginx_1 -d --pod=pod_laradock --requires=laradock_docker-in-docker_1,laradock_php-fpm_1,laradock_workspace_1 --label io.podman.compose.config-hash=966d890b9e136d0938d206925da7a196d3fc6a0d584623752a50d4683c4d5af5 --label io.podman.compose.project=laradock --label io.podman.compose.version=1.0.4 --label com.docker.compose.project=laradock --label com.docker.compose.project.working_dir=/home/thanatosdi/Desktop/task/laradock --label com.docker.compose.project.config_files=docker-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=nginx -v /home/thanatosdi/Desktop/task:/var/www -v /home/thanatosdi/Desktop/task/laradock/logs/nginx:/var/log/nginx -v /home/thanatosdi/Desktop/task/laradock/nginx/sites:/etc/nginx/sites-available -v /home/thanatosdi/Desktop/task/laradock/nginx/ssl:/etc/nginx/ssl --net laradock_backend,laradock_frontend --network-alias nginx,dev.task.tw,dev.task.tw -p 50000:80 -p 50001:443 -p 50011:81 laradock_nginx
ERRO[0000] Starting some container dependencies         
ERRO[0000] "a dependency of container b7ff3eaad68c31e912c4d00521016971509c69779ed12e76c6c3a69af786926e failed to start: container state improper" 
ERRO[0000] "a dependency of container 384411aa06cbac44832ab8d658ba78b81031d66a13a692407b203eb7750fa6be failed to start: container state improper" 
ERRO[0000] "failed to mount runtime directory for rootless netns: no such file or directory" 
Error: error starting some containers: internal libpod error
exit code: 126
podman start laradock_nginx_1
ERRO[0000] Starting some container dependencies         
ERRO[0000] "failed to mount runtime directory for rootless netns: no such file or directory" 
ERRO[0000] "a dependency of container b7ff3eaad68c31e912c4d00521016971509c69779ed12e76c6c3a69af786926e failed to start: container state improper" 
ERRO[0000] "a dependency of container 384411aa06cbac44832ab8d658ba78b81031d66a13a692407b203eb7750fa6be failed to start: container state improper" 
Error: unable to start container "e1230c499e4b78adaaa29678e14323b7863957d135003fdec44f79ae969b3398": error starting some containers: internal libpod error
exit code: 125
** skipping:  laradock_blackfire_1
** skipping:  laradock_apache2_1
** skipping:  laradock_memcached_1
** skipping:  laradock_beanstalkd_1
** skipping:  laradock_rabbitmq_1
** skipping:  laradock_cassandra_1
** skipping:  laradock_gearman_1
** skipping:  laradock_caddy_1
** skipping:  laradock_adminer_1
** skipping:  laradock_elasticsearch_1
** skipping:  laradock_beanstalkd-console_1
** skipping:  laradock_logstash_1
** skipping:  laradock_kibana_1
** skipping:  laradock_dejavu_1
** skipping:  laradock_haproxy_1
** skipping:  laradock_graylog_1
** skipping:  laradock_mailu_1

Describe the results you expected: start containers

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Client:       Podman Engine
Version:      4.0.2
API Version:  4.0.2
Go Version:   go1.18

Built:      Thu Jan  1 08:00:00 1970
OS/Arch:    linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: bdb4f6e56cd193d40b75ffc9725d4b74a18cb33c'
  cpus: 8
  distribution:
    codename: focal
    distribution: ubuntu
    version: "20.04"
  eventLogger: file
  hostname: ubuntu-1
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.13.0-37-generic
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 1688694784
  memTotal: 4115079168
  networkBackend: cni
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 1.4.4
      commit: 6521fcc5806f20f6187eb933f9f45130c86da230
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 1916227584
  swapTotal: 2147479552
  uptime: 7h 33m 26.79s (Approximately 0.29 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/thanatosdi/.config/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 0
    stopped: 4
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/thanatosdi/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 492
  runRoot: /run/user/1000/containers
  volumePath: /home/thanatosdi/.local/share/containers/storage/volumes
version:
  APIVersion: 4.0.2
  Built: 0
  BuiltTime: Thu Jan  1 08:00:00 1970
  GitCommit: ""
  GoVersion: go1.18
  OsArch: linux/amd64
  Version: 4.0.2

Package info (e.g. output of rpm -q podman or apt list podman):

podman/unknown,now 100:4.0.2-1 amd64 [installed]

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.): VirtualBox and system is Ubuntu 20.04

Luap99 commented 2 years ago

Do you have slirp4netns installed?

ThanatosDi commented 2 years ago

oh, yes! I don't have install slirp4netns. This is my first time to use podman, thanks~

fraschm1998 commented 2 years ago

I have slirp4netns and I'm having the same problem using Gentoo linux with openrc: Podman failed to mount runtime directory for rootless netns: no such file or directory. I'm not sure if it's an issue with SELinux: https://dpaste.com/ANRM893KY podman info logs: https://dpaste.com/CBPETY4TV

I found where the error message is printed (lines 128-138): https://fossies.org/linux/podman/libpod/networking_linux.go

echo $XDG_RUNTIME_DIR prints: /run/user/1000

Edit according to lines 114-119 it seems as tho I need to have the following binds mounted:

// The following bind mounts are needed
// 1. XDG_RUNTIME_DIR -> XDG_RUNTIME_DIR/rootless-netns/XDG_RUNTIME_DIR
// 2. /run/systemd -> XDG_RUNTIME_DIR/rootless-netns/run/systemd (only if it exists)
// 3. XDG_RUNTIME_DIR/rootless-netns/resolv.conf -> /etc/resolv.conf or XDG_RUNTIME_DIR/rootless-netns/run/symlink/target
// 4. XDG_RUNTIME_DIR/rootless-netns/var/lib/cni -> /var/lib/cni (if /var/lib/cni does not exists use the parent dir)
// 5. XDG_RUNTIME_DIR/rootless-netns/run -> /run

However in /run/user/1000 the only file related to netns is /run/user/1000/netns/rootless-netns-*

fd rootless in /

root@asus-g14 / # fd rootless
home/massimo/podman/cni/rootless-cni-infra
run/user/1000/libpod/tmp/rootless-netns.lock
run/user/1000/libpod/tmp/rootless-netns
run/user/1000/netns/rootless-netns-987c9a1aa493ae43558a
dev/shm/libpod_rootless_lock_1000
run/user/1000/libpod/tmp/rootless-netns/rootless-netns-slirp4netns.pid
var/db/repos/gentoo/metadata/md5-cache/sys-apps/rootlesskit-0.14.2
var/db/repos/gentoo/sys-apps/rootlesskit
var/db/repos/gentoo/sys-apps/rootlesskit/rootlesskit-0.14.2.ebuild
usr/share/doc/containerd-1.5.11/rootless.md.bz2
usr/share/selinux/targeted/include/services/rootlesskit.if
usr/share/selinux/strict/include/services/rootlesskit.if
usr/libexec/podman/rootlessport

root@asus-g14 / # fd rootless-netns
run/user/1000/libpod/tmp/rootless-netns.lock
run/user/1000/libpod/tmp/rootless-netns
run/user/1000/libpod/tmp/rootless-netns/rootless-netns-slirp4netns.pid
run/user/1000/netns/rootless-netns-987c9a1aa493ae43558a
root@asus-g14 / # v /run/user/1000/libpod/tmp/rootless-netns/resolv.conf
fraschm1998 commented 2 years ago

Edit I tried starting up the container again and get this:

ERRO[0000] failed to move the rootless netns slirp4netns process to the systemd user.slice: The name org.freedesktop.systemd1 was not provided by any .service files
Error: unable to start container 159d3d6111ff17c9c33108d37a547fb382e4909300ee07873f842ae2ac505fb7: could not create relabel rootless-netns run directory: setxattr /run/user/1000/libpod/tmp/
rootless-netns/run: invalid argument
Luap99 commented 2 years ago

@fraschm1998 Please create a new issue for this