Windows: Starting container in pod that publishes ports prints `Error: starting some containers: internal libpod error`

jd-boyd commented 2 months ago

Issue Description

On Windows 11 Pro running a container in a pod that publishes a port prints the error Error: starting some containers: internal libpod error.

While the steps to reproduce below use a postgres container, I get the same result for any other container I tried (minio, redis, and a custom built one based on python:3.12-slim-bookworm.

Steps to reproduce the issue

podman pod create -p 5672:5672 -n test_pod
podman run -d -e POSTGRES_USER=test_user -e "POSTGRES_PASSWORD=test_pass" -e POSTGRES_DB=test_db --pod test_pod --name pg_test docker.io/postgres:16

Describe the results you received

Error: starting some containers: internal libpod error Also, podman ps does not show the container running.

Describe the results you expected

The postgres container to be running, named pg_test, and to be able to connect to it.

podman info output

PS C:\Users\jdboyd> podman info
host:
  arch: amd64
  buildahVersion: 1.35.4
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.10-1.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 99.55
    systemPercent: 0.23
    userPercent: 0.21
  cpus: 16
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: container
    version: "40"
  eventLogger: journald
  freeLocks: 2044
  hostname: JDB-Z4G4-Win
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.15.146.1-microsoft-standard-WSL2
  linkmode: dynamic
  logDriver: journald
  memFree: 28736065536
  memTotal: 33548562432
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-3.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.15-1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.15
      commit: e6eacaf4034e84185fd8780ac9262bbf57082278
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240510.g7288448-1.fc40.x86_64
    version: |
      pasta 0^20240510.g7288448-1.fc40.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootlessNetworkCmd: ""
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: true
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 8589934592
  swapTotal: 8589934592
  uptime: 49h 1m 6.00s (Approximately 2.04 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /usr/lib/containers/storage
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 1081101176832
  graphRootUsed: 1146806272
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.0.3
  Built: 1715299200
  BuiltTime: Thu May  9 20:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.2
  Os: linux
  OsArch: linux/amd64
  Version: 5.0.3

PS C:\Users\jdboyd>

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

While I selected Privileged above, I originally ran into this while trying to use rootless, so I don't think this setting effects the issue.

This is running on Windows 11 Pro, systeminfo shows the version as:

OS Version:                10.0.22631 N/A Build 22631

Additional information

If I don't use the -p on podman pod create, then the postgres container starts correctly, but I can't connect to it.

If I skip the pod and run podman run -d -e POSTGRES_USER=test_user -e "POSTGRES_PASSWORD=test_pass" -e POSTGRES_DB=test_db -p 5672:5672 --name pg_test docker.io/postgres:16, the container runs as expected.

Luap99 commented 2 months ago

Can you run the same command in the VM, i.e. use podman machine ssh, this should print a better error I think.

If it works without -p then it most likely means the port is already in use.

jd-boyd commented 2 months ago

Yep, that does print a better error:

[root@JDB-Z4G4-Win ~]# podman pod create -p 5672:5672 -n test_pod
WARN[0000] Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning.
290d50c4c2c076ee2609f995579c85593639aaf1a0b94ccc5a500011262fab35
[root@JDB-Z4G4-Win ~]# podman run -d -e POSTGRES_USER=test_user -e "POSTGRES_PASSWORD=test_pass" -e POSTGRES_DB=test_db --pod test_pod --name pg_test docker.io/postgres:16
WARN[0000] Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning.
ERRO[0000] Starting some container dependencies
ERRO[0000] "cannot listen on the TCP port: listen tcp4 :5672: bind: address already in use"
Error: starting some containers: internal libpod error
[root@JDB-Z4G4-Win ~]#

Netstat shows this:

tcp6       0      0 :::5672                 :::*                    LISTEN      -

So the port I'm trying to use is being listened to by the kernel for some reason. This is the first time I've ssh'd into the VM though, and I don't have any pods or containers running currently, so why would something be listening to that port?

jd-boyd commented 2 months ago

Also, should a bug be filed about the better error not being shown by the windows podman command?

Luap99 commented 2 months ago

Yep, that does print a better error:

[root@JDB-Z4G4-Win ~]# podman pod create -p 5672:5672 -n test_pod
WARN[0000] Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning.
290d50c4c2c076ee2609f995579c85593639aaf1a0b94ccc5a500011262fab35
[root@JDB-Z4G4-Win ~]# podman run -d -e POSTGRES_USER=test_user -e "POSTGRES_PASSWORD=test_pass" -e POSTGRES_DB=test_db --pod test_pod --name pg_test docker.io/postgres:16
WARN[0000] Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning.
ERRO[0000] Starting some container dependencies
ERRO[0000] "cannot listen on the TCP port: listen tcp4 :5672: bind: address already in use"
Error: starting some containers: internal libpod error
[root@JDB-Z4G4-Win ~]#

Netstat shows this:

tcp6       0      0 :::5672                 :::*                    LISTEN      -

So the port I'm trying to use is being listened to by the kernel for some reason. This is the first time I've ssh'd into the VM though, and I don't have any pods or containers running currently, so why would something be listening to that port?

use ss -tlpn to list the ports with the processes

Luap99 commented 2 months ago

Also, should a bug be filed about the better error not being shown by the windows podman command?

yes sure

baude commented 2 months ago

@n1hility could this be a case where the socket is being set aside as a group for podman machine?

n1hility commented 2 months ago

if you don't see a process associated with it this likely means the process is in another pid namespace, which most likely means its coming from another wsl instance. My recommendation would be to stop podman and shutdown wsl, then restart podman:

podman machine stop
Start-Sleep 5
wsl --shutdown
Start-Sleep 5
podman machine start

github-actions[bot] commented 1 month ago

A friendly reminder that this issue had no activity for 30 days.

jd-boyd commented 1 month ago

After some rounds of Windows updates, I can't make it fail again on either test system, so I'm going to go ahead and close this. Thank you for the help. At least I learned somethings.

containers / podman