Regression in new podman machine images, machine does not start correctly

[POnakS]

Issue Description

After update to new machine images, machine does not start correctly and it's not possible to connect via ssh or use docker socket. Reproduces also for machine coming with podman 5.1.1 by default (I am not sure which machine is it).

Steps to reproduce the issue

Steps to reproduce the issue

1. Configure WSL to use mirrored mode
2. Install latest podman desktop (v1.10.3) and latest podman (v5.1.1)
3. Create machine via podman desktop or cli
4. call podman machine start

Describe the results you received

Starting machine "podman-machine-default"
API forwarding for Docker API clients is not available due to the following startup failures.
        CreateFile \\.\pipe\docker_engine: All pipe instances are busy.

Podman clients are still able to connect.
Error: machine did not transition into running state: ssh error: machine is not listening on ssh port

Machine is listed as running when I do podman machine list podman machine ssh times out. Docker socket is not available.

Describe the results you expected

Podman machine starts and socket is available, I can ssh into the machine.

podman info output

host:
    arch: amd64
    currentmachine: podman-machine-default
    defaultmachine: podman-machine-default
    eventsdir: C:\Users\redacted\.local\share\containers\podman\podman
    machineconfigdir: C:\Users\redacted\.config\containers\podman\machine\wsl
    machineimagedir: C:\Users\redacted\.local\share\containers\podman\machine\wsl
    machinestate: Running
    numberofmachines: 1
    os: windows
    vmtype: wsl
version:
    apiversion: 5.1.1
    version: 5.1.1
    goversion: go1.22.3
    gitcommit: bda6eb03dcbcf12a5b7ae004c1240e38dd056d24
    builttime: Tue Jun  4 23:40:05 2024
    built: 1717537205
    osarch: windows/amd64
    os: windows

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

I've tried multiple machines:

• Default for podman v5.1.1 does not work
• v20240614061401 - does not work
• v20240417162145 - newest one which works

Additional information

Maybe related to however these issues may have different root causes: https://github.com/containers/podman/issues/22975 https://github.com/containers/podman/issues/22240

[taoyouh]

I see this problem too, but I suggest you give a better title and format your content.

[n1hility]

I am not able to reproduce locally. Can you run the following:

wsl -d podman-machine-default -u root
ss -tln
systemctl --failed 

As well as (on powershell prompt)

Get-Content $HOME/.wslconfig

and also

podman machine inspect

[POnakS]

I see this problem too

Do you use VPN? Have you changed WSL subnets?

I think it should not be connected, because it happens even when my VPN is disconnected, however who knows :/

[POnakS]

PS C:\> wsl -d podman-machine-default -u root

You will be automatically entered into a nested process namespace where
systemd is running. If you need to access the parent namespace, hit ctrl-d
or type exit. This also means to log out you need to exit twice.

[root@D7C8HW3 c]# ss -tln
State          Recv-Q          Send-Q                   Local Address:Port                    Peer Address:Port
LISTEN         0               128                            0.0.0.0:50625                        0.0.0.0:*
LISTEN         0               128                               [::]:50625                           [::]:*
[root@D7C8HW3 c]# systemctl --failed
  UNIT                   LOAD   ACTIVE SUB    DESCRIPTION
● systemd-binfmt.service loaded failed failed Set Up Additional Binary Formats

Legend: LOAD   → Reflects whether the unit definition was properly loaded.
        ACTIVE → The high-level unit activation state, i.e. generalization of SUB.
        SUB    → The low-level unit activation state, values depend on unit type.

wslconfig

PS C:\> Get-Content $HOME/.wslconfig
[wsl2]
networkingMode=mirrored
#dnsTunneling=true
#autoProxy=true
#firewall=true

inspect

[
     {
          "ConfigDir": {
               "Path": "C:\\Users\\redacted\\.config\\containers\\podman\\machine\\wsl"
          },
          "ConnectionInfo": {
               "PodmanSocket": null,
               "PodmanPipe": {
                    "Path": "\\\\.\\pipe\\podman-machine-default"
               }
          },
          "Created": "2024-06-15T00:08:30.4454759+02:00",
          "LastUp": "0001-01-01T00:00:00Z",
          "Name": "podman-machine-default",
          "Resources": {
               "CPUs": 10,
               "DiskSize": 100,
               "Memory": 2048,
               "USBs": []
          },
          "SSHConfig": {
               "IdentityPath": "C:\\Users\\redacted\\.local\\share\\containers\\podman\\machine\\machine",
               "Port": 50625,
               "RemoteUsername": "user"
          },
          "State": "running",
          "UserModeNetworking": false,
          "Rootful": true,
          "Rosetta": false
     }
]

systemctl status systemd-binfmt.service

× systemd-binfmt.service - Set Up Additional Binary Formats
     Loaded: loaded (/usr/lib/systemd/system/systemd-binfmt.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Sat 2024-06-15 00:20:02 CEST; 1min 6s ago
       Docs: man:systemd-binfmt.service(8)
             man:binfmt.d(5)
             https://docs.kernel.org/admin-guide/binfmt-misc.html
             https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
    Process: 77 ExecStart=/usr/lib/systemd/systemd-binfmt (code=exited, status=1/FAILURE)
   Main PID: 77 (code=exited, status=1/FAILURE)

Jun 15 00:20:02 D7C8HW3 systemd[1]: Starting systemd-binfmt.service - Set Up Additional Binary Formats...
Jun 15 00:20:02 D7C8HW3 systemd-binfmt[77]: Failed to check if /proc/sys/fs/binfmt_misc is mounted: Too many levels of symbolic links
Jun 15 00:20:02 D7C8HW3 systemd[1]: systemd-binfmt.service: Main process exited, code=exited, status=1/FAILURE
Jun 15 00:20:02 D7C8HW3 systemd[1]: systemd-binfmt.service: Failed with result 'exit-code'.
Jun 15 00:20:02 D7C8HW3 systemd[1]: Failed to start systemd-binfmt.service - Set Up Additional Binary Formats.

When I run image which does not malfunction, it does not have this failed service.

[taoyouh]

Mine:

PS C:\Users\zhaoq> podman machine start
Starting machine "podman-machine-default"

This machine is currently configured in rootless mode. If your containers
require root permissions (e.g. ports < 1024), or if you run into compatibility
issues with non-podman clients, you can switch using the following command:

        podman machine set --rootful

API forwarding for Docker API clients is not available due to the following startup failures.
        could not start api proxy since expected pipe is not available: podman-machine-default

Podman clients are still able to connect.
Error: machine did not transition into running state: ssh error: machine is not listening on ssh port

ss -tln:

State          Recv-Q          Send-Q                    Local Address:Port                   Peer Address:Port
LISTEN         0               1000                     10.255.255.254:53                          0.0.0.0:*

systemctl --failed:

  UNIT                   LOAD   ACTIVE SUB    DESCRIPTION
● systemd-binfmt.service loaded failed failed Set Up Additional Binary Formats

Legend: LOAD   → Reflects whether the unit definition was properly loaded.
        ACTIVE → The high-level unit activation state, i.e. generalization of SUB.
        SUB    → The low-level unit activation state, values depend on unit type.

1 loaded units listed.

Get-Content $HOME/.wslconfig

[wsl2]
networkingMode=mirrored

podman machine inspect

[
     {
          "ConfigDir": {
               "Path": "C:\\Users\\zhaoq\\.config\\containers\\podman\\machine\\wsl"
          },
          "ConnectionInfo": {
               "PodmanSocket": null,
               "PodmanPipe": {
                    "Path": "\\\\.\\pipe\\podman-machine-default"
               }
          },
          "Created": "2024-06-14T19:58:15.5640189+08:00",
          "LastUp": "2024-06-14T20:16:21.5814581+08:00",
          "Name": "podman-machine-default",
          "Resources": {
               "CPUs": 8,
               "DiskSize": 100,
               "Memory": 2048,
               "USBs": []
          },
          "SSHConfig": {
               "IdentityPath": "C:\\Users\\zhaoq\\.local\\share\\containers\\podman\\machine\\machine",
               "Port": 53143,
               "RemoteUsername": "user"
          },
          "State": "running",
          "UserModeNetworking": false,
          "Rootful": false,
          "Rosetta": false
     }
]

[mheon]

networkingMode=mirrored may be the common factor?

[taoyouh]

I tried. With networkingMode=mirrored it doesn't work. Without that config, it works.

[Luap99]

per https://github.com/containers/podman/issues/22975 mirroed is not supported although you say it used to work with an older image so I am bit confused, @n1hility?

[taoyouh]

22975 says 4.9 worked but 5.0 doesn't.

[POnakS]

Yes, however I suspect that this is not related to podman verison, but machine version update (I am not sure how exactly is this connected).

Without mirrored networking mode, my VPN (Cisco anyconnect) messes everything up.