Podman container cannot access port on Windows host

otavio-silva commented 5 months ago

Issue Description

I'm trying to run the Open WebUI container and access the Ollama API exposed on http://localhost:11434, since Ollama is installed on my local Windows 11 machine. However, the container running in the Podman machine (podman-machine-default WSL2 distro) cannot reach addresses on the Windows host.

Steps to reproduce the issue

Install Ollama Windows Preview version and verify it is running
Run one of the commands below, where WIN_IP is the result of ip route show | grep -i default | awk '{ print $3}' on podman-machine-default: 1.podman run -d -p 11435:8080 --add-host=host.containers.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main 2.podman run -d -p 11435:8080 -e OLLAMA_BASE_URL=WIN_IP:11434 -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main 3.podman -d --network=host -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=http://WIN_IP:11434 --name open-webui --restart always ghcr.io/open-webui/open-webui:main
Open the adress http://localhost:11435 or http://localhost:8080 if using the --network=host flag
Open WebUI will give error that it could not connect to Ollama, podman logs shows the ERROR:apps.ollama.main:Connection error: Cannot connect to host WIN_IP:11434 ssl:default [Connection refused]

Describe the results you received

The container could not reach the Windows app in any way

Describe the results you expected

The container to retrieve results from the address on the Windows host

podman info output

host:
  arch: amd64
  buildahVersion: 1.36.0-dev
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.10-1.20240313132120223048.main.19.gaffab49.fc39.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 99.9
    systemPercent: 0.04
    userPercent: 0.06
  cpus: 20
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: container
    version: "39"
  eventLogger: journald
  freeLocks: 2047
  hostname: GE76RAIDER
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.15.146.1-microsoft-standard-WSL2
  linkmode: dynamic
  logDriver: journald
  memFree: 28308889600
  memTotal: 33502474240
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.20240329131657512331.main.31.g2c315a1.fc39.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.11.0-dev
    package: netavark-1.10.1-1.20240329131649297909.main.62.gad066d4.fc39.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.11.0-dev
  ociRuntime:
    name: crun
    package: crun-1.14.4-1.20240331210158645743.main.18.gd05a5dd.fc39.x86_64
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: 22fa748a7dd029cd597fc96a2afb78b84cf819e6
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240220.g1e6f92b-1.fc39.x86_64
    version: |
      pasta 0^20240220.g1e6f92b-1.fc39.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: true
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 8589934592
  swapTotal: 8589934592
  uptime: 2h 43m 36.00s (Approximately 0.08 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 1081101176832
  graphRootUsed: 6648487936
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 4
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.1.0-dev-45b809c06
  Built: 1711670400
  BuiltTime: Thu Mar 28 21:00:00 2024
  GitCommit: ""
  GoVersion: go1.21.8
  Os: linux
  OsArch: linux/amd64
  Version: 5.1.0-dev-45b809c06

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

The command wsl --version gives:

Versão do WSL: 2.1.5.0
Versão do kernel: 5.15.146.1-2
Versão do WSLg: 1.0.60
Versão do MSRDC: 1.2.5105
Versão do Direct3D: 1.611.1-81528511
Versão do DXCore: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Versão do Windows: 10.0.22631.3374

Additional information

I've used the issues #14933, #13966 and #13965 as references but none of them could actually fix the issue

M-S-ETB commented 5 months ago

I am not sure if my issue is the same, but I have updated podman from 4.9.3 to 5.0.1 two days ago, and since then it looks like my host services cannot be accessed from within the container context.

In the picture you can see my grafana dashboard monitoring my development API. RabbitMQ runs in a container, the oracle DB and the main API service run on the host. The gap is where I performed the update. broken host connection

The oracledb prometheus exporter keeps spamming

2024-04-05T07:55:29+02:00 time="2024-04-05T05:55:29Z" level=error msg="Error pinging oracle: ORA-12541: TNS:no listener\n" source="main.go:215"

as it cannot reach the host at host.docker.internal or host.containers.internal.

As a workaround I looked into the compose spec and tried different things like a host network, but those aren't supported by podman (yet?)

As a sanity check I have reinstalled the older version of podman (4.9.3), with the following results: working host connection

So for the time being I will pin/hold my package at that version, but I hope this can be resolved in 5.0.2 or some other version soon. I could look into fixing this myself, but I have very little experience in Go, VMs, or networking...

Excerpt of my docker-compose file:

  clipboard-oracledb-exporter:
    image: iamseth/oracledb_exporter
    networks:
      - clipboard_api_default
    environment:
      DATA_SOURCE_NAME: oracle://username:password@host.docker.internal:1521/XE
    ports:
      - "9161:9161"

maxtorete commented 4 months ago

I found a similar issue when upgrading from Fedora Silverblue 39 (Podman 4.9.4) to 40 (podman 5.0.1). In FS39 I can reach host ports inside a container attached to a network defined in docker-compose.yml file. In FS40, the connection is refused. Toolbox created containers can connect to host on both FS39 and FS40.

I tried reseting and recreating again with podman system reset in FS40, but it didn't work.

I rebased again to FS39 (I didn't pin before rebasing to FS40 :disappointed: ) and it works again.

PS: I'm using podman-remote inside a toolbox to create the containers in the host, maybe it could be related...

dontfreakout commented 4 months ago

I am not sure about Windows, but on Linux, I have this problem from Podman 5 (after upgrading to Fedora 40). Just like @maxtorete

The problem is probably related to the Pasta network driver, which has been default since v5.

When I switched from pasta back to slirp4netns, it worked again.

$ cat ~/.config/containers/containers.conf 
[containers]

[engine]

[machine]

[network]
default_rootless_network_cmd="slirp4netns"

[secrets]

[configmaps]

kontza commented 4 months ago

@dontfreakout Cheers for this tip! This got my Cloudflare SSH-tunnel working again.

However, I need to read Pasta docs to see how host access should really be done with it.

timheuer commented 3 months ago

@kontza can you share the steps you did to make this change...I must be doing it wrong as still not working for me.

kontza commented 3 months ago

@kontza can you share the steps you did to make this change...I must be doing it wrong as still not working for me.

Short story: I started to use host networking.

otavio-silva commented 3 months ago

For the record, and I think it's important: using the Ollama app instead of just using the command line, a UAC prompt asked me to enable access for ollama.exe to networking. After giving permission, using podman run -d -p 11435:8080 -e OLLAMA_BASE_URL=WIN_IP:11434 -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main worked just fine.

containers / podman