search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.92k stars 2.42k forks source link

Dangling Network / aardvark-dns / Container host file causes DNS Resolution Failure Switching between Valid and Invalid IP Address #22407

Closed luckylinux closed 2 months ago

luckylinux commented 7 months ago

Issue Description

It seems that for some reason, in some cases (maybe killing a podman create or podman run or running podman-compose up -d before running podman-compose down first) some Network Configuration is never removed.

This leads to Inconsistent Behavior when running "Client" Containers.

In the Specific case, I am running a PostgreSQL Database Server First migration-postgresql-testing.

The I run a PGLoader Image as well as several other PostgreSQL Database Server Containers with Client-only Functions (overkill, but I did not find an image which only contained psql, pg_dump and pg_restore).

The Client Container, then, cannot consistently resolve the DNS name of migration-postgresql-testing.

I lost approximatively 2 days trying to sort this out, because sometimes some Container Instances seem to work, adding a sleep 5-60 prior to executing the command seemed to help in some cases, but never really fix the issue.

I'd say, statistically, 50% of the Client Containers would fail to reach the Database Server, with message beeing either of:

The Faulty Configuration can be found by just issueing a grep -r {{DANGLING_IP_ADDRESS}}:

Steps to reproduce the issue

Steps to reproduce the issue

  1. I attempted to manufacture a "Reproducing Script" at https://github.com/luckylinux/migrate-sql-database/blob/main/debug_podman_dns_issues.sh but unfortunately I could NOT manage to replicate the issue on a consistent basis (even with Networks that still contain "Dangling Containers" IP References)

When I was having this issue I tried a lot of different steps, including:

It is possible that the issue occurs for instance when doing podman-compose up then sending SIGTERM with CTRL+C, then re-running podman-compose up -d. Or similar Situation with podman directly Not sure.

Another Option, which I had many Issues Yesterday, was related to how podman takes arguments as well as (my very basic :upside_down_face: ) knowledge of BASH Variable Expansions inside functions. You can have a look at https://github.com/luckylinux/migrate-sql-database/blob/main/functions.sh but right now I settled with a BASH Array for every argument, then expand that using "${ARRAY[*]}" and finally using BASH eval to run the command.

I say this because many times my script migrate.sh would improperly not quote correctly the argument to bash -c, resulting in Command not found errors. Other times, it seemed that the podman run (which I was using previously instead of podman create + podman start which podman-compose up -d is using) didn't like the arguments that I was passing (either due to double quotes or lack thereof, Variable Expansion not working well with Items containing Spaces, ...). To try to replicate it's maybe possible to look at previous versions of functions.sh as well as old versions of migrate.sh.

I was using code such as for instance https://github.com/luckylinux/migrate-sql-database/commit/b88e6876b0bcd7ae420e3a9f8bd297cfada380e1 or previous commits, beforing using BASH ARRAYS for Argument Expansion like I'm doing now.

Unsure. Maybe something there left podman in an unclean State ?

Describe the results you received

Describe the results you received.

Example of Result BEFORE removing the offending /run/user/1000/networks/aardvark-dns/homeassistant file. Please note that the DNS Hostname Resolution keeps jumping between 10.89.0.109 (correct) and 10.89.0.21 (dangling Configuration of the Same Container name but NOT id):

Running Container migration-postgresql-testing
migration-postgresql-testing
migration-postgresql-testing
e71de1b4326de94369784e94f0e54888308a4dd5e3f19b63b21368a17ace8c0f
migration-postgresql-testing
Running Container network-debug-utils
317c32b04abef42f3b8720d164f4530656ce3f3610a8d217432e5a3cc868ee32
===============================================================================================
===============================================================================================
===============================================================================================
Performing Run 001
Performing NSLOOKUP from Container network-debug-utils Querying DNS for migration-postgresql-testing
Server:     10.89.0.1
Address:    10.89.0.1#53

Non-authoritative answer:
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.109
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.109

Processing Container migration-postgresql-testing
     Network: homeassistant , IP: 10.89.0.109
Processing Container network-debug-utils
     Network: homeassistant , IP: 10.89.0.110
Restarting Container network-debug-utils
network-debug-utils

===============================================================================================
===============================================================================================
===============================================================================================
Performing Run 002
Performing NSLOOKUP from Container network-debug-utils Querying DNS for migration-postgresql-testing
Server:     10.89.0.1
Address:    10.89.0.1#53

Non-authoritative answer:
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.109
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.109

Processing Container migration-postgresql-testing
     Network: homeassistant , IP: 10.89.0.109
Processing Container network-debug-utils
     Network: homeassistant , IP: 10.89.0.111
Restarting Container network-debug-utils
network-debug-utils

===============================================================================================
===============================================================================================
===============================================================================================
Performing Run 003
Performing NSLOOKUP from Container network-debug-utils Querying DNS for migration-postgresql-testing
Server:     10.89.0.1
Address:    10.89.0.1#53

Non-authoritative answer:
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.109
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.109

Processing Container migration-postgresql-testing
     Network: homeassistant , IP: 10.89.0.109
Processing Container network-debug-utils
     Network: homeassistant , IP: 10.89.0.112
Restarting Container network-debug-utils
network-debug-utils

===============================================================================================
===============================================================================================
===============================================================================================
Performing Run 004
Performing NSLOOKUP from Container network-debug-utils Querying DNS for migration-postgresql-testing
Server:     10.89.0.1
Address:    10.89.0.1#53

Non-authoritative answer:
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.21
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.21

Processing Container migration-postgresql-testing
     Network: homeassistant , IP: 10.89.0.109
Processing Container network-debug-utils
     Network: homeassistant , IP: 10.89.0.113
Restarting Container network-debug-utils
network-debug-utils

===============================================================================================
===============================================================================================
===============================================================================================
Performing Run 005
Performing NSLOOKUP from Container network-debug-utils Querying DNS for migration-postgresql-testing
Server:     10.89.0.1
Address:    10.89.0.1#53

Non-authoritative answer:
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.109
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.109

Processing Container migration-postgresql-testing
     Network: homeassistant , IP: 10.89.0.109
Processing Container network-debug-utils
     Network: homeassistant , IP: 10.89.0.114
Restarting Container network-debug-utils
network-debug-utils

===============================================================================================
===============================================================================================
===============================================================================================
Performing Run 006
Performing NSLOOKUP from Container network-debug-utils Querying DNS for migration-postgresql-testing
Server:     10.89.0.1
Address:    10.89.0.1#53

Non-authoritative answer:
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.21
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.21

Processing Container migration-postgresql-testing
     Network: homeassistant , IP: 10.89.0.109
Processing Container network-debug-utils
     Network: homeassistant , IP: 10.89.0.115
Restarting Container network-debug-utils
network-debug-utils

Example AFTER removing the offending /run/user/1000/networks/aardvark-dns/homeassistant file:

Create Target Container migration-postgresql-testing attached to homeassistant and database
35e70e7a603c5269b30b9044a5816616c727d6d3a1b134ef328157cfbaa2f52f
Start Target Container migration-postgresql-testing
Create Target Container migration-postgresql-testing attached to homeassistant Only
ddcfd18a404fd2eb11076b44aee13080fa7c5e4bb8758ac41a232251b4e8a842
Start Target Container migration-postgresql-testing
Running Container network-debug-utils
0f54543dcfaf19ab5f5913ea7f7c7ef8b4743c43a36a4c18ba71f8afdd01404a
===============================================================================================
===============================================================================================
===============================================================================================
Performing Run 001
Performing NSLOOKUP from Container network-debug-utils Querying DNS for migration-postgresql-testing
Server:     10.89.0.1
Address:    10.89.0.1#53

Non-authoritative answer:
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.120
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.120

Processing Container migration-postgresql-testing
     Network: homeassistant , IP: 10.89.0.120
Processing Container network-debug-utils
     Network: homeassistant , IP: 10.89.0.121
Restarting Container network-debug-utils
network-debug-utils

===============================================================================================
===============================================================================================
===============================================================================================
Performing Run 002
Performing NSLOOKUP from Container network-debug-utils Querying DNS for migration-postgresql-testing
Server:     10.89.0.1
Address:    10.89.0.1#53

Non-authoritative answer:
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.120
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.120

Processing Container migration-postgresql-testing
     Network: homeassistant , IP: 10.89.0.120
Processing Container network-debug-utils
     Network: homeassistant , IP: 10.89.0.122
Restarting Container network-debug-utils
network-debug-utils

===============================================================================================
===============================================================================================
===============================================================================================
Performing Run 003
Performing NSLOOKUP from Container network-debug-utils Querying DNS for migration-postgresql-testing
Server:     10.89.0.1
Address:    10.89.0.1#53

Non-authoritative answer:
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.120
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.120

Processing Container migration-postgresql-testing
     Network: homeassistant , IP: 10.89.0.120
Processing Container network-debug-utils
     Network: homeassistant , IP: 10.89.0.123
Restarting Container network-debug-utils
network-debug-utils

===============================================================================================
===============================================================================================
===============================================================================================
Performing Run 004
Performing NSLOOKUP from Container network-debug-utils Querying DNS for migration-postgresql-testing
Server:     10.89.0.1
Address:    10.89.0.1#53

Non-authoritative answer:
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.120
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.120

Processing Container migration-postgresql-testing
     Network: homeassistant , IP: 10.89.0.120
Processing Container network-debug-utils
     Network: homeassistant , IP: 10.89.0.124
Restarting Container network-debug-utils
network-debug-utils

===============================================================================================
===============================================================================================
===============================================================================================
Performing Run 005
Performing NSLOOKUP from Container network-debug-utils Querying DNS for migration-postgresql-testing
Server:     10.89.0.1
Address:    10.89.0.1#53

Non-authoritative answer:
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.120
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.120

Processing Container migration-postgresql-testing
     Network: homeassistant , IP: 10.89.0.120
Processing Container network-debug-utils
     Network: homeassistant , IP: 10.89.0.125
Restarting Container network-debug-utils
network-debug-utils

===============================================================================================
===============================================================================================
===============================================================================================
Performing Run 006
Performing NSLOOKUP from Container network-debug-utils Querying DNS for migration-postgresql-testing
Server:     10.89.0.1
Address:    10.89.0.1#53

Non-authoritative answer:
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.120
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.120

Processing Container migration-postgresql-testing
     Network: homeassistant , IP: 10.89.0.120
Processing Container network-debug-utils
     Network: homeassistant , IP: 10.89.0.126
Restarting Container network-debug-utils
network-debug-utils

===============================================================================================
===============================================================================================
===============================================================================================
Performing Run 007
Performing NSLOOKUP from Container network-debug-utils Querying DNS for migration-postgresql-testing
Server:     10.89.0.1
Address:    10.89.0.1#53

Non-authoritative answer:
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.120
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.120

Processing Container migration-postgresql-testing
     Network: homeassistant , IP: 10.89.0.120
Processing Container network-debug-utils
     Network: homeassistant , IP: 10.89.0.127
Restarting Container network-debug-utils
network-debug-utils

===============================================================================================
===============================================================================================
===============================================================================================
Performing Run 008
Performing NSLOOKUP from Container network-debug-utils Querying DNS for migration-postgresql-testing
Server:     10.89.0.1
Address:    10.89.0.1#53

Non-authoritative answer:
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.120
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.0.120

Processing Container migration-postgresql-testing
     Network: homeassistant , IP: 10.89.0.120
Processing Container network-debug-utils
     Network: homeassistant , IP: 10.89.0.128
Restarting Container network-debug-utils
network-debug-utils

Finding the culprit by running grep -r "10.89.0.21" /run/user/1000/

/run/user/1000/networks/aardvark-dns/database_internal:04ca53f4ab287e907b06c05bfe2c060859a7fa1eca4ac7b43812c2c592cb664e 10.89.0.21  migration-postgresql-testing,migration-postgresql-testing,04ca53f4ab28
grep: /run/user/1000/networks/ipam.db: binary file matches
/run/user/1000/overlay-containers/04ca53f4ab287e907b06c05bfe2c060859a7fa1eca4ac7b43812c2c592cb664e/userdata/hosts:10.89.0.21    04ca53f4ab28 migration-postgresql-testing
grep: /run/user/1000/systemd/inaccessible/dir: Permission denied
grep: /run/user/1000/systemd/inaccessible/reg: Permission denied

Even if a "Dangling Network" has been previously causing issues, the offending Containers (that have now been deleted) still reside inside the file.

cat /run/user/1000/networks/aardvark-dns/homeassistant_internal yields for instance:

10.89.1.1
04ca53f4ab287e907b06c05bfe2c060859a7fa1eca4ac7b43812c2c592cb664e 10.89.1.21  migration-postgresql-testing,migration-postgresql-testing,04ca53f4ab28
5d27f4e0608cdbca50b054785513351a57cae5dc79a995121f939bad15a0c5dc 10.89.1.22  migration-timescaledb-testing,migration-timescaledb-testing,5d27f4e0608c
eef5d0348aeb2852ac93a86bde9b5b177fefac3e050272830af8663b4b2729fd 10.89.1.109  migration-postgresql-testing,migration-postgresql-testing,eef5d0348aeb
0f945cc17fc22675ef2fac7dcdc0e0bd01e5bdead182f288f797db723421fa08 10.89.1.110  migration-timescaledb-testing,migration-timescaledb-testing,0f945cc17fc2

This occurs also if, to try to replicate the issue, I use the same network name homeassistant_internal. Newly Created Containers get correctly added to the file, and removed, if they are removed and the Network is not listed in the updated --net section of podman run or podman create.

But if I now use homeassistant_internal for my "loop" script "normal" Container Network, then I can replicate the issue, provided that the dangling file is around: cat /run/user/1000/networks/aardvark-dns/homeassistant_internal yields:

10.89.1.1
04ca53f4ab287e907b06c05bfe2c060859a7fa1eca4ac7b43812c2c592cb664e 10.89.1.21  migration-postgresql-testing,migration-postgresql-testing,04ca53f4ab28
5d27f4e0608cdbca50b054785513351a57cae5dc79a995121f939bad15a0c5dc 10.89.1.22  migration-timescaledb-testing,migration-timescaledb-testing,5d27f4e0608c
eef5d0348aeb2852ac93a86bde9b5b177fefac3e050272830af8663b4b2729fd 10.89.1.109  migration-postgresql-testing,migration-postgresql-testing,eef5d0348aeb
0f945cc17fc22675ef2fac7dcdc0e0bd01e5bdead182f288f797db723421fa08 10.89.1.110  migration-timescaledb-testing,migration-timescaledb-testing,0f945cc17fc2
156ad022d11b90ac302a1b41961d4576e802a34682a97c0b11b02b9a494e3b52 10.89.1.229  migration-postgresql-testing,migration-postgresql-testing,156ad022d11b
954d063412cbf12ddcb453931a3b980d41a5ddef53e88cbc665c624dcd041429 10.89.1.243  network-debug-utils,954d063412cb

With Script Content changed slightly:

#!/bin/bash

# Legacy Networks PREVIOUSLY Associated with the Container are still listed in /run/user/1000/networks/aardvark-dns/${networkname} as well as some host file in /run/user/1000/overlay-containers/*/userdata/hosts

# Determine toolpath if not set already
relativepath="./" # Define relative path to go from this script to the root level of the tool
if [[ ! -v toolpath ]]; then scriptpath=$(cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd ); toolpath=$(realpath --canonicalize-missing $scriptpath/$relativepath); fi

# Set Secrets and Settings
CONTAINER_DANGLING_NETWORK="homeassistant"
CONTAINER_NETWORK="homeassistant_internal"
POSTGRES_DB="homeassistant"
POSTGRES_USER="homeassistant"
POSTGRES_PASSWORD="MySuperSecretPassword"

# Configure Timing Parameters
delayedstart="15"
loopinterval="5"

# Setup Traps
trap "echo signal; exit 0" SIGTERM

# Define Debug Running Container Name
debugcontainer="network-debug-utils"

# Define Target Container for nslookup
targetcontainer="migration-postgresql-testing"

# Stop Containers in Case they are Running
podman stop --ignore ${debugcontainer} 1> /dev/null 2> /dev/null
podman stop --ignore ${targetcontainer} 1> /dev/null 2> /dev/null
podman rm --ignore ${debugcontainer} 1> /dev/null 2> /dev/null
podman rm --ignore ${targetcontainer} 1> /dev/null 2> /dev/null

# Echo
echo "Create Target Container ${targetcontainer} attached to ${CONTAINER_NETWORK} and ${CONTAINER_DANGLING_NETWORK}"

# Create Extra Network
podman network create --ignore ${CONTAINER_DANGLING_NETWORK} 1> /dev/null 2> /dev/null

# Create Target Container
podman stop --ignore ${targetcontainer} 1> /dev/null 2> /dev/null
podman rm --ignore ${targetcontainer} 1> /dev/null 2> /dev/null
podman create --name=${targetcontainer} --env-file ./.env -e POSTGRES_DB=${INTERMEDIARY_DATABASE_DB} -e POSTGRES_USER=${INTERMEDIARY_DATABASE_USER} -e POSTGRES_PASSWORD=${INTERMEDIARY_DATABASE_PASSWORD} -v ./test/containers/data/migration-postgresql-testing:/var/lib/postgresql/data --network=${CONTAINER_NETWORK},${CONTAINER_DANGLING_NETWORK} --network-alias ${targetcontainer} --expose 5432 -p 5433:5432 -u root --pull missing --restart unless-stopped postgres:latest

# Echo
echo "Start Target Container ${targetcontainer}"

# Start Target Container
podman start ${targetcontainer} 1> /dev/null 2> /dev/null

# List all IPs
./list_ips.sh

# Wait a bit
sleep ${delayedstart}

# Echo
echo "Create Target Container ${targetcontainer} attached to ${CONTAINER_NETWORK} Only"

# Create Target Container
podman stop --ignore ${targetcontainer} 1> /dev/null 2> /dev/null
podman rm --ignore ${targetcontainer} 1> /dev/null 2> /dev/null

# Remove Network
####podman network rm ${CONTAINER_DANGLING_NETWORK}

# Start Container
podman create --name=${targetcontainer} --env-file ./.env -e POSTGRES_DB=${INTERMEDIARY_DATABASE_DB} -e POSTGRES_USER=${INTERMEDIARY_DATABASE_USER} -e POSTGRES_PASSWORD=${INTERMEDIARY_DATABASE_PASSWORD} -v ./test/containers/data/migration-postgresql-testing:/var/lib/postgresql/data --network=${CONTAINER_NETWORK} --network-alias ${targetcontainer} --expose 5432 -p 5433:5432 -u root --pull missing --restart unless-stopped postgres:latest

# Echo
echo "Start Target Container ${targetcontainer}"

# Start Target Container
podman start ${targetcontainer} 1> /dev/null 2> /dev/null

# List all IPs
./list_ips.sh

# Wait a bit
sleep ${delayedstart}

# Echo
echo "Running Container ${debugcontainer}"

# Run Unattended Container and let it Loop
# Replace Container if it already exists
podman run -d --rm -v ./loop.sh:/loop.sh --name="${debugcontainer}" --user root --net "${CONTAINER_NETWORK}" arunvelsriram/utils bash -c "/loop.sh"

# Initialize Counter
counter=1

# Run nslookup upon multiple Restarts
while [ true ]
do
    # Wait 5 seconds
    sleep ${loopinterval} & wait $!

    # Echo
    echo "==============================================================================================="
    echo "==============================================================================================="
    echo "==============================================================================================="

    # Echo
    c=$(printf "%03d" $counter)
    echo "Performing Run ${c}"

    # Echo
    echo "Performing NSLOOKUP from Container ${debugcontainer} Querying DNS for ${targetcontainer}"

    # Run nslookup
    podman exec -it ${debugcontainer} nslookup ${targetcontainer}

    # List all IPs
    ./list_ips.sh

    # Echo
    echo "Restarting Container ${debugcontainer}"

    # Restart Container
    podman restart ${debugcontainer} 1> /dev/null 2> /dev/null

    # Echo
    echo -e "\n\n"

    # Increment Counter
    counter=$((counter + 1))
done

And Result:

Create Target Container migration-postgresql-testing attached to homeassistant_internal and homeassistant
5488a9861ad6d97107b37cc04d6c56ad5e33d12ae7535cc56d539559d056b6e0
Start Target Container migration-postgresql-testing
Processing Container migration-postgresql-testing
     Network: homeassistant , IP: 10.89.0.79
     Network: homeassistant_internal , IP: 10.89.1.247
Create Target Container migration-postgresql-testing attached to homeassistant_internal Only
a4e4bb8084f3ffddb2c592eae0ed7a5abe3098c45be39943f8ffcbfd28d90003
Start Target Container migration-postgresql-testing
Processing Container migration-postgresql-testing
     Network: homeassistant_internal , IP: 10.89.1.248
Running Container network-debug-utils
87ae31768205706306247761d9d0cb9d7711c15a9128f0654cf3584fcc607445
===============================================================================================
===============================================================================================
===============================================================================================
Performing Run 001
Performing NSLOOKUP from Container network-debug-utils Querying DNS for migration-postgresql-testing
Server:     10.89.1.1
Address:    10.89.1.1#53

Non-authoritative answer:
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.21
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.21
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.109
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.109
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.248
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.248

Processing Container migration-postgresql-testing
     Network: homeassistant_internal , IP: 10.89.1.248
Processing Container network-debug-utils
     Network: homeassistant_internal , IP: 10.89.1.249
Restarting Container network-debug-utils

===============================================================================================
===============================================================================================
===============================================================================================
Performing Run 002
Performing NSLOOKUP from Container network-debug-utils Querying DNS for migration-postgresql-testing
Server:     10.89.1.1
Address:    10.89.1.1#53

Non-authoritative answer:
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.21
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.21
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.109
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.109
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.248
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.248

Processing Container migration-postgresql-testing
     Network: homeassistant_internal , IP: 10.89.1.248
Processing Container network-debug-utils
     Network: homeassistant_internal , IP: 10.89.1.250
Restarting Container network-debug-utils

===============================================================================================
===============================================================================================
===============================================================================================
Performing Run 003
Performing NSLOOKUP from Container network-debug-utils Querying DNS for migration-postgresql-testing
Server:     10.89.1.1
Address:    10.89.1.1#53

Non-authoritative answer:
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.21
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.21
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.109
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.109
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.248
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.248

Processing Container migration-postgresql-testing
     Network: homeassistant_internal , IP: 10.89.1.248
Processing Container network-debug-utils
     Network: homeassistant_internal , IP: 10.89.1.251
Restarting Container network-debug-utils

===============================================================================================
===============================================================================================
===============================================================================================
Performing Run 004
Performing NSLOOKUP from Container network-debug-utils Querying DNS for migration-postgresql-testing
Server:     10.89.1.1
Address:    10.89.1.1#53

Non-authoritative answer:
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.21
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.21
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.109
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.109
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.248
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.248

Processing Container migration-postgresql-testing
     Network: homeassistant_internal , IP: 10.89.1.248
Processing Container network-debug-utils
     Network: homeassistant_internal , IP: 10.89.1.252
Restarting Container network-debug-utils

===============================================================================================
===============================================================================================
===============================================================================================
Performing Run 005
Performing NSLOOKUP from Container network-debug-utils Querying DNS for migration-postgresql-testing
Server:     10.89.1.1
Address:    10.89.1.1#53

Non-authoritative answer:
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.21
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.21
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.109
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.109
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.248
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.248

Processing Container migration-postgresql-testing
     Network: homeassistant_internal , IP: 10.89.1.248
Processing Container network-debug-utils
     Network: homeassistant_internal , IP: 10.89.1.253
Restarting Container network-debug-utils

===============================================================================================
===============================================================================================
===============================================================================================
Performing Run 006
Performing NSLOOKUP from Container network-debug-utils Querying DNS for migration-postgresql-testing
Server:     10.89.1.1
Address:    10.89.1.1#53

Non-authoritative answer:
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.21
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.21
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.109
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.109
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.248
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.248

Processing Container migration-postgresql-testing
     Network: homeassistant_internal , IP: 10.89.1.248
Processing Container network-debug-utils
     Network: homeassistant_internal , IP: 10.89.1.254
Restarting Container network-debug-utils

===============================================================================================
===============================================================================================
===============================================================================================
Performing Run 007
Performing NSLOOKUP from Container network-debug-utils Querying DNS for migration-postgresql-testing
Server:     10.89.1.1
Address:    10.89.1.1#53

Non-authoritative answer:
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.21
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.21
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.109
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.109
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.248
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.248

Processing Container migration-postgresql-testing
     Network: homeassistant_internal , IP: 10.89.1.248
Processing Container network-debug-utils
     Network: homeassistant_internal , IP: 10.89.1.2
Restarting Container network-debug-utils

===============================================================================================
===============================================================================================
===============================================================================================
Performing Run 008
Performing NSLOOKUP from Container network-debug-utils Querying DNS for migration-postgresql-testing
Server:     10.89.1.1
Address:    10.89.1.1#53

Non-authoritative answer:
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.21
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.21
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.109
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.109
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.248
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.248

Processing Container migration-postgresql-testing
     Network: homeassistant_internal , IP: 10.89.1.248
Processing Container network-debug-utils
     Network: homeassistant_internal , IP: 10.89.1.3
Restarting Container network-debug-utils

===============================================================================================
===============================================================================================
===============================================================================================
Performing Run 009
Performing NSLOOKUP from Container network-debug-utils Querying DNS for migration-postgresql-testing
Server:     10.89.1.1
Address:    10.89.1.1#53

Non-authoritative answer:
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.21
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.21
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.109
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.109
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.248
Name:   migration-postgresql-testing.dns.podman
Address: 10.89.1.248

Processing Container migration-postgresql-testing
     Network: homeassistant_internal , IP: 10.89.1.248
Processing Container network-debug-utils
     Network: homeassistant_internal , IP: 10.89.1.4
Restarting Container network-debug-utils

But why is there a file dangling in the first place ???

And why isn't podman rm deleting OLD entries of that container from /run/user/1000/networks/aardvark-dns/{{DANGLING_NETWORK}} ?

Describe the results you expected

DNS Resolution working Correctly.

podman stop & podman rm removing the deprecated Hostname-IP Address association from the Temporary /run/user/1000/networks/aardvark-dns/{{NETWORK_NAME}} File.

For instance, why doesn't podman rm migration-postgresql-testing change the file /run/user/1000/networks/aardvark-dns/homeassistant_internal to: `` 10.89.1.1 954d063412cbf12ddcb453931a3b980d41a5ddef53e88cbc665c624dcd041429 10.89.1.243 network-debug-utils,954d063412cb


### podman info output

```yaml
host:
  arch: amd64
  buildahVersion: 1.33.5
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.10+ds1-1build2_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: unknown'
  cpuUtilization:
    idlePercent: 58.14
    systemPercent: 19.44
    userPercent: 22.43
  cpus: 8
  databaseBackend: sqlite
  distribution:
    codename: mantic
    distribution: ubuntu
    version: "23.10"
  eventLogger: journald
  freeLocks: 2044
  hostname: UbuntuWorkstation02
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.5.0-27-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 1728823296
  memTotal: 33603158016
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.4.0-5_amd64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.4.0
    package: netavark_1.4.0-4_amd64
    path: /usr/lib/podman/netavark
    version: netavark 1.4.0
  ociRuntime:
    name: crun
    package: crun_1.14.1-1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.1
      commit: de537a7965bfbe9992e2cfae0baeb56a08128171
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_0.0~git20231230.f091893-1_amd64
    version: |
      pasta unknown version
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.0-1_amd64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 53h 38m 3.00s (Approximately 2.21 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  docker.io:
    Blocked: false
    Insecure: false
    Location: docker.MYDOMAIN.TLD
    MirrorByDigestOnly: false
    Mirrors:
    - Insecure: false
      Location: docker.MYDOMAIN.TLD/docker.io
      PullFromMirror: ""
    - Insecure: false
      Location: docker.MYDOMAIN.TLD/docker.io/library
      PullFromMirror: ""
    Prefix: docker.io
    PullFromMirror: ""
  docker.MYDOMAIN.TLD:
    Blocked: false
    Insecure: false
    Location: docker.MYDOMAIN.TLD
    MirrorByDigestOnly: false
    Mirrors:
    - Insecure: false
      Location: docker.MYDOMAIN.TLD/docker.io
      PullFromMirror: ""
    - Insecure: false
      Location: docker.MYDOMAIN.TLD/docker.io/library
      PullFromMirror: ""
    - Insecure: false
      Location: docker.MYDOMAIN.TLD/ghcr.io
      PullFromMirror: ""
    - Insecure: false
      Location: docker.MYDOMAIN.TLD/ghcr.io/library
      PullFromMirror: ""
    Prefix: docker.MYDOMAIN.TLD
    PullFromMirror: ""
  ghcr.io:
    Blocked: false
    Insecure: false
    Location: docker.MYDOMAIN.TLD/ghcr.io
    MirrorByDigestOnly: false
    Mirrors:
    - Insecure: false
      Location: docker.MYDOMAIN.TLD/ghcr.io
      PullFromMirror: ""
    - Insecure: false
      Location: docker.MYDOMAIN.TLD/ghcr.io/library
      PullFromMirror: ""
    Prefix: ghcr.io
    PullFromMirror: ""
  search:
  - docker.MYDOMAIN.TLD
  - docker.io
  - quay.io
store:
  configFile: /home/USER/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs_1.13-1_amd64
      Version: |-
        fusermount3 version: 3.14.0
        fuse-overlayfs: version 1.13-dev
        FUSE library version 3.14.0
        using FUSE kernel interface version 7.31
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /home/USER/containers/storage
  graphRootAllocated: 448781615104
  graphRootUsed: 163656368128
  graphStatus:
    Backing Filesystem: zfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 5
  runRoot: /run/user/1000
  transientStore: false
  volumePath: /home/USER/containers/volumes
version:
  APIVersion: 4.9.3
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.22.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.3

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

Baremetal Ubuntu AMD64 Host.

Ubuntu Mantic 23.10 with Podman 4.9.3 pinned from Ubuntu Testing/Noble.

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

github-actions[bot] commented 6 months ago

A friendly reminder that this issue had no activity for 30 days.

luckylinux commented 6 months ago

Yep, difficult to replicate unfortunately :(.

Luap99 commented 5 months ago

@luckylinux What podman version are you using currently? Do you still see this issue?

luckylinux commented 5 months ago

@Luap99: Podman 4.9.4 on Debian/Ubuntu, Podman 5.1.1 on Fedora

As for the Issue, well, the moment I say "No" is the moment where the Issue will occur, so I'm NOT going to say anything :laughing:.

Luap99 commented 5 months ago

I did some changes around the auto restart networking behaviour recently (https://github.com/containers/podman/commit/15b8bb72a8e984c56a5f9a38986b651971182e84) v5.1 so I wonder if it had anything to do with that.

Luap99 commented 2 months ago

I am going to close this but feel free to ping me when it happens again