search

flatcar / Flatcar

Flatcar project repository for issue tracking, project documentation, etc.
https://www.flatcar.org/
Apache License 2.0
688 stars 30 forks source link

systemd-resolved fail to start after rebooting ( mostly) #416

Closed xh116 closed 3 years ago

xh116 commented 3 years ago

Description

systemd-resolved fails on booting sometimes. /etc/resolv.conf( symlink to /run/systemd/resolve/resolv.conf ) will be missing.

resolv

resolv2

Impact

dns will have problem and not able to "ping"

Environment and steps to reproduce

  1. Set-up: running on oracle cloud vps and can be seen on ESXi also (seldom). stable (2765.2.6)
  2. Task: fresh install by flatcar-install script + every reboot
  3. Action(s): when rebooting
  4. Error: 
    systemctl status systemd-resolved 
● systemd-resolved.service - Network Name Resolution
 Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: disabled)
 Active: inactive (dead)
   Docs: man:systemd-resolved.service(8)
         man:org.freedesktop.resolve1(5)
         https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
         https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients

Expected behavior

sudo systemctl restart systemd-resolved cloud solve it.

or by creating a static.network interface ? :

    - name: 20-dhcp.network
      contents: |
        [Match]
        Name=*

        [Network]
        DHCP=yes

Additional information

error I can also see this error.

but I can not 100% reproduce it.

pothos commented 3 years ago

Hello, DHCP is the default already. Can you please provide the journalctl -u systemd-resolved --no-pager output when the unit fails to run? The output of networkctl status eth0 would also be good to know what DHCP DNS information was sent.

xh116 commented 3 years ago

Thanks for reply. 

-- Journal begins at Tue 2021-06-22 18:44:20 UTC, ends at Wed 2021-06-23 13:07:12 UTC. --
Jun 22 18:44:22 localhost systemd[1]: Starting Network Name Resolution...
Jun 22 18:44:22 localhost systemd-resolved[332]: Positive Trust Anchors:
Jun 22 18:44:22 localhost systemd-resolved[332]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Jun 22 18:44:22 localhost systemd-resolved[332]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Jun 22 18:44:22 localhost systemd-resolved[332]: Defaulting to hostname 'linux'.
Jun 22 18:44:22 localhost systemd[1]: Started Network Name Resolution.
Jun 22 18:44:28 localhost systemd[1]: Stopping Network Name Resolution...
Jun 22 18:44:28 localhost systemd[1]: systemd-resolved.service: Succeeded.
Jun 22 18:44:28 localhost systemd[1]: Stopped Network Name Resolution.
Jun 22 18:44:47 Oracle01 systemd[1]: Starting Network Name Resolution...
Jun 22 18:44:48 Oracle01 systemd-resolved[809]: Positive Trust Anchors:
Jun 22 18:44:48 Oracle01 systemd-resolved[809]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Jun 22 18:44:48 Oracle01 systemd-resolved[809]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Jun 22 18:44:48 Oracle01 systemd-resolved[809]: Using system hostname 'Oracle01'.
Jun 22 18:44:48 Oracle01 systemd[1]: Started Network Name Resolution.
Jun 22 18:46:04 Oracle01 systemd[1]: Stopping Network Name Resolution...
Jun 22 18:46:04 Oracle01 systemd[1]: systemd-resolved.service: Succeeded.
Jun 22 18:46:04 Oracle01 systemd[1]: Stopped Network Name Resolution.
-- Boot cb1ebf83089e4a509125bb405d53f03a --
Jun 22 18:46:46 Oracle01 systemd[1]: Starting Network Name Resolution...
Jun 22 18:46:47 Oracle01 systemd-resolved[698]: Positive Trust Anchors:
Jun 22 18:46:47 Oracle01 systemd-resolved[698]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Jun 22 18:46:47 Oracle01 systemd-resolved[698]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Jun 22 18:46:47 Oracle01 systemd-resolved[698]: Using system hostname 'Oracle01'.
Jun 22 18:46:47 Oracle01 systemd[1]: Started Network Name Resolution.
Jun 22 18:47:34 Oracle01 systemd[1]: Stopping Network Name Resolution...
Jun 22 18:47:34 Oracle01 systemd[1]: systemd-resolved.service: Succeeded.
Jun 22 18:47:34 Oracle01 systemd[1]: Stopped Network Name Resolution.
-- Boot a820c224ffa94a21aaf0489e4563699c --
Jun 22 18:48:15 Oracle01 systemd[1]: Starting Network Name Resolution...
Jun 22 18:48:17 Oracle01 systemd-resolved[698]: Positive Trust Anchors:
Jun 22 18:48:17 Oracle01 systemd-resolved[698]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Jun 22 18:48:17 Oracle01 systemd-resolved[698]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Jun 22 18:48:17 Oracle01 systemd-resolved[698]: Using system hostname 'Oracle01'.
Jun 22 18:48:17 Oracle01 systemd[1]: Started Network Name Resolution.
Jun 22 19:12:01 Oracle01 systemd-resolved[698]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 22 19:12:01 Oracle01 systemd-resolved[698]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 22 19:12:01 Oracle01 systemd-resolved[698]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 22 19:12:01 Oracle01 systemd[1]: Stopping Network Name Resolution...
Jun 22 19:12:01 Oracle01 systemd[1]: systemd-resolved.service: Succeeded.
Jun 22 19:12:01 Oracle01 systemd[1]: Stopped Network Name Resolution.
-- Boot 60322840ccae48ef8b0569d7bc0c4d8d --
Jun 22 20:27:58 Oracle01 systemd[1]: Starting Network Name Resolution...
Jun 22 20:27:58 Oracle01 systemd-resolved[1176]: Positive Trust Anchors:
Jun 22 20:27:58 Oracle01 systemd-resolved[1176]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Jun 22 20:27:58 Oracle01 systemd-resolved[1176]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Jun 22 20:27:58 Oracle01 systemd-resolved[1176]: Using system hostname 'Oracle01'.
Jun 22 20:27:58 Oracle01 systemd[1]: Started Network Name Resolution.
Jun 22 20:30:28 Oracle01 systemd-resolved[1176]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 22 20:30:28 Oracle01 systemd-resolved[1176]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 22 20:30:28 Oracle01 systemd[1]: Stopping Network Name Resolution...
Jun 22 20:30:28 Oracle01 systemd[1]: systemd-resolved.service: Succeeded.
Jun 22 20:30:28 Oracle01 systemd[1]: Stopped Network Name Resolution.
-- Boot 7b6fdb8f325442fa947486b7dc117cbe --
Jun 22 21:15:32 Oracle01 systemd[1]: Starting Network Name Resolution...
Jun 22 21:15:32 Oracle01 systemd-resolved[1524]: Positive Trust Anchors:
Jun 22 21:15:32 Oracle01 systemd-resolved[1524]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Jun 22 21:15:32 Oracle01 systemd-resolved[1524]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Jun 22 21:15:32 Oracle01 systemd-resolved[1524]: Using system hostname 'Oracle01'.
Jun 22 21:15:32 Oracle01 systemd[1]: Started Network Name Resolution.
Jun 23 00:59:59 Oracle01 systemd-resolved[1524]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 00:59:59 Oracle01 systemd-resolved[1524]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 00:59:59 Oracle01 systemd-resolved[1524]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 00:59:59 Oracle01 systemd-resolved[1524]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 00:59:59 Oracle01 systemd-resolved[1524]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 00:59:59 Oracle01 systemd-resolved[1524]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 00:59:59 Oracle01 systemd[1]: Stopping Network Name Resolution...
Jun 23 00:59:59 Oracle01 systemd[1]: systemd-resolved.service: Succeeded.
Jun 23 00:59:59 Oracle01 systemd[1]: Stopped Network Name Resolution.
-- Boot 598c424b8e93467c8e3b7becc39a9406 --
Jun 23 02:01:51 Oracle01 systemd[1]: Starting Network Name Resolution...
Jun 23 02:01:51 Oracle01 systemd-resolved[2122]: Positive Trust Anchors:
Jun 23 02:01:51 Oracle01 systemd-resolved[2122]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Jun 23 02:01:51 Oracle01 systemd-resolved[2122]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Jun 23 02:01:51 Oracle01 systemd-resolved[2122]: Using system hostname 'Oracle01'.
Jun 23 02:01:51 Oracle01 systemd[1]: Started Network Name Resolution.
Jun 23 02:07:48 Oracle01 systemd-resolved[2122]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 02:07:48 Oracle01 systemd-resolved[2122]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 02:07:48 Oracle01 systemd-resolved[2122]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 02:07:48 Oracle01 systemd-resolved[2122]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 02:07:48 Oracle01 systemd[1]: Stopping Network Name Resolution...
Jun 23 02:07:48 Oracle01 systemd[1]: systemd-resolved.service: Succeeded.
Jun 23 02:07:48 Oracle01 systemd[1]: Stopped Network Name Resolution.
-- Boot d522652365544892b4e9a9d740543b83 --
Jun 23 02:27:58 Oracle01 systemd[1]: Starting Network Name Resolution...
Jun 23 02:27:59 Oracle01 systemd-resolved[1293]: Positive Trust Anchors:
Jun 23 02:27:59 Oracle01 systemd-resolved[1293]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Jun 23 02:27:59 Oracle01 systemd-resolved[1293]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Jun 23 02:27:59 Oracle01 systemd-resolved[1293]: Using system hostname 'Oracle01'.
Jun 23 02:27:59 Oracle01 systemd[1]: Started Network Name Resolution.
Jun 23 03:03:30 Oracle01 systemd-resolved[1293]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 03:03:30 Oracle01 systemd-resolved[1293]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 03:03:30 Oracle01 systemd-resolved[1293]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 03:03:30 Oracle01 systemd-resolved[1293]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 03:03:30 Oracle01 systemd-resolved[1293]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 03:03:30 Oracle01 systemd-resolved[1293]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 03:03:30 Oracle01 systemd[1]: Stopping Network Name Resolution...
Jun 23 03:03:30 Oracle01 systemd[1]: systemd-resolved.service: Succeeded.
Jun 23 03:03:30 Oracle01 systemd[1]: Stopped Network Name Resolution.
-- Boot bf825c8eecb0450ba1887ddf98eac773 --
Jun 23 03:04:09 Oracle01 systemd[1]: Starting Network Name Resolution...
Jun 23 03:04:11 Oracle01 systemd-resolved[696]: Positive Trust Anchors:
Jun 23 03:04:11 Oracle01 systemd-resolved[696]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Jun 23 03:04:11 Oracle01 systemd-resolved[696]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Jun 23 03:04:11 Oracle01 systemd-resolved[696]: Using system hostname 'Oracle01'.
Jun 23 03:04:11 Oracle01 systemd[1]: Started Network Name Resolution.
Jun 23 04:01:29 Oracle01 systemd-resolved[696]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 04:01:29 Oracle01 systemd-resolved[696]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 04:01:29 Oracle01 systemd-resolved[696]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 04:01:29 Oracle01 systemd-resolved[696]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 04:01:29 Oracle01 systemd[1]: Stopping Network Name Resolution...
Jun 23 04:01:29 Oracle01 systemd[1]: systemd-resolved.service: Succeeded.
Jun 23 04:01:29 Oracle01 systemd[1]: Stopped Network Name Resolution.
-- Boot 42f3b4881638451fb623697a1ad2b287 --
Jun 23 04:27:33 Oracle01 systemd[1]: Starting Network Name Resolution...
Jun 23 04:27:33 Oracle01 systemd-resolved[1461]: Positive Trust Anchors:
Jun 23 04:27:33 Oracle01 systemd-resolved[1461]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Jun 23 04:27:33 Oracle01 systemd-resolved[1461]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Jun 23 04:27:33 Oracle01 systemd-resolved[1461]: Using system hostname 'Oracle01'.
Jun 23 04:27:33 Oracle01 systemd[1]: Started Network Name Resolution.

 2: eth0                                                                      
                     Link File: /usr/lib64/systemd/network/98-virtio.link      
                  Network File: /usr/lib64/systemd/network/zz-default.network  
                          Type: ether                                          
                         State: routable (configured)
             Alternative Names: enp0s3                                         
                                ens3                                           
                          Path: pci-0000:00:03.0                               
                        Driver: virtio_net                                     
                        Vendor: Red Hat, Inc                                   
                         Model: Virtio network device                          
                    HW Address: 02:00:17:01:b9:ca                              
                           MTU: 9000 (min: 68, max: 65535)                     
                         QDisc: fq                                             
  IPv6 Address Generation Mode: eui64                                          
          Queue Length (Tx/Rx): 1/1                                            
              Auto negotiation: no                                             
                         Speed: n/a                                            
                       Address: 10.0.0.9 (DHCP4 via 169.254.169.254)           
                                fe80::17ff:fe01:b9ca                           
                       Gateway: 10.0.0.1 (Oracle)                              
                           DNS: 169.254.169.254                                
                Search Domains: vcn.oraclevcn.com                              
               DHCP4 Client ID: IAID:0xb55e67ff/DUID                           
             DHCP6 Client DUID: DUID-EN/Vendor:0000ab11431ff6f40877b52f0000    

Jun 23 04:02:08 Oracle01 systemd-networkd[661]: eth0: Link UP
Jun 23 04:02:08 Oracle01 systemd-networkd[661]: eth0: Gained carrier
Jun 23 04:02:08 Oracle01 systemd-networkd[661]: eth0: DHCPv4 address 10.0.0.9/24 via 10.0.0.1
Jun 23 04:02:08 Oracle01 systemd-networkd[661]: eth0: Classless static routes received from DHCP server: ignoring router option
Jun 23 04:02:10 Oracle01 systemd-networkd[661]: eth0: Gained IPv6LL
pothos commented 3 years ago

There are these Failed to emit notification about changed property DNS: Transport endpoint is not connected errors and then the service suddenly stopps and isn't restarted? That's strange because it has Restart=always set. Can you share more about your setup (Ignition configuration or any other changes)?

Maybe for debugging it helps to add this as a unit drop-in file:

[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

While it runs, what is the output of resolvectl?

xh116 commented 3 years ago

Just rebooted with creating debugging config. (Now tested on another machine. )

iginition file 

{
    "ignition":
    {
        "config":
        {},
        "security":
        {
            "tls":
            {}
        },
        "timeouts":
        {},
        "version": "2.2.0"
    },
    "networkd":
    {},
    "passwd":
    {
        "users":
        [
            {
                "name": "core",
                "sshAuthorizedKeys":
                [
                    "ssh xxx"
                 ]
            }
        ]
    },
    "storage":
    {
        "files":
        [
            {
                "filesystem": "root",
                "path": "/etc/hostname",
                "contents":
                {
                    "source": "data:,Oracle",
                    "verification":
                    {}
                },
                "mode": 420
            },
            {
                "filesystem": "root",
                "path": "/etc/sysctl.d/80-swappiness.conf",
                "contents":
                {
                    "source": "data:,vm.swappiness%3D10",
                    "verification":
                    {}
                },
                "mode": 420
            },
            {
                "filesystem": "root",
                "path": "/etc/sysctl.d/congest_control.conf",
                "contents":
                {
                    "source": "data:,net.core.default_qdisc%3Dfq%0Anet.ipv4.tcp_congestion_control%3Dbbr%0Anet.ipv4.tcp_slow_start_after_idle%3D0%0A",
                    "verification":
                    {}
                },
                "mode": 420
            }
        ]
    },
    "systemd":
    {
        "units":
        [
            {
                "enabled": true,
                "name": "docker.service"
            },
            {
                "contents": "[Unit]\nDescription=Turn on swap\nRequires=create-swapfile.service\nAfter=create-swapfile.service\n\n[Swap]\nWhat=/var/vm/swapfile\n\n[Install]\nWantedBy=multi-user.target        \n",
                "enabled": true,
                "name": "var-vm-swapfile.swap"
            },
            {
                "contents": "[Unit]\nDescription=Create a swapfile\nRequiresMountsFor=/var\nConditionPathExists=!/var/vm/swapfile\n\n[Service]\nType=oneshot\nExecStart=/usr/bin/mkdir -p /var/vm\nExecStart=/usr/bin/fallocate -l 1024m /var/vm/swapfile\nExecStart=/usr/bin/chmod 600 /var/vm/swapfile\nExecStart=/usr/sbin/mkswap /var/vm/swapfile\nRemainAfterExit=true           ",
                "name": "create-swapfile.service"
            }
        ]
    }
}

debugging info for systemd-resolved


1. mkdir -p /etc/systemd/system/systemd-resolved.service.d
2. vi /etc/systemd/system/systemd-resolved.service.d/10-debug.conf
3. [Service]
4. Environment=SYSTEMD_LOG_LEVEL=debug
5. systemctl daemon-reload  
6. systemctl restart systemd-resolved
7. sudo reboot 
8. journalctl -u systemd-resolved --no-pager

output 

Jun 23 04:38:39 Oracle02 systemd[1]: Starting Network Name Resolution...
Jun 23 04:38:40 Oracle02 systemd-resolved[694]: Positive Trust Anchors:
Jun 23 04:38:40 Oracle02 systemd-resolved[694]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Jun 23 04:38:40 Oracle02 systemd-resolved[694]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Jun 23 04:38:40 Oracle02 systemd-resolved[694]: Using system hostname 'Oracle02'.
Jun 23 04:38:40 Oracle02 systemd[1]: Started Network Name Resolution.
Jun 23 14:13:20 Oracle02 systemd[1]: Stopping Network Name Resolution...
Jun 23 14:13:20 Oracle02 systemd[1]: systemd-resolved.service: Succeeded.
Jun 23 14:13:20 Oracle02 systemd[1]: Stopped Network Name Resolution.
Jun 23 14:13:20 Oracle02 systemd[1]: Starting Network Name Resolution...
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: SELinux enabled state cached to: enabled
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Successfully loaded SELinux database in 361us, size on heap is 75K.
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Positive Trust Anchors:
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Using system hostname 'Oracle02'.
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: New scope on link *, protocol dns, family *
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Found new link 6/veth06905af
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Found new link 4/br-eb076133b8f7
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Found new link 3/docker0
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Found new link 2/eth0
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Found new link 1/lo
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: New scope on link eth0, protocol dns, family *
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Bus bus-api-resolve: changing state UNSET → OPENING
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: sd-bus: starting bus bus-api-resolve by connecting to /run/dbus/system_bus_socket...
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Bus bus-api-resolve: changing state OPENING → AUTHENTICATING
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Registering bus object implementation for path=/org/freedesktop/resolve1 iface=org.freedesktop.resolve1.Manager
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Registering bus object implementation for path=/org/freedesktop/resolve1/link iface=org.freedesktop.resolve1.Link
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Registering bus object implementation for path=/org/freedesktop/resolve1/dnssd iface=org.freedesktop.resolve1.DnssdService
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Registering bus object implementation for path=/org/freedesktop/LogControl1 iface=org.freedesktop.LogControl1
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Creating stub listener using UDP/TCP.
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Bus bus-api-resolve: changing state AUTHENTICATING → HELLO
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=Hello cookie=1 reply_cookie=0 signature=n/a error-name=n/a error-message=n/a
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=RequestName cookie=2 reply_cookie=0 signature=su error-name=n/a error-message=n/a
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=AddMatch cookie=3 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Jun 23 14:13:20 Oracle02 systemd[1]: Started Network Name Resolution.
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Got message type=method_return sender=org.freedesktop.DBus destination=:1.24 path=n/a interface=n/a member=n/a cookie=1 reply_cookie=1 signature=s error-name=n/a error-message=n/a
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Bus bus-api-resolve: changing state HELLO → RUNNING
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Got message type=signal sender=org.freedesktop.DBus.Local destination=n/a path=/org/freedesktop/DBus/Local interface=org.freedesktop.DBus.Local member=Connected cookie=4294967295 reply_cookie=0 signature=n/a error-name=n/a error-message=n/a
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Got message type=signal sender=org.freedesktop.DBus destination=:1.24 path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=NameAcquired cookie=2 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Got message type=signal sender=org.freedesktop.DBus destination=:1.24 path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=NameAcquired cookie=3 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Got message type=method_return sender=org.freedesktop.DBus destination=:1.24 path=n/a interface=n/a member=n/a cookie=4 reply_cookie=2 signature=u error-name=n/a error-message=n/a
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Successfully acquired requested service name.
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Got message type=method_return sender=org.freedesktop.DBus destination=:1.24 path=n/a interface=n/a member=n/a cookie=5 reply_cookie=3 signature=n/a error-name=n/a error-message=n/a
Jun 23 14:13:20 Oracle02 systemd-resolved[10646]: Match type='signal',sender='org.freedesktop.login1',path='/org/freedesktop/login1',interface='org.freedesktop.login1.Manager',member='PrepareForSleep' successfully installed.
Jun 23 14:13:28 Oracle02 systemd-resolved[10646]: Bus bus-api-resolve: changing state RUNNING → CLOSING
Jun 23 14:13:28 Oracle02 systemd-resolved[10646]: Bus bus-api-resolve: changing state CLOSING → CLOSED
Jun 23 14:13:29 Oracle02 systemd-resolved[10646]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 14:13:29 Oracle02 systemd-resolved[10646]: Found new link 5/veth69b3800
Jun 23 14:13:29 Oracle02 systemd-resolved[10646]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 14:13:29 Oracle02 systemd-resolved[10646]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 14:13:29 Oracle02 systemd-resolved[10646]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 14:13:29 Oracle02 systemd-resolved[10646]: Removing link 5/veth69b3800
Jun 23 14:13:29 Oracle02 systemd-resolved[10646]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 14:13:29 Oracle02 systemd-resolved[10646]: Removing link 6/veth06905af
Jun 23 14:13:29 Oracle02 systemd-resolved[10646]: Failed to emit notification about changed property DNS: Transport endpoint is not connected
Jun 23 14:13:29 Oracle02 systemd[1]: Stopping Network Name Resolution...
Jun 23 14:13:29 Oracle02 systemd-resolved[10646]: Removing scope on link eth0, protocol dns, family *
Jun 23 14:13:29 Oracle02 systemd-resolved[10646]: Removing scope on link *, protocol dns, family *
Jun 23 14:13:29 Oracle02 systemd[1]: systemd-resolved.service: Succeeded.
Jun 23 14:13:29 Oracle02 systemd[1]: Stopped Network Name Resolution.

Yes, systemd-resolved may start up itself but mostly it will not work.

resolvectl status will print : Failed to get global data: Unit dbus-org.freedesktop.resolve1.service not found.

xh116 commented 3 years ago

Waited about half an hour, it doesn't restart itself. so manually restart it. 

-- Boot 9114905ac90d48389f25bf26ebe5a213 --
Jun 23 14:47:53 Oracle02 systemd[1]: Starting Network Name Resolution...
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: SELinux enabled state cached to: enabled
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Successfully loaded SELinux database in 408us, size on heap is 75K.
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Positive Trust Anchors:
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Using system hostname 'Oracle02'.
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: New scope on link *, protocol dns, family *
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Found new link 6/vethfae97d0
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Found new link 4/br-eb076133b8f7
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Found new link 3/docker0
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Found new link 2/eth0
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Found new link 1/lo
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: New scope on link eth0, protocol dns, family *
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Bus bus-api-resolve: changing state UNSET → OPENING
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: sd-bus: starting bus bus-api-resolve by connecting to /run/dbus/system_bus_socket...
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Bus bus-api-resolve: changing state OPENING → AUTHENTICATING
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Registering bus object implementation for path=/org/freedesktop/resolve1 iface=org.freedesktop.resolve1.Manager
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Registering bus object implementation for path=/org/freedesktop/resolve1/link iface=org.freedesktop.resolve1.Link
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Registering bus object implementation for path=/org/freedesktop/resolve1/dnssd iface=org.freedesktop.resolve1.DnssdService
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Registering bus object implementation for path=/org/freedesktop/LogControl1 iface=org.freedesktop.LogControl1
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Creating stub listener using UDP/TCP.
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Bus bus-api-resolve: changing state AUTHENTICATING → HELLO
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=Hello cookie=1 reply_cookie=0 signature=n/a error-name=n/a error-message=n/a
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=RequestName cookie=2 reply_cookie=0 signature=su error-name=n/a error-message=n/a
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=AddMatch cookie=3 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Jun 23 14:47:53 Oracle02 systemd[1]: Started Network Name Resolution.
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Got message type=method_return sender=org.freedesktop.DBus destination=:1.22 path=n/a interface=n/a member=n/a cookie=1 reply_cookie=1 signature=s error-name=n/a error-message=n/a
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Bus bus-api-resolve: changing state HELLO → RUNNING
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Got message type=signal sender=org.freedesktop.DBus.Local destination=n/a path=/org/freedesktop/DBus/Local interface=org.freedesktop.DBus.Local member=Connected cookie=4294967295 reply_cookie=0 signature=n/a error-name=n/a error-message=n/a
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Got message type=signal sender=org.freedesktop.DBus destination=:1.22 path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=NameAcquired cookie=2 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Got message type=signal sender=org.freedesktop.DBus destination=:1.22 path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=NameAcquired cookie=3 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Got message type=method_return sender=org.freedesktop.DBus destination=:1.22 path=n/a interface=n/a member=n/a cookie=4 reply_cookie=2 signature=u error-name=n/a error-message=n/a
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Successfully acquired requested service name.
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Got message type=method_return sender=org.freedesktop.DBus destination=:1.22 path=n/a interface=n/a member=n/a cookie=5 reply_cookie=3 signature=n/a error-name=n/a error-message=n/a
Jun 23 14:47:53 Oracle02 systemd-resolved[1897]: Match type='signal',sender='org.freedesktop.login1',path='/org/freedesktop/login1',interface='org.freedesktop.login1.Manager',member='PrepareForSleep' successfully installed.

Global
       Protocols: -LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: uplink                                        

Link 2 (eth0)
Current Scopes: DNS                                                         
     Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
   DNS Servers: 169.254.169.254                                             
    DNS Domain: vcn.oraclevcn.com                                           

Link 3 (docker0)
Current Scopes: none                                                        
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 4 (br-eb076133b8f7)
Current Scopes: none                                                        
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 6 (vethfae97d0)
Current Scopes: none                                                        
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
pothos commented 3 years ago

Thank you for the quick follow-ups! It's quite strange… we need to file an upstream bug report (or find an existing one) and also check if this is reproducible with the latest systemd version. Unfortunately we had to hold back a newer version in Alpha as long as we are on cgroup v1.

One more question, when the service terminates, do the files in /run/systemd/resolve/ still exist? Then it could work to ensure that the symlink goes to the merged resolv.conf (169.254.169.254) and not the stub resolv.conf (127.0.0.53). In your case you can even try to create /etc/resolv.conf as a normal file and hardcode the 169.254.169.254 server as long as the bug is present.

xh116 commented 3 years ago

Thanks for reply also. Only systemd-resolved is running, the files /run/systemd/resolve/ exist. or missing.

For now, I think I can only try to do it manually.

Thanks again.

xh116 commented 3 years ago

I confirm the problem should be with the ignition file. The way to create swap cause the problem. () After reinstalling with following, problem solved. 

storage:
  disks:
    - device: /dev/sda
      wipe_table: false
      partitions:
        - label : SWAP
          number : 10 
          start: 46GiB
          type_guid: swap_partition
  filesystems:
    - name: ROOT
      mount:
        device: /dev/disk/by-partlabel/ROOT
        format: xfs
        wipe_filesystem: true
        label: ROOT
    - name: SWAP 
      mount:
        device: /dev/disk/by-partlabel/SWAP 
        format: swap
        wipe_filesystem: true
        label: SWAP

this should be a bug.