SSH Key not work after some minutes

[infnada]

Issue Report

Bug

On node restart I'm able to login using SSH key but not after 2/3 minutes. I've got 4 SSH keys and non of them works with the error (putty):

login as: XXXX
Server refused our key

The key is in ~/.ssh/authorized_keys & ~/.ssh/authorized_keys.d/coreos-cloudinit

Container Linux Version

$ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1967.6.0
VERSION_ID=1967.6.0
BUILD_ID=2019-02-12-2138
PRETTY_NAME="Container Linux by CoreOS 1967.6.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

Environment

VMware ESXi, 6.5.0, 5310538 Default VM options.

Configured with:

coreos-install -d /dev/sda -c ci.yml -o vmware_raw

#cloud-config
coreos:
  update:
    reboot-strategy: "off"
  units:
    - name: systemd-timesyncd.service
      command: stop
      mask: true
    - name: ntpd.service
      command: start
      enable: true
    - name: systemd-networkd.service
      command: stop
    - name: down-interfaces.service
      command: start
      content: |
        [Service]
        Type=oneshot
        ExecStart=/usr/bin/ip link set ens192 down
        ExecStart=/usr/bin/ip addr flush dev ens192
    ############################### CHANGE ###############################
    ############################### CHANGE ###############################
    ############################### CHANGE ###############################
    ############################### CHANGE ###############################
    - name: 00-ens192.network
      runtime: true
      content: |
        [Match]
        Name=ens192

        [Network]
        DNS= XXX
        Address= XXX
        Gateway=XXX
    - name: systemd-networkd.service
      command: restart
    - name: docker.service
      drop-ins:
        - name: 10-memlock.conf
          content: |
            [Service]
            LimitMEMLOCK=infinity
    - name: docker.service
      command: start
    - name: runsysctl.service
      command: start
      content: |
        [Unit]
        Description=Set Max Map Count

        [Service]
        Type=oneshot
        ExecStart=/usr/sbin/sysctl -w vm.max_map_count=16777216
    - name: runcmd.service
      command: start
      content: |
        [Unit]
        Description=Install vsphere plugin

        [Service]
        Type=oneshot
        ExecStart=/usr/bin/docker plugin ls | grep -q 'vsphere' && echo "matched" || /usr/bin/docker plugin install --grant-all-permissions --alias vsphere vmware/vsphere-storage-for-docker:latest "VDVS_SOCKET_GID=233"
        ExecStart=/usr/bin/docker plugin ls | grep -q 'vfile' && echo "matched" || /usr/bin/docker plugin install --grant-all-permissions --alias vfile vmware/vfile:latest VFILE_TIMEOUT_IN_SECOND=90 "VDVS_SOCKET_GID=233"
ssh_authorized_keys:
  - "XXXX"
############################### CHANGE ###############################
############################### CHANGE ###############################
############################### CHANGE ###############################
############################### CHANGE ###############################
hostname: "XXXXX"
users:
  - name: "core"
    passwd: "XXX"
    groups:
      - "sudo"
      - "docker"
  - name: "XXX"
    groups:
      - "sudo"
      - "docker"
    ssh-authorized-keys:
      - "XXXX"
write_files:
  - path: "/etc/resolv.conf"
    permissions: "0644"
    owner: "root"
    content: |
      nameserver XXXX
      nameserver XXXX
  - path: /etc/ntp.conf
    content: |
      server XXXXX

      # - Allow only time queries, at a limited rate.
      # - Allow all local queries (IPv4, IPv6)
      restrict default nomodify nopeer noquery limited kod
      restrict 127.0.0.1
      restrict [::1]
  - path: /etc/sysctl.conf
    content: |
      vm.max_map_count=262144
  - path: /etc/vfile.conf
    content: |
      {
        "MaxLogAgeDays": 28,
        "MaxLogFiles": 10,
        "MaxLogSizeMb": 10,
        "LogPath": "/var/log/vfile.log"
      }
  - path: /etc/vsphere-storage-for-docker.conf
    content: |
      {
        "MaxLogAgeDays": 28,
        "MaxLogFiles": 10,
        "MaxLogSizeMb": 10,
        "LogPath": "/var/log/vsphere-storage-for-docker.log"
      }
  - path: /etc/security/limits.conf
    content: |
      *            hard   memlock           unlimited
      *            soft   memlock           unlimited
  - path: /etc/sysctl.conf
    content: |
      vm.max_map_count=16777216

Expected Behavior

Be able to login with SSH using some of cloud-config provided keys

Actual Behavior

Can't login after 2/3 minutes of node restart.

Reproduction Steps

1. Restart

2. Instant login -> OK

3. Wait 2/3 minutes -> Login fail

4. Restart

5. Wait 2/3 minutes -> Login fail

[bgilbert]

Is this for the core user or the other one? Is there a previous version of Container Linux where this was working properly?

[infnada]

This is for every user. Have 4 new users + core user and none of them can login. I'm new to CoreOs never tried it before.

[bgilbert]

If you're just starting with Container Linux, please use a Container Linux Config rather than a cloud-config. coreos-cloudinit is long deprecated and has unfixable race conditions. It seems likely that you're seeing one of them here.

[infnada]

Can i just delete /var/lib/coreos-install/user_data, copy the new ignition.json somewhere and reboot the VM?

[bgilbert]

At this point you should start with a new VM. Ignition only runs on the first boot, and anyway coreos-cloudinit has already made a bunch of changes to your system.

[infnada]

Exact same issue:

{
  "ignition": {
    "config": {},
    "timeouts": {},
    "version": "2.1.0"
  },
  "networkd": {
    "units": [
      {
        "contents": "[Match]\nName=ens192\n\n[Network]\nAddress=xxxxxx/24\nGateway=xxxxx\nDNS=xxxxx",
        "name": "static.network"
      }
    ]
  },
  "passwd": {
    "users": [
      {
        "groups": [
          "sudo",
          "docker"
        ],
        "name": "core",
        "passwordHash": "xxxxx",
        "sshAuthorizedKeys": [
          "ssh-rsa xxxxxxx"
        ]
      },
      {
        "groups": [
          "sudo",
          "docker"
        ],
        "name": "xxxx",
        "sshAuthorizedKeys": [
          "ssh-rsa xxxxx"
        ]
      },
      {
        "groups": [
          "sudo",
          "docker"
        ],
        "name": "xxx",
        "sshAuthorizedKeys": [
          "ssh-rsa xxxxx"
        ]
      },
      {
        "groups": [
          "sudo",
          "docker"
        ],
        "name": "xxxx",
        "sshAuthorizedKeys": [
          "ssh-rsa xxxx"
        ]
      },
      {
        "groups": [
          "sudo",
          "docker"
        ],
        "name": "xxxx",
        "sshAuthorizedKeys": [
          "ssh-rsa xxxxxx"
        ]
      }
    ]
  },
  "storage": {
    "files": [
      {
        "filesystem": "root",
        "group": {},
        "path": "/etc/coreos/update.conf",
        "user": {},
        "contents": {
          "source": "data:,GROUP%3Dstable%0AREBOOT_STRATEGY%3D%22off%22%0ASERVER%3Dhttps%3A%2F%2Fpublic.update.core-os.net%2Fv1%2Fupdate%2F%0A",
          "verification": {}
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "group": {},
        "path": "/etc/hostname",
        "user": {},
        "contents": {
          "source": "data:,xxxxx%0A",
          "verification": {}
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "group": {},
        "path": "/etc/resolv.conf",
        "user": {},
        "contents": {
          "source": "data:,nameserver%09xxxxx%0Anameserver%09xxxx%0A",
          "verification": {}
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "group": {},
        "path": "/etc/sysctl.conf",
        "user": {},
        "contents": {
          "source": "data:,vm.max_map_count=262144%0A",
          "verification": {}
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "group": {},
        "path": "/etc/vfile.conf",
        "user": {},
        "contents": {
          "source": "data:,%7B%22MaxLogAgeDays%22%3A28%2C%22MaxLogFiles%22%3A10%2C%22MaxLogSizeMb%22%3A10%2C%22LogPath%22%3A%22%2Fvar%2Flog%2Fvfile%2Elog%22%7D%0A",
          "verification": {}
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "group": {},
        "path": "/etc/vsphere-storage-for-docker.conf",
        "user": {},
        "contents": {
          "source": "data:,%7B%22MaxLogAgeDays%22%3A28%2C%22MaxLogFiles%22%3A10%2C%22MaxLogSizeMb%22%3A10%2C%22LogPath%22%3A%22%2Fvar%2Flog%2Fvsphere%2Dstorage%2Dfor%2Ddocker%2Elog%22%7D%0A",
          "verification": {}
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "group": {},
        "path": "/etc/security/limits.conf",
        "user": {},
        "contents": {
          "source": "data:,%2A%09hard%09memlock%09unlimited%0A%2A%09soft%09memlock%09unlimited%0A",
          "verification": {}
        },
        "mode": 420
      }
    ]
  },
  "systemd": {
    "units": [
      {
        "enable": true,
        "dropins": [
          {
            "contents": "[Service]\nLimitMEMLOCK=infinity",
            "name": "10-memlock.conf"
          }
        ],
        "name": "docker.service"
      },
      {
        "contents": "[Unit]\nDescription=Set Max Map Count\n\n[Service]\nType=oneshot\nExecStart=/usr/sbin/sysctl -w vm.max_map_count=16777216\n\n[Install]\nWantedBy=multi-user.target",
        "enable": true,
        "name": "runsysctl.service"
      },
      {
        "contents": "[Unit]\nDescription=Install vsphere plugin\nAfter=docker.service\nRequires=docker.service\n\n[Service]\nType=oneshot\nExecStart=/usr/bin/docker plugin ls | grep -q 'vsphere' && echo \"matched\" || /usr/bin/docker plugin install --alias vsphere vmware/vsphere-storage-for-docker:latest --grant-all-permissions \"VDVS_SOCKET_GID=233\"\nExecStart=/usr/bin/docker plugin ls | grep -q 'vfile' && echo \"matched\" || /usr/bin/docker plugin install --alias vfile vmware/vfile:latest VFILE_TIMEOUT_IN_SECOND=90 \"VDVS_SOCKET_GID=233\" --grant-all-permissions\n\n[Install]\nWantedBy=multi-user.target",
        "enable": true,
        "name": "runcmd.service"
      }
    ]
  }
}