search

coreos / bugs

Issue tracker for CoreOS Container Linux
https://coreos.com/os/eol/
147 stars 30 forks source link

Error mounting NVME device on m5d instances #2613

Open dhawal55 opened 5 years ago

dhawal55 commented 5 years ago

Issue Report

I'm trying to provision a m5d instance that comes with NVMe storage. In my ignition config, I try to format the disk as follows:

filesystems:
  - name: etcddatafs
    mount:
      device: /dev/nvme1n1
      format: ext4
      wipe_filesystem: true

However, it fails with the following error:

[0;1;31mdisks: createFilesystems: op(f): op(11): op(13): [failed]   creating "ext4" filesystem on "/run/ignition/dev_aliases/dev/nvme1n1": exit status 1: Cmd: "/usr/sbin/mkfs.ext4" "-F" "/run/ignition/dev_aliases/dev/nvme1n1" Stdout: "" Stderr: "mke2fs 1.44.5 (15-Dec-2018)
Found a gpt partition table in /run/ignition/dev_aliases/dev/nvme1n1
/run/ignition/dev_aliases/dev/nvme1n1 is apparently in use by the system; will not make a filesystem here!\n"

Bug

Container Linux Version

2191.5.0

Environment

AWS m5d.xlarge instance

Expected Behavior

/dev/nvme1n1 should be formatted and mounted

Actual Behavior

Format fails for nvme

Other Information

Full system log:


[   37.556774] audit: type=1131 audit(1568330461.401:21): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-vconsole-setup comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[  OK  ] Started File System Check on /dev/disk/by-label/ROOT.
[   37.276794] ignition[846]:     "disks": [
[   37.563522] audit: type=1130 audit(1568330461.410:22): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-fsck-root comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[  OK  ] Started [0;[   37.568427] audit: type=1130 audit(1568330461.417:23): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=verity-setup comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
1;39mVerity Setup for /dev/mapper/usr.
[   37.283428] ignition[846]:       {
[   37.289779] ignition[846]:         "device": "/dev/nvme1n1",
[   37.290935] ignition[846]:         "partitions": [
[   37.291902] ignition[846]:           {
[   37.292687] ignition[846]:             "label": "ETCDDATAFS",
[   37.293843] ignition[846]:             "size": 0,
[   37.294786] ignition[846]:             "start": 0
[   37.295742] ignition[846]:           }
[   37.296532] ignition[846]:         ],
[  OK  ] Found device /dev/mapper/usr.
[   37.297328] ignition         [846]: Mounting /sysroot...        "wipeTable": true

         Starting Emergency Shell...
[   37.299878] ignition[846]:       },
[   37.300789] ignition[846]:       {
[   37.301461] ignition[846]:         "device": "/dev/xvdp",
[   37.302137] ignition[846]:         "partitions": [
[   37.302594] ignition[846]:           {
[   37.303048] ignition[846]:             "label": "IMAGEFS",
[   37.303508] ignition[846]:             "size": 0,
[   37.303959] ignition[846]:             "start": 0
[   37.304420] ignition[846]:           }
[   37.304872] ignition[846]:         ],
[   37.305114] ignition[846]:         "wipeTable": true
[   37.305567] ignition[846]:       },
[   37.305824] ignition[846]:       {
[   37.306268] ignition[846]:         "device": "/dev/xvdc",
[   37.306501] ignition[846]:         "partitions": [
[   37.306732] ignition[846]:           {
[   37.306963] ignition[846]:             "label": "NODEFS",
[   37.307407] ignition[846]:             "size": 0,
[   37.307644] ignition[846]:             "start": 0
[   37.307868] ignition[846]:           }
[   37.308096] ignition[846]:         ],
[   37.308541] ignition[846]:         "wipeTable": true
[   37.308775] ignition[846]:       }
[   37.309000] ignition[846]:     ],
[   37.309443] ignition[846]:     "files": [
[   37.309674] ignition[846]:       {
[   37.309899] ignition[846]:         "filesystem": "root",
[   37.310125] ignition[846]:         "path": "/etc/kubernetes/kubelet/kubeconfig",
[   37.319978] ignition[846]:         "contents": {
[   37.323192] ignition[846]:           "source": "data:,apiVersion%3A%20v1%0Akind%3A%20Config%0Aclusters%3A%0A-%20name%3A%20local%0A%20%20cluster%3A%0A%20%20%20%20server%3A%20https%3A%2F%2Faws-9405.platform-dev.r53.nordstrom.net%0A%20%20%20%20certificate-authority%3A%20%2Fetc%2Fkubernetes%2Fca.crt%0Ausers%3A%0A-%20name%3A%20kubelet%0A%20%20user%3A%0A%20%20%20%20client-certificate%3A%20%2Fetc%2Fkubernetes%2Ftls.crt%0A%20%20%20%20client-key%3A%20%2Fetc%2Fkubernetes%2Ftls.key%0Acontexts%3A%0A-%20context%3A%0A%20%20%20%20cluster%3A%20local%0A%20%20%20%20user%3A%20kubelet%0A",
[   37.323283] ignition[846]:           "verification": {}
[   37.323754] ignition[846]:         },
[   37.324221] ignition[846]:         "mode": 420
[   37.324676] ignition[846]:       },
[   37.325135] ignition[846]:       {
[   37.325600] igni[   37.618585] EXT4-fs (nvme0n1p9): mounted filesystem with ordered data mode. Opts: (null)
tion[846]:         "filesystem": "root",
[   37.325833] ignition[846]:         "path": "/etc/torcx/profiles/combined.json",
[   37.326224] ignition[846]:         "contents": {
[   37.326598] ignition[846]:           "source": "data:,%7B%22kind%22%3A%22profile-manifest-v0%22%2C%22value%22%3A%7B%22images%22%3A%5B%7B%22name%22%3A%22docker%22%2C%22reference%22%3A%2217.03%22%7D%2C%7B%22name%22%3A%22crio%22%2C%22reference%22%3A%22v1.14.6a%22%7D%5D%7D%7D%0A",
[   37.326973] ignition[846]:           "verification": {}
[   37.327374] ignition[846]:         },
[   37.328436] ignition[846]:         "mode": 420
[   37.328742] ignition[846]:       },
[   37.329120] ignition[846]:       {
[   37.329524] ignition[846]:         "filesystem": "root",
[   37.329894] ignition[846]:         "path": "/etc/torcx/next-profile",
[   37.330102] ignition[846]:         "contents": {
[   37.330515] ignition[846]:           "source": "data:,combined%0A",
[   37.330702] ignition[846]:           "verification": {}
[   37.331092] ignition[846]:         },
[   37.331469] ignition[846]:         "mode": 420
[   37.331880] ignition[846]:       },
[   37.332351] ignition[846]:       {
[   37.332737] ignition[846]:         "filesystem": "root",
[   37.333128] ignition[846]:         "path": "/var/lib/torcx/store/docker:17.03.torcx.tgz",
[   37.333536] ignition[846]:         "contents": {
[   37.336041] ignition[846]:           "source": "https://tectonic-torcx.release.core-os.net/pkgs/amd64-usr/docker/e3819eef5185486cd192ae03108003695e98e9cea4bc4b8082a0a79f212fc7eda263996c78fdac3690b0f2ebbfbdf5632f654483cc8bcd30bfb165ed9d322fbf/docker:17.03.torcx.tgz",[  OK  ] 
Mounted /sysroot.
[   37.336524] ignition[846]:           "verification": {
[   37.336981] ignition[846]:             "hash": "sha512-e3819eef5185486cd192ae03108003695e98e9cea4bc4b8082a0a79f212fc7eda263996c78fdac3690b0f2ebbfbdf5632f654483cc8bcd30bfb165ed9d322fbf"
         Mounting /sysroot/usr...
[  OK  ] Reached target Initrd Root File System.
[   37.337458] ignition[846]:           }
[   37.338390] ignition[846]:         },
[   37.338842] ignition[846]:         "mode": 420
[   37.339304] ignition[846]:       },
[   37.339[   37.649533] EXT4-fs (dm-0): mounted filesystem without journal. Opts: (null)
535] ignition[846]:       {
[   37.339765] ignition[846]:         "filesystem": "root",
[   37.339973] ignition[846]:         "path": "/etc/docker/daemon.json",
[   37.340373] ignition[846]:         "contents": {
[   37.340762] ignition[846]:           "source": "data:,%7B%0A%20%20%22metrics-addr%22%20%3A%20%220.0.0.0%3A9323%22%2C%0A%20%20%22experimental%22%20%3A%20true%0A%7D%0A",
[   37.341160] ignition[846]:           "verification": {}
[   37.341561] ignition[846]:         },
[   37.341941] ignition[846]:         "mode": 420
[   37.342332] ignition[846]:       },
[   37.342711] ignition[846]:       {
[   37.343101] ignition[846]:         "filesystem": "root",
[   37.343502] ignition[846]:         "path": "/var/lib/torcx/store/crio:v1.14.6a.torcx.tgz",
[   37.343868] ignition[846]:         "contents": {
[   37.344080] ignition[846]:           "source": "s3://platform-bootstrap-dev/binaries/crio:v1.14.6a.torcx.tgz",
[   37.344457] ignition[846]:           "verification": {}
[   37.344847] ignition[846]:         },
[   37.345240] ignition[846]:         "mode": 420
[   37.345614] ignition[846]:       },
[   37.345991] ignition[846]:       {
[   37.346388] ignition[846]:         "filesystem": "root",
[   37.346804] ignition[846]:         "path": "/etc/crio/crio.conf",
[   37.347197] ignition[846]:         "contents": {
[   37.347575] ignition[846]:           "source": "s3://platform-bootstrap-dev/binaries/etccrio/v1.14.6a/crio.conf",
[   37.347958] ignition[846]:           "verification": {}
[   37.348377] ignition[846]:         },
[   37.348768] ignition[846]:         "mode": 420
[   37.349170] ignition[846]:       },
[   37.349535] ignition[846]:       {
[   37.349743] ignition[846]:         "filesystem": "root",
[   37.350141] ignition[846]:         "path": "/etc/crio/seccomp.json",
[   37.350544] ignition[846]:         "contents": {
[   37.350923] ignition[846]:           "source": "s3://platform-bootstrap-dev/binaries/etccrio/v1.14.6a/seccomp.json",
[   37.351305] ignition[846]:           "verification": {}
[   37.351714] ignition[846]:         },
[   37.352083] ignition[846]:         "mode": 420
[   37.352481] ignition[846]:       },
[   37.352895] ignition[846]:       {
[   37.353110] ignition[846]:         "filesystem": "root",
[   37.353562] ignition[846]:         "path": "/opt/bin/generate_certs.sh",
[   37.354077] systemd[1]: ignition-disks.service: Main process exited, code=exited, status=1/FAILURE
[   37.354422] ignition[846]:         "contents": {
[   37.354696] ignition[846]:           "source": "s3://platform-bootstrap-dev-us-west-2/platform-instances/aws-9405/artifacts/generate_certs.sh",
[   37.354889] ignition[846]:           "verification": {}
[   37.355269] ignition[846]:         },
[   37.355656] ignition[846]:         "mode": 493
[   37.356027] ignition[846]:       },
[   37.356430] ignition[846]:       {
[   37.356621] ignition[846]:         "filesystem": "root",
[   37.356996] ignition[846]:         "path": "/root-ca/etcd-ca.crt",
[   37.357415] ignition[846]:         "contents": {
[   37.357654] ignition[846]:           "source": "s3://platform-bootstrap-dev-us-west-2/platform-instances/aws-9405/etcd-ca.crt",
[   37.358042] ignition[846]:           "verification": {}
[   37.358432] ignition[846]:         },
[   37.358790] ignition[846]:         "mode": 420
[   37.358983] ignition[846]:       },
[   37.359375] ignition[846]:       {
[   37.359755] ignition[846]:         "filesystem": "root",
[   37.360127] ignition[846]:         "path": "/root-ca/etcd-ca.key",
[   37.360536] ignition[846]:         "contents": {
[   37.360964] ignition[846]:           "source": "s3://platform-bootstrap-dev-us-west-2/platform-instances/aws-9405/etcd-ca.key",
[   37.361380] ignition[846]:           "verification": {}
[   37.361819] ignition[846]:         },
[   37.362285] ignition[846]:         "mode": 420
[   37.362745] ignition[846]:       },[
  OK  ] Mounted /sysroot/usr.
[   37.411095] ignition[846]:       {
[  OK  ] Reached target Local File Systems.
[   37.411428] ignition[846]:         "filesystem": "root",
[   37.411815] ignition[846]:         "path": "/root-ca/k8s-ca.crt",
[   37.412203] ignition[846]:         "contents": {
[   37.412573] ignition[846]:           "source": "s3://platform-bootstrap-dev-us-west-2/platform-root-ca/k8s-ca.crt",
[   37.412962] ignition[846]:           "verification": {}
[   37.413348] ignition[846]:         },
[   37.413721] ignition[846]:         "mode": 420
[   37.414092] ignition[846]:       },
[   37.414477] ignition[846]:       {
[   37.414845] ignition[846]:         "filesystem": "root",
[   37.415426] ignition[846]:         "path": "/root-ca/k8s-ca.key",
[   37.415810] ignition[846]:         "contents": {
[   37.416190] ignition[846]:           "source": "s3://platform-bootstrap-dev-us-west-2/platform-root-ca/k8s-ca.key",
[   37.416556] ignition[846]:           "verification": {}
[   37.416930] ignition[846]:         },
[   37.417313] ignition[846]: [   37.714200] serial8250: too much work for irq4
        "mode": 420
[   37.417696] ignition[846]:       },
[   37.418082] ignition[846]:       {
[   37.418468] ignition[846]:         "filesystem": "root",
[   37.418854] ignition[846]:         "path": "/root/.docker/config.json",
[   37.419327] ignition[846]:         "contents": {
[   37.419813] ignition[846]:           "source": "data:,%7B%0A%09%22auths%22%3A%20%7B%0A%09%09%22gitlab-registry.nordstrom.com%22%3A%20%7B%0A%09%09%09%22auth%22%3A%20%22azhzLXJlYWQtb25seTotLXhrZVgySjZEeTRSOTZ3ZXhuVA%3D%3D%22%0A%09%09%7D%0A%09%7D%0A%7D%0A",
[   37.420286] ignition[846]:           "verification": {}
[   37.420746] ignition[846]:         },
         Starting Root filesystem setup...
[   37.421920] ignition[846]:         "mode": 493
[   37.422385] ignition[846]:       },
[   37.422620] ignition[846]:       {
[   37.422849] ignition[846]:         "filesystem": "root",
[   37.423091] ignition[846]:         "path": "/etc/rkt/auth.d/gitlab-registry.json",
[   37.423554] ignition[846]:         "contents": {
[   37.423918] systemd[1]: ignition-disks.service: Failed with result 'exit-code'.
[   37.424334] ignition[846]:           "source": "data:,%7B%0A%09%22rktKind%22%3A%20%22dockerAuth%22%2C%0A%09%22rktVersion%22%3A%20%22v1%22%2C%0A%09%22registries%22%3A%20%5B%22gitlab-registry.nordstrom.com%22%5D%2C%0A%09%22credentials%22%3A%20%7B%0A%09%09%22user%22%3A%20%22k8s-read-only%22%2C%0A%09%09%22password%22%3A%20%22--xkeX2J6Dy4R96wexnT%22%0A%09%7D%0A%7D%0A",
[   37.424779] ignition[846]:           "verification": {}
[   37.425013] ignition[846]:         },
[   37.425490] ignition[846]:         "mode": 493
[   37.425891] ignition[846]:       },
[   37.426415] ignition[846]:       {
[   37.426643] ignition[846]:         "filesystem": "root",
[   37.427083] ignition[846]:         "path": "/etc/coreos/update.conf",
[   37.427477] ignition[846]:         "contents": {
[   37.427743] ignition[846]:           "source": "data:,%0AREBOOT_STRATEGY%3D%22off%22",
[   37.428121] ignition[846]:           "verification": {}
[   37.428546] ignition[846]:         },
[   37.428844] ignition[846]:         "mode": 420
[   37.429090] ignition[846]:       }
[   37.429498] ignition[846]:     ],
[   37.431038] ignition[846]:     "filesystems": [
[   37.431569] ignition[846]:       {
[   37.432558] ignition[846]:         "name": "root",
[   37.432996] ignition[846]:         "path": "/sysroot"
[   37.433426] ignition[846]:       },
[   37.433855] ignition[846]:       {
[   37.434285] ignition[846]:         "mount": {
[   37.434710] ignition[846]:           "device": "/dev/nvme1n1",
[   37.435159] ignition[846]:           "format": "ext4",
[   37.435582] ignition[846]:           "wipeFilesystem": true
[   37.436018] ignition[846]:         },
[   37.436446] ignition[846]:         "name": "etcddatafs"
[   37.436884] ignition[846]:       },
[   37.437113] ignition[846]:       {
[   37.437552] ignition[846]:         "mount": {
[   37.437969] ignition[846]:           "device": "/dev/xvdp",
[   37.438400] ignition[846]:           "format": "ext4",
[   37.438837] ignition[846]:           "wipeFilesystem": true
[   37.439276] ignition[846]:         },
[   37.439716] ignition[846]:         "name": "imagefs"
[   37.440138] ignition[846]:       },
[   37.440546] ignition[846]:       {
[   37.440976] ignition[846]:         "mount": {
[   37.441406] ignition[846]:           "device": "/dev/xvdc",
[   37.441848] ignition[846]:           "format": "ext4",
[   37.442302] ignition[846]:           "wipeFilesystem": true
[   37.442732] ignition[846]:         },
[   37.443177] ignition[846]:         "name": "nodefs"
[   37.443421] ignition[846]:       }
[   37.443839] ignition[846]:     ]
[   37.444074] ignition[846]:   },
[   37.444498] ignition[846]:   "systemd": {
[   37.444911] ignition[846]:     "units": [
[   37.445408] ignition[846]:       {
[   37.445863] ignition[846]:         "enabled": true,
[   37.446314] ignition[846]:         "name": "coreos-metadata-sshkeys@.service"
[   37.446757] ignition[846]:       },
[   37.447196] ignition[846]:       {
[   37.447467] ignition[846]:         "dropins": [
[   37.447933] ignition[846]:           {
[   37.448356] ignition[846]:             "contents": "[Service]\nEnvironment=\"DOCKER_OPTS=--log-opt max-size=20m --log-opt max-file=10\"",
[   37.448786] ignition[846]:             "name": "20-clct-docker.conf"
[   37.449230] ignition[846]:           }
[   37.449670] ignition[846]:         ],
[   37.450090] ignition[846]:         "enable": true,
[   37.450532] ignition[846]:         "name": "docker.service"
[   37.450962] ignition[846]:       },
[   37.451395] ignition[846]:       {
[   37.451884] ignition[846]:         "dropins": [
[   37.452132] ignition[846]:           {
[   37.452676] systemd[1]: Failed to start Ignition (disks).
[   37.453399] ignition[846]:             "contents": "[Unit]\nRequires=coreos-metadata.service\nAfter=coreos-metadata.service\n\n[Service]\nEnvironmentFile=/run/metadata/coreos\nEnvironment=\"ETCD_IMAGE_TAG=v3.3.15\"\nExecStart=\nExecStart=/usr/lib/coreos/etcd-wrapper $ETCD_OPTS \\\n  --name=\"${COREOS_EC2_HOSTNAME}\" \\\n  --data-dir=\"/var/lib/etcd/data\" \\\n  --listen-peer-urls=\"https://${COREOS_EC2_IPV4_LOCAL}:2380\" \\\n  --listen-client-urls=\"https://0.0.0.0:2379\" \\\n  --initial-advertise-peer-urls=\"https://${COREOS_EC2_HOSTNAME}:2380\" \\\n  --advertise-client-urls=\"https://${COREOS_EC2_HOSTNAME}:2379\" \\\n  --cert-file=\"/etc/ssl/certs/tls.crt\" \\\n  --key-file=\"/etc/ssl/certs/tls.key\" \\\n  --client-cert-auth=true \\\n  --trusted-ca-file=\"/etc/ssl/certs/ca.crt\" \\\n  --peer-cert-file=\"/etc/ssl/certs/tls.crt\" \\\n  --peer-key-file=\"/etc/ssl/certs/tls.key\" \\\n  --peer-client-cert-auth=true \\\n  --peer-trusted-ca-file=\"/etc/ssl/certs/ca.crt\"",
[[   37.500954]   OK  ] Started Root filesystem setup.
ignition[846]:             "name": "20-clct-etcd-member.conf"
[   37.502631] ignition[846]:           }
[   37.503042] ignition[846]:         ],
         Starting Reload Configuration from the Real Root...
[   37.503483] ignition[846]:         "enable": true,
[   37.503916] ignition[846]:         "name": "etcd-member.service"
[   37.504366] ignition[846]:       },
[   37.504807] ignition[846]:       {
[   37.505264] ignition[846]:         "mask": true,
[   37.505706] ignition[846]:         "name": "update-engine.service"
[   37.506162] ignition[846]:       },
[   37.506598] ignition[846]:       {
[   37.507033] ignition[846]:         "mask": true,
[   37.507482] ignition[846]:         "name": "locksmithd.service"
[   37.507923] ignition[846]:       },
[   37.508378] ignition[846]:       {
[   37.508821] ignition[846]:         "enable": true,
[   37.509267] ignition[846]:         "name": "fstrim.timer"
[   37.509694] ignition[846]:       },
[   37.510124] ignition[846]:       {
[   37.510568] ignition[846]:         "contents": "[Unit]\nDescription=Mount instance storage to /var/lib/etcd\n[Mount]\nWhat=/dev/nvme1n1\nWhere=/var/lib/etcd\nType=ext4\n",
[   37.511007] ignition[846]:         "name": "var-lib-etcd.mount"
[   37.511437] ignition[846]:       },
[   37.511864] ignition[846]:       {
[   37.512325] ignition[846]:         "contents": "[Unit]\nDescription=Mount ephemeral to /var/lib/docker\n[Mount]\nWhat=/dev/xvdp\nWhere=/var/lib/docker\nType=ext4\n",
[   37.512774] ignition[846]:         "name": "var-lib-docker.mount"
[   37.513235] ignition[846]:       },
[   37.513647] ignition[846]:       {
[   37.514072] ignition[846]:         "contents": "[Unit]\nDescription=Mount ephemeral to /var/lib/kubelet\n[Mount]\nWhat=/dev/xvdc\nWhere=/var/lib/kubelet\nType=ext4\n",
[   37.514492] ignition[846]:         "name": "var-lib-kubelet.mount"
[   37.514929] ignition[846]:       },
[   37.515369] ignition[846]:       {
[   37.515970] systemd[1]: Dependency failed for Initrd Default Target.
[   37.516590] ignition[846]:         "contents": "[Unit]\nDescription=Update kernel tuning params\n[Service]\nExecStartPre=/sbin/sysctl -w net.ipv4.neigh.default.gc_thresh3=15000 ;\\\n  /sbin/sys[   37.821764] serial8250: too much work for irq4
ctl -w net.ipv4.neigh.default.gc_thresh2=12500 ;\\\n  /sbin/sysctl -w net.ipv4.neigh.default.gc_thresh1=10000 ;\\\n  /sbin/sysctl -w fs.inotify.max_user_watches=131072\nExecStart=/bin/echo \"Set kernel tuning params\"\nRemainAfterExit=true\nRestart=on-failure\nRestartSec=10\n[Install]\nWantedBy=multi-user.target\n",
[   37.516929] ignition[846]:         "enable": true,
[   37.517593] ignition[846]:         "name": "params.service"
[   37.518023] ignition[846]:       },
[   37.518462] ignition[846]:       {
[   37.518899] ignition[846]:         "dropins": [
[   37.519338] ignition[846]:           {
[   37.519761] ignition[846]:             "contents": "[Unit]\nAfter=var-lib-docker.mount\nRequires=var-lib-docker.mount\n",
[   37.520193] ignition[846]:             "name": "20-await-var-lib-docker-mount.conf"
[   37.520622] ignition[846]:           }
[   37.521045] ignition[846]:         ],
[   37.521476] ignition[846]:         "name": "docker.service"
[   37.521906] ignition[846]:       },
[   37.522347] ignition[846]:       {
[   37.522813] systemd[1]: initrd.target: Job initrd.target/start failed with result 'dependency'.
[   37.523463] ignition[846]:         "contents": "[Unit]\nDescription=Generate etcd certs\nRequires=coreos-metadata.service\nAfter=coreos-metadata.service\nConditionFileNotEmpty=/opt/bin/generate_certs.sh\nConditionFileNotEmpty=!/etc/ssl/etcd/tls.crt\nConditionFileNotEmpty=!/etc/ssl/etcd/tls.key\nConditionFileNotEmpty=!/etc/ssl/etcd/ca.crt\n[Service]\nEnvironmentFile=/run/metadata/coreos\nEnvironment=ASSETS_DIR=/etc/ssl/etcd\nExecStartPre=/usr/bin/mkdir -p $ASSETS_DIR\nExecStartPre=/bin/sh -c \"\\\n  CERT_COMMON_NAME=etcd.aws-9405.platform-dev.r53.nordstrom.net \\\n  CERT_IP_SANS=${COREOS_EC2_IPV4_LOCAL},127.0.0.1 \\\n  CERT_ALT_NAMES=etcd.aws-9405.platform-dev.r53.nordstrom.net,${COREOS_EC2_HOSTNAME},localhost \\\n  ROOT_CA_KEY=\"/root-ca/etcd-ca.key\" \\\n  ROOT_CA_CERT=\"/root-ca/etcd-ca.crt\" \\\n  /opt/bin/generate_certs.sh\"\nExecStartPre=/bin/rm /root-ca/etcd-ca.key /root-ca/etcd-ca.crt\nExecStartPre=/usr/bin/curl -v https://discovery.etcd.io\nExecStartPre=/usr/bin/sh -c \[2019-09-12T23:21:02.140590]"echo | \\\n  openssl s_client -connect discovery.etcd.io:443 -prexit -showcerts 2\u003e /dev/null \\\n  | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' \\\n  \u003e /etc/ssl/etcd/ca-certificates.crt\"\nExecStart=/bin/echo \"Etcd certs generated\"\nRemainAfterExit=true\nRestart=on-failure\nRestartSec=10\n[Install]\nWantedBy=multi-user.target\n",
[   37.523892] ignition[846]:         "enable": true,
[   37.524314] ignition[846]:         "name": "generate-etcd-certs.service"
[   37.524742] ignition[846]:       },
[   37.525956] ignition[846]:       {
[   37.526530] ignition[846]:         "dropins": [
[   37.526966] ignition[846]:           {
[   37.527427] systemd[1]: initrd.target: Triggering OnFailure= dependencies.
[   37.528087] systemd-fsck[1074]: ROOT: clean, 565/553792 files, 37484/553472 blocks
[   37.528743] ignition[846]:             "contents": "[Unit]\nRequires=var-lib-etcd.mount\nAfter=var-lib-etcd.mount\nRequires=generate-etcd-certs.service\nAfter=generate-etcd-certs.service\n[Service]\nEnvironment=\"ETCD_IMAGE_URL=quay.io/coreos/etcd\"\nEnvironment=\"RKT_GLOBAL_ARGS=\"\nEnvironment=\"ETCD_STRICT_RECONFIG_CHECK=true\"\nEnvironment=\"ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:9379\"\nEnvironment=\"ETCD_DISCOVERY=https://discovery.etcd.io/9cc5ec5fe416532f25a4cde484d304f8\"\nEnvironment=\"ETCD_SSL_DIR=/etc/ssl/etcd\"\nOOMScoreAdjust=-999\nExecStartPre=/usr/bin/chown -R etcd:etcd /var/lib/etcd\nExecStartPre=/usr/bin/chown -R etcd:etcd /etc/ssl/etcd\n",
[   37.579942] ignition[846]:             "name": "01-conf.conf"
[   37.580596] ignition[846]:           }
[   37.581008] ignition[846]:         ],
[   37.581385] ignition[846]:         "enable": true,
[   37.581765] ignition[846]:         "name": "etcd-member.service"
[   37.582123] ignition[846]:       },
[   37.582495] ignition[846]:       {
[   37.582872] ignition[846]:         "contents": "[Unit]\nDescription=Generate kubelet certs\nRequires=coreos-metadata.service\nAfter=coreos-metadata.service\nConditionFileNotEmpty=/opt/bin/generate_certs.sh\nConditionFileNotEmpty=!/etc/kubernetes/tls.crt\nConditionFileNotEmpty=!/etc/kubernetes/tls.key\nConditionFileNotEmpty=!/etc/kubernetes/ca.crt\n[Service]\nEnvironmentFile=/run/metadata/coreos\nEnvironment=ASSETS_DIR=/etc/kubernetes\nExecStartPre=/usr/bin/mkdir -p $ASSETS_DIR\nExecStartPre=/bin/sh -c \"\\\n  CERT_COMMON_NAME=system:node:${COREOS_EC2_HOSTNAME} \\\n  CERT_ORGANIZATION=system:nodes \\\n  CERT_ALT_NAMES=${COREOS_EC2_HOSTNAME} \\\n  ROOT_CA_KEY=/root-ca/k8s-ca.key \\\n  ROOT_CA_CERT=/root-ca/k8s-ca.crt \\\n  /opt/bin/generate_certs.sh\"\nExecStartPre=/bin/rm /root-ca/k8s-ca.key /root-ca/k8s-ca.crt\nExecStart=/bin/echo \"Kubelet certs generated\"\nRemainAfterExit=true\nRestart=on-failure\nRestartSec=10\n[Install]\nWantedBy=multi-user.target\n",
[   37.583442] ignition[846]:         "enable": true,
[   37.583805] ignition[846]:         "name": "generate-kubelet-certs.service"
[   37.584363] ignition[846]:       },
[   37.584742] ignition[846]:       {
[   37.585246] systemd[1]: Dependency failed for Ignition (files).
[   37.585888] systemd[1]: Dependency failed for Populate torcx store to satisfy profile.
[   37.586745] ignition[846]:         "contents": "[Unit]\nDescription=Kubernetes Kubelet\nDocumentation=https://github.com/kubernetes/kubernetes\nRequires=generate-kubelet-certs.service\nAfter=generate-kubelet-certs.service\nRequires=var-lib-kubelet.mount\nAfter=var-lib-kubelet.mount\n[Service]\nEnvironmentFile=/run/metadata/coreos\nEnvironment=KUBELET_IMAGE=docker://gcr.io/google_containers/hyperkube:v1.15.3\n# additive to kubelet-wrapper\nEnvironment=\"RKT_GLOBAL_ARGS=--insecure-options=image\"\nEnvironment=\"RKT_RUN_ARGS= \\\n  --volume dns,kind=host,source=/etc/resolv.conf \\\n  --volume var-lib-cni,kind=host,source=/var/lib/cni \\\n  --volume var-log,kind=host,source=/var/log \\\n  --volume cni-bin,kind=host,source=/run/torcx/unpack/crio/usr/share/cni/bin/ \\\n  --mount volume=cni-bin,target=/opt/cni/bin \\\n  --mount volume=dns,target=/etc/resolv.conf \\\n  --mount volume=var-lib-cni,target=/var/lib/cni \\\n  --mount volume=var-log,target=/var/log\"\nExecStartPre=/bin/mkdir[2019-09-12T23:21:02.239909] -p /etc/kubernetes/manifests\nExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d\nExecStartPre=/bin/mkdir -p /etc/kubernetes/checkpoint-secrets\nExecStartPre=/bin/mkdir -p /srv/kubernetes/manifests\nExecStartPre=/bin/mkdir -p /var/lib/cni\nExecStartPre=/usr/bin/mkdir -p /var/log/containers\nExecStartPre=/bin/cp /root/.docker/config.json /var/lib/kubelet/config.json\nExecStart=/usr/lib/coreos/kubelet-wrapper \\\n  --anonymous-auth=false \\\n  --read-only-port=0 \\\n  --cgroup-driver=cgroupfs \\\n  --cloud-provider=aws \\\n  --client-ca-file=/etc/kubernetes/ca.crt \\\n  --tls-cert-file=/etc/kubernetes/tls.crt \\\n  --tls-private-key-file=/etc/kubernetes/tls.key \\\n  --cluster_dns=${COREOS_EC2_IPV4_LOCAL},25.0.0.10 \\\n  --cluster_domain=cluster.local \\\n  --exit-on-lock-contention \\\n  --feature-gates=LocalStorageCapacityIsolation=true \\\n  --lock-file=/var/run/lock/kubelet.lock \\\n  --kubeconfig=/etc/kubernetes/kubelet/kubeconfig \\\n  --network-plugin=cni \\\n  --cni-conf-[2019-09-12T23:21:02.239929]dir=/etc/kubernetes/cni/net.d \\\n  --pod-manifest-path=/etc/kubernetes/manifests \\\n  --node-labels=master=true,\"beta.nordstrom.net/node-pool\"=\"etcd-pool\",\"node-role.kubernetes.io/master\"=true \\\n  --register-with-taints=dedicate=\"etcd:NoSchedule\" \\\n  --system-reserved=\"cpu=350m,memory=3Gi\" \\\n  --eviction-hard=\"memory.available\u003c10%,nodefs.available\u003c10%,nodefs.inodesFree\u003c10%,imagefs.available\u003c10%,imagefs.inodesFree\u003c10%\" \\\n  --eviction-soft=\"memory.available\u003c15%,nodefs.available\u003c20%,nodefs.inodesFree\u003c20%,imagefs.available\u003c20%,imagefs.inodesFree\u003c20%\" \\\n  --eviction-soft-grace-period=\"memory.available=2m,nodefs.available=30s,nodefs.inodesFree=30s,imagefs.available=30s,imagefs.inodesFree=30s\" \\\n  --eviction-minimum-reclaim=\"memory.available=0Mi,nodefs.available=5Gi,imagefs.available=5Gi\" \\\n  --e[   37.925411] serial8250: too much work for irq4
viction-max-pod-grace-period=\"300\" \\\n  --authentication-token-webhook \\\n  --cpu-manager-policy=static\nRestart=always\nRestartSec=5\n[Install]\nWantedBy=multi-user.target\n",
[   37.633021] ignition[846]:         "enable": true,
[   37.633418] ignition[846]:         "name": "kubelet.service"
[   37.633851] ignition[846]:       }
[   37.634505] ignition[846]:     ]
[   37.634941] ignition[846]:   }
[   37.635419] systemd[1]: torcx-profile-populate.service: Job torcx-profile-populate.service/start failed with result 'dependency'.
[   37.636127] ignition[846]: }
[   37.636557] systemd[1]: ignition-files.service: Job ignition-files.service/start failed with result 'dependency'.
[   37.637188] systemd[1]: Reached target Local File Systems (Pre).
[   37.637434] systemd[1]: Starting Verity Setup for /dev/mapper/usr...
[   37.637876] systemd[1]: Starting Setup Virtual Console...
[   37.638315] systemd[1]: Stopping Network Service...
[   37.638817] systemd[1]: dracut-pre-mount.service: Succeeded.
[   37.639213] systemd[1]: Stopped dracut pre-mount hook.
[   37.639635] systemd[1]: Starting File System Check on /dev/disk/by-label/ROOT...
[   37.639914] systemd[1]: dracut-initqueue.service: Succeeded.
[   37.640459] systemd[1]: Stopped dracut initqueue hook.
[   37.640756] systemd[1]: dracut-pre-trigger.service: Succeeded.
[   37.652578] systemd[1]: Stopped dracut pre-trigger hook.
[[   37.653284] systemd  OK  [1]: ] systemd-ask-password-console.path: Succeeded.Started Reload Configuration from the Real Root.

[   37.653945] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch.
[  OK  ] Reached target Initrd File Systems.
[   37.654252] systemd[1]: dracut-pre-udev.service: Succeeded.
[   37.654692] systemd[1]: Stopped dracut pre-udev hook.
[   37.654923] systemd[1]: dracut-cmdline.service: Succeeded.
         Starting Cleaning Up and Shutting Down Daemons...
[   37.655390] systemd[1]: Stopped dracut cmdline hook.
[   37.655838] systemd[1]: systemd-vconsole-setup.service: Succeeded.
[   37.656164] systemd[1]: Started Setup Virtual Console.
[   37.656260] systemd[1]: Started File System Check on /dev/disk/by-label/ROOT.
[   37.656487] systemd[1]: Started Verity Setup for /dev/mapper/usr.
[   37.656722] systemd[1]: Found device /dev/mapper/usr.
[   37.656955] systemd[1]: Condition check resulted in File System Check on /dev/mapper/usr being skipped.
[   37.657413] systemd[1]: Mounting /sysroot...
[   37.657648] systemd[1]: Starting Emergency Shell...
[   37.657879] systemd[1]: Mounted /sysroot.
[   37.658114] systemd[1]: Mounting /sysroot/usr...
[   37.658571] systemd[1]: Reached target Initrd Root File System.
[   37.658809] systemd[1]: Mounted /sysroot/usr.
[   37.659043] systemd[1]: Reached target Local File Systems.
[   37.659499] systemd[1]: Starting Root filesystem setup...
[   37.659730] systemd[1]: initrd-setup-root.service: Succeeded.
[   37.659958] systemd[1]: Started Root filesystem setup.
[   37.660430] systemd[1]: Starting Reload Configuration from the Real Root...
[   37.660651] systemd[1]: Reloading.
[   37.660876] systemd[1]: initrd-parse-etc.service: Succeeded.
[   37.661105] systemd[1]: Started Reload Configuration from the Real Root.
[   37.661583] systemd[1]: Reached target Initrd File Systems.
[   37.662001] systemd[1]: Starting Cleaning Up and Shutting Down Daemons...
[   37.662756] systemctl[1234]: Failed to start initrd-switch-root.target: Transaction for initrd-switch-root.target/start is destructive (emergency.service has 'start' job queued, but 'stop' is included in transaction).
[   37.663021] systemctl[1234]: See system logs and 'systemctl status initrd-switch-root.target' for details.
[   37.663419] systemd[1]: Requested transaction contradicts existing jobs: Transaction for initrd-switch-root.target/start is destructive (emergency.service has 'start' job queued, but 'stop' is included in transaction).
[FAILED] Failed to start Cleaning Up and Shutting Down Daemons.
See 'systemctl status initrd-cleanup.service' for details.
         Starting Setup Virtual Console...
[   37.666892] systemd[1]: initrd-cleanup.service: Main process exited, code=exited, status=4/NOPERMISSION
[   37.671334] systemd[1]: initrd-cleanup.service: Failed with result 'exit-code'.
[   37.823943] systemd[1]: Failed to start Cleaning Up and Shutting Down Daemons.
[  OK  ] Started Setup Virtual Console.
[   37.827097] systemd[1]: initrd-cleanup.service: Triggering OnFailure= dependencies.
[   37.828783] systemd[1]: Starting Setup Virtual Console...
[   37.829608] systemd[1]: systemd-vconsole-setup.service: Succeeded.
[   37.830615] systemd[1]: Started Setup Virtual Console.
[   37.917350] systemd-networkd[266]: lo: Lost carrier
[   37.925219] systemd-networkd[266]: eth0: Lost carrier
[   37.925365] systemd-networkd[266]: eth0: DHCP lease lost
[   37.925889] systemd-networkd[266]: eth0: IPv6 successfully enabled
[   37.937444] systemd[1]: systemd-networkd.service: Succeeded.
[  OK  ] Stopped Network Service.
[   37.937776] systemd[1]: Stopped Network Service.
cat: unrecognized option '-------------------------------------------------------------------------------'
Try 'cat --help' for more information.
Ignition v0.33.0
reading system config file "/usr/lib/ignition/base.ign"
parsing config with SHA512: 865c03baa79b8c74023d13a0b3666474fa06a165421a1e05731b76e0f557d42c5c89d4870a0b9c4182ad7d4d8209de20dca9c9da63d637e0410fbd60314cac6c
parsed url from cmdline: ""
no config URL provided
reading system config file "/usr/lib/ignition/user.ign"
no config at "/usr/lib/ignition/user.ign"
GET http://169.254.169.254/2009-04-04/user-data: attempt #1
GET result: OK
parsing config with SHA512: c192cb154c8d98f80afade2650befb222840b50419ba7876edfb4f2bcbb3a60f9f3d2430f868a163906d3d17da870dbde35e78aa5217a07454a05dad10914771
disks: createPartitions: op(1): [started]  waiting for devices [/dev/nvme1n1 /dev/xvdp /dev/xvdc]
disks: createPartitions: op(1): [finished] waiting for devices [/dev/nvme1n1 /dev/xvdp /dev/xvdc]
disks: createPartitions: created device alias for "/dev/nvme1n1": "/run/ignition/dev_aliases/dev/nvme1n1" -> "/dev/nvme1n1"
disks: createPartitions: created device alias for "/dev/xvdp": "/run/ignition/dev_aliases/dev/xvdp" -> "/dev/nvme2n1"
disks: createPartitions: created device alias for "/dev/xvdc": "/run/ignition/dev_aliases/dev/xvdc" -> "/dev/nvme1n1"
disks: createPartitions: op(2): [started]  partitioning "/run/ignition/dev_aliases/dev/nvme1n1"
disks: createPartitions: op(2): wiping partition table requested on "/run/ignition/dev_aliases/dev/nvme1n1"
disks: createPartitions: op(2): running sgdisk with options: [--zap-all /run/ignition/dev_aliases/dev/nvme1n1]
disks: createPartitions: op(2): op(3): [started]  deleting 0 partitions and creating 0 partitions on "/run/ignition/dev_aliases/dev/nvme1n1"
disks: createPartitions: op(2): op(3): executing: "/usr/sbin/sgdisk" "--zap-all" "/run/ignition/dev_aliases/dev/nvme1n1"
disks: createPartitions: op(2): op(3): [finished] deleting 0 partitions and creating 0 partitions on "/run/ignition/dev_aliases/dev/nvme1n1"
disks: createPartitions: op(2): op(4): [started]  reading partition table of "/run/ignition/dev_aliases/dev/nvme1n1"
disks: createPartitions: op(2): op(4): [finished] reading partition table of "/run/ignition/dev_aliases/dev/nvme1n1"
disks: createPartitions: op(2): running sgdisk with options: [--pretend --new=0:0:+0 /run/ignition/dev_aliases/dev/nvme1n1]
disks: createPartitions: op(2): running sgdisk with options: [--new=0:0:+0 --change-name=0:ETCDDATAFS /run/ignition/dev_aliases/dev/nvme1n1]
disks: createPartitions: op(2): op(5): [started]  deleting 0 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/nvme1n1"
disks: createPartitions: op(2): op(5): executing: "/usr/sbin/sgdisk" "--new=0:0:+0" "--change-name=0:ETCDDATAFS" "/run/ignition/dev_aliases/dev/nvme1n1"
disks: createPartitions: op(2): op(5): [finished] deleting 0 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/nvme1n1"
disks: createPartitions: op(2): [finished] partitioning "/run/ignition/dev_aliases/dev/nvme1n1"
disks: createPartitions: op(6): [started]  partitioning "/run/ignition/dev_aliases/dev/xvdp"
disks: createPartitions: op(6): wiping partition table requested on "/run/ignition/dev_aliases/dev/xvdp"
disks: createPartitions: op(6): running sgdisk with options: [--zap-all /run/ignition/dev_aliases/dev/xvdp]
disks: createPartitions: op(6): op(7): [started]  deleting 0 partitions and creating 0 partitions on "/run/ignition/dev_aliases/dev/xvdp"
disks: createPartitions: op(6): op(7): executing: "/usr/sbin/sgdisk" "--zap-all" "/run/ignition/dev_aliases/dev/xvdp"
disks: createPartitions: op(6): op(7): [finished] deleting 0 partitions and creating 0 partitions on "/run/ignition/dev_aliases/dev/xvdp"
disks: createPartitions: op(6): op(8): [started]  reading partition table of "/run/ignition/dev_aliases/dev/xvdp"
disks: createPartitions: op(6): op(8): [finished] reading partition table of "/run/ignition/dev_aliases/dev/xvdp"
disks: createPartitions: op(6): running sgdisk with options: [--pretend --new=0:0:+0 /run/ignition/dev_aliases/dev/xvdp]
disks: createPartitions: op(6): running sgdisk with options: [--new=0:0:+0 --change-name=0:IMAGEFS /run/ignition/dev_aliases/dev/xvdp]
disks: createPartitions: op(6): op(9): [started]  deleting 0 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/xvdp"
disks: createPartitions: op(6): op(9): executing: "/usr/sbin/sgdisk" "--new=0:0:+0" "--change-name=0:IMAGEFS" "/run/ignition/dev_aliases/dev/xvdp"
disks: createPartitions: op(6): op(9): [finished] deleting 0 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/xvdp"
disks: createPartitions: op(6): [finished] partitioning "/run/ignition/dev_aliases/dev/xvdp"
disks: createPartitions: op(a): [started]  partitioning "/run/ignition/dev_aliases/dev/xvdc"
disks: createPartitions: op(a): wiping partition table requested on "/run/ignition/dev_aliases/dev/xvdc"
disks: createPartitions: op(a): running sgdisk with options: [--zap-all /run/ignition/dev_aliases/dev/xvdc]
disks: createPartitions: op(a): op(b): [started]  deleting 0 partitions and creating 0 partitions on "/run/ignition/dev_aliases/dev/xvdc"
disks: createPartitions: op(a): op(b): executing: "/usr/sbin/sgdisk" "--zap-all" "/run/ignition/dev_aliases/dev/xvdc"
disks: createPartitions: op(a): op(b): [finished] deleting 0 partitions and creating 0 partitions on "/run/ignition/dev_aliases/dev/xvdc"
disks: createPartitions: op(a): op(c): [started]  reading partition table of "/run/ignition/dev_aliases/dev/xvdc"
disks: createPartitions: op(a): op(c): [finished] reading partition table of "/run/ignition/dev_aliases/dev/xvdc"
disks: createPartitions: op(a): running sgdisk with options: [--pretend --new=0:0:+0 /run/ignition/dev_aliases/dev/xvdc]
disks: createPartitions: op(a): running sgdisk with options: [--new=0:0:+0 --change-name=0:NODEFS /run/ignition/dev_aliases/dev/xvdc]
disks: createPartitions: op(a): op(d): [started]  deleting 0 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/xvdc"
disks: createPartitions: op(a): op(d): executing: "/usr/sbin/sgdisk" "--new=0:0:+0" "--change-name=0:NODEFS" "/run/ignition/dev_aliases/dev/xvdc"
disks: createPartitions: op(a): op(d): [finished] deleting 0 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/xvdc"
disks: createPartitions: op(a): [finished] partitioning "/run/ignition/dev_aliases/dev/xvdc"
disks: createFilesystems: op(e): [started]  waiting for devices [/dev/nvme1n1 /dev/xvdp /dev/xvdc]
disks: createFilesystems: op(e): [finished] waiting for devices [/dev/nvme1n1 /dev/xvdp /dev/xvdc]
disks: createFilesystems: created device alias for "/dev/nvme1n1": "/run/ignition/dev_aliases/dev/nvme1n1" -> "/dev/nvme1n1"
disks: createFilesystems: created device alias for "/dev/xvdp": "/run/ignition/dev_aliases/dev/xvdp" -> "/dev/nvme2n1"
disks: createFilesystems: created device alias for "/dev/xvdc": "/run/ignition/dev_aliases/dev/xvdc" -> "/dev/nvme1n1"
disks: createFilesystems: op(f): [started]  determining filesystem type of "/dev/nvme1n1"
disks: createFilesystems: op(f): op(10): [started]  determining filesystem type of "/dev/xvdp"
disks: createFilesystems: op(f): op(10): op(11): [started]  determining filesystem type of "/dev/xvdc"
disks: createFilesystems: op(f): op(10): op(11): found  filesystem at "/dev/nvme1n1" with uuid "" and label ""
disks: createFilesystems: op(f): op(10): op(11): [finished] determining filesystem type of "/dev/nvme1n1"
disks: createFilesystems: op(f): op(10): op(12): [started]  creating "ext4" filesystem on "/run/ignition/dev_aliases/dev/nvme1n1"
disks: createFilesystems: op(f): op(10): op(12): executing: "/usr/sbin/mkfs.ext4" "-F" "/run/ignition/dev_aliases/dev/nvme1n1"
disks: createFilesystems: op(f): op(10): op(12): found  filesystem at "/dev/xvdc" with uuid "" and label ""
disks: createFilesystems: op(f): op(10): op(12): [finished] determining filesystem type of "/dev/xvdc"
disks: createFilesystems: op(f): op(10): op(13): [started]  creating "ext4" filesystem on "/run/ignition/dev_aliases/dev/xvdc"
disks: createFilesystems: op(f): op(10): op(13): executing: "/usr/sbin/mkfs.ext4" "-F" "/run/ignition/dev_aliases/dev/xvdc"
disks: createFilesystems: op(f): op(10): op(13): [failed]   creating "ext4" filesystem on "/run/ignition/dev_aliases/dev/xvdc": exit status 1: Cmd: "/usr/sbin/mkfs.ext4" "-F" "/run/ignition/dev_aliases/dev/xvdc" Stdout: "" Stderr: "mke2fs 1.44.5 (15-Dec-2018)\nFound a gpt partition table in /run/ignition/dev_aliases/dev/xvdc\n/run/ignition/dev_aliases/dev/xvdc is apparently in use by the system; will not make a filesystem here!\n"
disks: createFilesystems: op(f): op(10): found  filesystem at "/dev/xvdp" with uuid "" and label ""
disks: createFilesystems: op(f): op(10): [finished] determining filesystem type of "/dev/xvdp"
disks: createFilesystems: op(f): op(14): [started]  creating "ext4" filesystem on "/run/ignition/dev_aliases/dev/xvdp"
disks: createFilesystems: op(f): op(14): executing: "/usr/sbin/mkfs.ext4" "-F" "/run/ignition/dev_aliases/dev/xvdp"
disks: createFilesystems: op(f): op(14): [finished] creating "ext4" filesystem on "/run/ignition/dev_aliases/dev/xvdp"
disks: createFilesystems: op(f): [finished] creating "ext4" filesystem on "/run/ignition/dev_aliases/dev/nvme1n1"
Ignition failed: failed to create filesystems: mkfs failed: exit status 1: Cmd: "/usr/sbin/mkfs.ext4" "-F" "/run/ignition/dev_aliases/dev/xvdc" Stdout: "" Stderr: "mke2fs 1.44.5 (15-Dec-2018)\nFound a gpt partition table in /run/ignition/dev_aliases/dev/xvdc\n/run/ignition/dev_aliases/dev/xvdc is apparently in use by the system; will not make a filesystem here!\n"
disks failedFull config:
{
  "ignition": {
    "config": {},
    "proxy": {},
    "security": {
      "tls": {}
    },
    "timeouts": {},
    "version": "2.4.0-experimental"
  },
  "networkd": {},
  "passwd": {},
  "storage": {
    "disks": [
      {
        "device": "/dev/nvme1n1",
        "partitions": [
          {
            "label": "ETCDDATAFS",
            "size": 0,
            "start": 0
          }
        ],
        "wipeTable": true
      },
      {
        "device": "/dev/xvdp",
        "partitions": [
          {
            "label": "IMAGEFS",
            "size": 0,
            "start": 0
          }
        ],
        "wipeTable": true
      },
      {
        "device": "/dev/xvdc",
        "partitions": [
          {
            "label": "NODEFS",
            "size": 0,
            "start": 0
          }
        ],
        "wipeTable": true
      }
    ],
    "files": [
      {
        "filesystem": "root",
        "path": "/etc/kubernetes/kubelet/kubeconfig",
        "contents": {
          "source": "data:,apiVersion%3A%20v1%0Akind%3A%20Config%0Aclusters%3A%0A-%20name%3A%20local%0A%20%20cluster%3A%0A%20%20%20%20server%3A%20https%3A%2F%2Faws-9405.platform-dev.r53.nordstrom.net%0A%20%20%20%20certificate-authority%3A%20%2Fetc%2Fkubernetes%2Fca.crt%0Ausers%3A%0A-%20name%3A%20kubelet%0A%20%20user%3A%0A%20%20%20%20client-certificate%3A%20%2Fetc%2Fkubernetes%2Ftls.crt%0A%20%20%20%20client-key%3A%20%2Fetc%2Fkubernetes%2Ftls.key%0Acontexts%3A%0A-%20context%3A%0A%20%20%20%20cluster%3A%20local%0A%20%20%20%20user%3A%20kubelet%0A",
          "verification": {}
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "path": "/etc/torcx/profiles/combined.json",
        "contents": {
          "source": "data:,%7B%22kind%22%3A%22profile-manifest-v0%22%2C%22value%22%3A%7B%22images%22%3A%5B%7B%22name%22%3A%22docker%22%2C%22reference%22%3A%2217.03%22%7D%2C%7B%22name%22%3A%22crio%22%2C%22reference%22%3A%22v1.14.6a%22%7D%5D%7D%7D%0A",
          "verification": {}
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "path": "/etc/torcx/next-profile",
        "contents": {
          "source": "data:,combined%0A",
          "verification": {}
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "path": "/var/lib/torcx/store/docker:17.03.torcx.tgz",
        "contents": {
          "source": "https://tectonic-torcx.release.core-os.net/pkgs/amd64-usr/docker/e3819eef5185486cd192ae03108003695e98e9cea4bc4b8082a0a79f212fc7eda263996c78fdac3690b0f2ebbfbdf5632f654483cc8bcd30bfb165ed9d322fbf/docker:17.03.torcx.tgz",
          "verification": {
            "hash": "sha512-e3819eef5185486cd192ae03108003695e98e9cea4bc4b8082a0a79f212fc7eda263996c78fdac3690b0f2ebbfbdf5632f654483cc8bcd30bfb165ed9d322fbf"
          }
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "path": "/etc/docker/daemon.json",
        "contents": {
          "source": "data:,%7B%0A%20%20%22metrics-addr%22%20%3A%20%220.0.0.0%3A9323%22%2C%0A%20%20%22experimental%22%20%3A%20true%0A%7D%0A",
          "verification": {}
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "path": "/var/lib/torcx/store/crio:v1.14.6a.torcx.tgz",
        "contents": {
          "source": "s3://platform-bootstrap-dev/binaries/crio:v1.14.6a.torcx.tgz",
          "verification": {}
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "path": "/etc/crio/crio.conf",
        "contents": {
          "source": "s3://platform-bootstrap-dev/binaries/etccrio/v1.14.6a/crio.conf",
          "verification": {}
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "path": "/etc/crio/seccomp.json",
        "contents": {
          "source": "s3://platform-bootstrap-dev/binaries/etccrio/v1.14.6a/seccomp.json",
          "verification": {}
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "path": "/opt/bin/generate_certs.sh",
        "contents": {
          "source": "s3://platform-bootstrap-dev-us-west-2/platform-instances/aws-9405/artifacts/generate_certs.sh",
          "verification": {}
        },
        "mode": 493
      },
      {
        "filesystem": "root",
        "path": "/root-ca/etcd-ca.crt",
        "contents": {
          "source": "s3://platform-bootstrap-dev-us-west-2/platform-instances/aws-9405/etcd-ca.crt",
          "verification": {}
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "path": "/root-ca/etcd-ca.key",
        "contents": {
          "source": "s3://platform-bootstrap-dev-us-west-2/platform-instances/aws-9405/etcd-ca.key",
          "verification": {}
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "path": "/root-ca/k8s-ca.crt",
        "contents": {
          "source": "s3://platform-bootstrap-dev-us-west-2/platform-root-ca/k8s-ca.crt",
          "verification": {}
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "path": "/root-ca/k8s-ca.key",
        "contents": {
          "source": "s3://platform-bootstrap-dev-us-west-2/platform-root-ca/k8s-ca.key",
          "verification": {}
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "path": "/root/.docker/config.json",
        "contents": {
          "source": "data:,%7B%0A%09%22auths%22%3A%20%7B%0A%09%09%22gitlab-registry.nordstrom.com%22%3A%20%7B%0A%09%09%09%22auth%22%3A%20%22azhzLXJlYWQtb25seTotLXhrZVgySjZEeTRSOTZ3ZXhuVA%3D%3D%22%0A%09%09%7D%0A%09%7D%0A%7D%0A",
          "verification": {}
        },
        "mode": 493
      },
      {
        "filesystem": "root",
        "path": "/etc/rkt/auth.d/gitlab-registry.json",
        "contents": {
          "source": "data:,%7B%0A%09%22rktKind%22%3A%20%22dockerAuth%22%2C%0A%09%22rktVersion%22%3A%20%22v1%22%2C%0A%09%22registries%22%3A%20%5B%22gitlab-registry.nordstrom.com%22%5D%2C%0A%09%22credentials%22%3A%20%7B%0A%09%09%22user%22%3A%20%22k8s-read-only%22%2C%0A%09%09%22password%22%3A%20%22--xkeX2J6Dy4R96wexnT%22%0A%09%7D%0A%7D%0A",
          "verification": {}
        },
        "mode": 493
      },
      {
        "filesystem": "root",
        "path": "/etc/coreos/update.conf",
        "contents": {
          "source": "data:,%0AREBOOT_STRATEGY%3D%22off%22",
          "verification": {}
        },
        "mode": 420
      }
    ],
    "filesystems": [
      {
        "name": "root",
        "path": "/sysroot"
      },
      {
        "mount": {
          "device": "/dev/nvme1n1",
          "format": "ext4",
          "wipeFilesystem": true
        },
        "name": "etcddatafs"
      },
      {
        "mount": {
          "device": "/dev/xvdp",
          "format": "ext4",
          "wipeFilesystem": true
        },
        "name": "imagefs"
      },
      {
        "mount": {
          "device": "/dev/xvdc",
          "format": "ext4",
          "wipeFilesystem": true
        },
        "name": "nodefs"
      }
    ]
  },
  "systemd": {
    "units": [
      {
        "enabled": true,
        "name": "coreos-metadata-sshkeys@.service"
      },
      {
        "dropins": [
          {
            "contents": "[Service]\nEnvironment=\"DOCKER_OPTS=--log-opt max-size=20m --log-opt max-file=10\"",
            "name": "20-clct-docker.conf"
          }
        ],
        "enable": true,
        "name": "docker.service"
      },
      {
        "dropins": [
          {
            "contents": "[Unit]\nRequires=coreos-metadata.service\nAfter=coreos-metadata.service\n\n[Service]\nEnvironmentFile=/run/metadata/coreos\nEnvironment=\"ETCD_IMAGE_TAG=v3.3.15\"\nExecStart=\nExecStart=/usr/lib/coreos/etcd-wrapper $ETCD_OPTS \\\n  --name=\"${COREOS_EC2_HOSTNAME}\" \\\n  --data-dir=\"/var/lib/etcd/data\" \\\n  --listen-peer-urls=\"https://${COREOS_EC2_IPV4_LOCAL}:2380\" \\\n  --listen-client-urls=\"https://0.0.0.0:2379\" \\\n  --initial-advertise-peer-urls=\"https://${COREOS_EC2_HOSTNAME}:2380\" \\\n  --advertise-client-urls=\"https://${COREOS_EC2_HOSTNAME}:2379\" \\\n  --cert-file=\"/etc/ssl/certs/tls.crt\" \\\n  --key-file=\"/etc/ssl/certs/tls.key\" \\\n  --client-cert-auth=true \\\n  --trusted-ca-file=\"/etc/ssl/certs/ca.crt\" \\\n  --peer-cert-file=\"/etc/ssl/certs/tls.crt\" \\\n  --peer-key-file=\"/etc/ssl/certs/tls.key\" \\\n  --peer-client-cert-auth=true \\\n  --peer-trusted-ca-file=\"/etc/ssl/certs/ca.crt\"",
            "name": "20-clct-etcd-member.conf"
          }
        ],
        "enable": true,
        "name": "etcd-member.service"
      },
      {
        "mask": true,
        "name": "update-engine.service"
      },
      {
        "mask": true,
        "name": "locksmithd.service"
      },
      {
        "enable": true,
        "name": "fstrim.timer"
      },
      {
        "contents": "[Unit]\nDescription=Mount instance storage to /var/lib/etcd\n[Mount]\nWhat=/dev/nvme1n1\nWhere=/var/lib/etcd\nType=ext4\n",
        "name": "var-lib-etcd.mount"
      },
      {
        "contents": "[Unit]\nDescription=Mount ephemeral to /var/lib/docker\n[Mount]\nWhat=/dev/xvdp\nWhere=/var/lib/docker\nType=ext4\n",
        "name": "var-lib-docker.mount"
      },
      {
        "contents": "[Unit]\nDescription=Mount ephemeral to /var/lib/kubelet\n[Mount]\nWhat=/dev/xvdc\nWhere=/var/lib/kubelet\nType=ext4\n",
        "name": "var-lib-kubelet.mount"
      },
      {
        "contents": "[Unit]\nDescription=Update kernel tuning params\n[Service]\nExecStartPre=/sbin/sysctl -w net.ipv4.neigh.default.gc_thresh3=15000 ;\\\n  /sbin/sysctl -w net.ipv4.neigh.default.gc_thresh2=12500 ;\\\n  /sbin/sysctl -w net.ipv4.neigh.default.gc_thresh1=10000 ;\\\n  /sbin/sysctl -w fs.inotify.max_user_watches=131072\nExecStart=/bin/echo \"Set kernel tuning params\"\nRemainAfterExit=true\nRestart=on-failure\nRestartSec=10\n[Install]\nWantedBy=multi-user.target\n",
        "enable": true,
        "name": "params.service"
      },
      {
        "dropins": [
          {
            "contents": "[Unit]\nAfter=var-lib-docker.mount\nRequires=var-lib-docker.mount\n",
            "name": "20-await-var-lib-docker-mount.conf"
          }
        ],
        "name": "docker.service"
      },
      {
        "contents": "[Unit]\nDescription=Generate etcd certs\nRequires=coreos-metadata.service\nAfter=coreos-metadata.service\nConditionFileNotEmpty=/opt/bin/generate_certs.sh\nConditionFileNotEmpty=!/etc/ssl/etcd/tls.crt\nConditionFileNotEmpty=!/etc/ssl/etcd/tls.key\nConditionFileNotEmpty=!/etc/ssl/etcd/ca.crt\n[Service]\nEnvironmentFile=/run/metadata/coreos\nEnvironment=ASSETS_DIR=/etc/ssl/etcd\nExecStartPre=/usr/bin/mkdir -p $ASSETS_DIR\nExecStartPre=/bin/sh -c \"\\\n  CERT_COMMON_NAME=etcd.aws-9405.platform-dev.r53.nordstrom.net \\\n  CERT_IP_SANS=${COREOS_EC2_IPV4_LOCAL},127.0.0.1 \\\n  CERT_ALT_NAMES=etcd.aws-9405.platform-dev.r53.nordstrom.net,${COREOS_EC2_HOSTNAME},localhost \\\n  ROOT_CA_KEY=\"/root-ca/etcd-ca.key\" \\\n  ROOT_CA_CERT=\"/root-ca/etcd-ca.crt\" \\\n  /opt/bin/generate_certs.sh\"\nExecStartPre=/bin/rm /root-ca/etcd-ca.key /root-ca/etcd-ca.crt\nExecStartPre=/usr/bin/curl -v https://discovery.etcd.io\nExecStartPre=/usr/bin/sh -c \"echo | \\\n  ope[2019-09-12T23:21:07.839669]nssl s_client -connect discovery.etcd.io:443 -prexit -showcerts 2\u003e /dev/null \\\n  | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' \\\n  \u003e /etc/ssl/etcd/ca-certificates.crt\"\nExecStart=/bin/echo \"Etcd certs generated\"\nRemainAfterExit=true\nRestart=on-failure\nRestartSec=10\n[Install]\nWantedBy=multi-user.target\n",
        "enable": true,
        "name": "generate-etcd-certs.service"
      },
      {
        "dropins": [
          {
            "contents": "[Unit]\nRequires=var-lib-etcd.mount\nAfter=var-lib-etcd.mount\nRequires=generate-etcd-certs.service\nAfter=generate-etcd-certs.service\n[Service]\nEnvironment=\"ETCD_IMAGE_URL=quay.io/coreos/etcd\"\nEnvironment=\"RKT_GLOBAL_ARGS=\"\nEnvironment=\"ETCD_STRICT_RECONFIG_CHECK=true\"\nEnvironment=\"ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:9379\"\nEnvironment=\"ETCD_DISCOVERY=https://discovery.etcd.io/9cc5ec5fe416532f25a4cde484d304f8\"\nEnvironment=\"ETCD_SSL_DIR=/etc/ssl/etcd\"\nOOMScoreAdjust=-999\nExecStartPre=/usr/bin/chown -R etcd:etcd /var/lib/etcd\nExecStartPre=/usr/bin/chown -R etcd:etcd /etc/ssl/etcd\n",
            "name": "01-conf.conf"
          }
        ],
        "enable": true,
        "name": "etcd-member.service"
      },
      {
        "contents": "[Unit]\nDescription=Generate kubelet certs\nRequires=coreos-metadata.service\nAfter=coreos-metadata.service\nConditionFileNotEmpty=/opt/bin/generate_certs.sh\nConditionFileNotEmpty=!/etc/kubernetes/tls.crt\nConditionFileNotEmpty=!/etc/kubernetes/tls.key\nConditionFileNotEmpty=!/etc/kubernetes/ca.crt\n[Service]\nEnvironmentFile=/run/metadata/coreos\nEnvironment=ASSETS_DIR=/etc/kubernetes\nExecStartPre=/usr/bin/mkdir -p $ASSETS_DIR\nExecStartPre=/bin/sh -c \"\\\n  CERT_COMMON_NAME=system:node:${COREOS_EC2_HOSTNAME} \\\n  CERT_ORGANIZATION=system:nodes \\\n  CERT_ALT_NAMES=${COREOS_EC2_HOSTNAME} \\\n  ROOT_CA_KEY=/root-ca/k8s-ca.key \\\n  ROOT_CA_CERT=/root-ca/k8s-ca.crt \\\n  /opt/bin/generate_certs.sh\"\nExecStartPre=/bin/rm /root-ca/k8s-ca.key /root-ca/k8s-ca.crt\nExecStart=/bin/echo \"Kubelet certs generated\"\nRemainAfterExit=true\nRestart=on-failure\nRestartSec=10\n[Install]\nWantedBy=multi-user.target\n",
        "enable": true,
        "name": "generate-kubelet-certs.service"
      },
      {
        "contents": "[Unit]\nDescription=Kubernetes Kubelet\nDocumentation=https://github.com/kubernetes/kubernetes\nRequires=generate-kubelet-certs.service\nAfter=generate-kubelet-certs.service\nRequires=var-lib-kubelet.mount\nAfter=var-lib-kubelet.mount\n[Service]\nEnvironmentFile=/run/metadata/coreos\nEnvironment=KUBELET_IMAGE=docker://gcr.io/google_containers/hyperkube:v1.15.3\n# additive to kubelet-wrapper\nEnvironment=\"RKT_GLOBAL_ARGS=--insecure-options=image\"\nEnvironment=\"RKT_RUN_ARGS= \\\n  --volume dns,kind=host,source=/etc/resolv.conf \\\n  --volume var-lib-cni,kind=host,source=/var/lib/cni \\\n  --volume var-log,kind=host,source=/var/log \\\n  --volume cni-bin,kind=host,source=/run/torcx/unpack/crio/usr/share/cni/bin/ \\\n  --mount volume=cni-bin,target=/opt/cni/bin \\\n  --mount volume=dns,target=/etc/resolv.conf \\\n  --mount volume=var-lib-cni,target=/var/lib/cni \\\n  --mount volume=var-log,target=/var/log\"\nExecStartPre=/bin/mkdir -p /etc/kubernet[2019-09-12T23:21:07.839995]es/manifests\nExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d\nExecStartPre=/bin/mkdir -p /etc/kubernetes/checkpoint-secrets\nExecStartPre=/bin/mkdir -p /srv/kubernetes/manifests\nExecStartPre=/bin/mkdir -p /var/lib/cni\nExecStartPre=/usr/bin/mkdir -p /var/log/containers\nExecStartPre=/bin/cp /root/.docker/config.json /var/lib/kubelet/config.json\nExecStart=/usr/lib/coreos/kubelet-wrapper \\\n  --anonymous-auth=false \\\n  --read-only-port=0 \\\n  --cgroup-driver=cgroupfs \\\n  --cloud-provider=aws \\\n  --client-ca-file=/etc/kubernetes/ca.crt \\\n  --tls-cert-file=/etc/kubernetes/tls.crt \\\n  --tls-private-key-file=/etc/kubernetes/tls.key \\\n  --cluster_dns=${COREOS_EC2_IPV4_LOCAL},25.0.0.10 \\\n  --cluster_domain=cluster.local \\\n  --exit-on-lock-contention \\\n  --feature-gates=LocalStorageCapacityIsolation=true \\\n  --lock-file=/var/run/lock/kubelet.lock \\\n  --kubeconfig=/etc/kubernetes/kubelet/kubeconfig \\\n  --network-plugin=cni \\\n  --cni-conf-dir=/etc/kubernet[2019-09-12T23:21:07.840019]es/cni/net.d \\\n  --pod-manifest-path=/etc/kubernetes/manifests \\\n  --node-labels=master=true,\"beta.nordstrom.net/node-pool\"=\"etcd-pool\",\"node-role.kubernetes.io/master\"=true \\\n  --register-with-taints=dedicate=\"etcd:NoSchedule\" \\\n  --system-reserved=\"cpu=350m,memory=3Gi\" \\\n  --eviction-hard=\"memory.available\u003c10%,nodefs.available\u003c10%,nodefs.inodesFree\u003c10%,imagefs.available\u003c10%,imagefs.inodesFree\u003c10%\" \\\n  --eviction-soft=\"memory.available\u003c15%,nodefs.available\u003c20%,nodefs.inodesFree\u003c20%,imagefs.available\u003c20%,imagefs.inodesFree\u003c20%\" \\\n  --eviction-soft-grace-period=\"memory.available=2m,nodefs.available=30s,nodefs.inodesFree=30s,imagefs.available=30s,imagefs.inodesFree=30s\" \\\n  --eviction-minimum-reclaim=\"memory.available=0Mi,nodefs.available=5Gi,imagefs.available=5Gi\" \\\n  --eviction-max-pod-grace-period=\"300\" \\\n  --authentication-token-webhook \\\n  --cpu-manager-policy=static\nRestart=always\nRes[2019-09-12T23:21:07.840040]tartSec=5\n[Install]\nWantedBy=multi-user.target\n",
        "enable": true,
        "name": "kubelet.service"
      }
    ]
  }
```
dharmab commented 5 years ago

Try reparitioning the device in Ignition in addition to formatting the filesystem:

  disks:
    # Repartition NVME Instance Storage
    - device: /dev/nvme1n1
      wipe_table: True
      partitions:
        - number: 0
          size: 0
          start: 0
  filesystems:
    # Format NVME Instance Storage
    - mount:  # Not actually mounting, just formatting
        device: /dev/nvme1n1p1
        format: ext4