openshift / os

90 stars 107 forks source link

SCOS: rhcos.upgrade.luks failed #922

Closed HuijingHei closed 2 years ago

HuijingHei commented 2 years ago

From CI log in https://github.com/openshift/os/pull/905

=== RUN   rhcos.upgrade.luks/verify
--- FAIL: rhcos.upgrade.luks (854.93s)
    --- PASS: rhcos.upgrade.luks/setup (99.47s)
    --- FAIL: rhcos.upgrade.luks/upgrade-from-previous (613.83s)
            upgrade.go:165: Failed to reboot machine: machine "1c8a2489-3065-4c59-a847-13a7055a0ae5" failed to start: ssh journalctl failed: time limit exceeded
    --- SKIP: rhcos.upgrade.luks/verify (0.00s)
            cluster.go:54: A previous test has already failed
HuijingHei commented 2 years ago

Boot VM with 46.82.202011260640-0, after rebase with 412.91.202207290659-0, reboot failed. See https://github.com/openshift/os/blob/master/overlay.d/05rhcos/usr/libexec/coreos-cryptfs#L104 or https://github.com/openshift/os/blob/master/overlay.d/05rhcos/usr/libexec/coreos-cryptfs#L107

[   13.519204] coreos-cryptfs[548]: Usage: cryptsetup [-?Vqrvy] [-?|--help] [--usage] [-V|--version]
[   13.529700] coreos-cryptfs[548]:         [--active-name=STRING] [--align-payload=SECTORS] [--allow-discards]
[   13.541068] coreos-cryptfs[548]:         [-q|--batch-mode] [--cancel-deferred] [-c|--cipher=STRING] [--debug]
[   13.552711] coreos-cryptfs[548]:         [--debug-json] [--deferred] [--device-size=bytes] [--decrypt]
[   13.563861] coreos-cryptfs[548]:         [--disable-external-tokens] [--disable-keyring] [--disable-locks]
[   13.575109] coreos-cryptfs[548]:         [--disable-veracrypt] [--dump-json-metadata] [--dump-master-key]
[   13.586134] coreos-cryptfs[548]:         [--encrypt] [--force-password] [-h|--hash=STRING] [--header=STRING]
[   13.597555] coreos-cryptfs[548]:         [--header-backup-file=STRING] [--hotzone-size=bytes] [--init-only]
[   13.608853] coreos-cryptfs[548]:         [-I|--integrity=STRING] [--integrity-legacy-padding]
[   13.618877] coreos-cryptfs[548]:         [--integrity-no-journal] [--integrity-no-wipe] [-i|--iter-time=msecs]
[   13.630542] coreos-cryptfs[548]:         [--iv-large-sectors] [--json-file=STRING] [--key-description=STRING]
[   13.641917] coreos-cryptfs[548]:         [-d|--key-file=STRING] [-s|--key-size=BITS] [-S|--key-slot=INT]
[   13.652661] coreos-cryptfs[548]:         [--keyfile-offset=bytes] [-l|--keyfile-size=bytes]
[   13.662705] coreos-cryptfs[548]:         [--keyslot-cipher=STRING] [--keyslot-key-size=BITS] [--label=STRING]
[   13.674304] coreos-cryptfs[548]:         [--luks2-keyslots-size=bytes] [--luks2-metadata-size=bytes]
[   13.685430] coreos-cryptfs[548]:         [--master-key-file=STRING] [--new-keyfile-offset=bytes]
[   13.695864] coreos-cryptfs[548]:         [--new-keyfile-size=bytes] [-o|--offset=SECTORS] [--pbkdf=STRING]
[   13.706941] coreos-cryptfs[548]:         [--pbkdf-force-iterations=LONG] [--pbkdf-memory=kilobytes]
[   13.717498] coreos-cryptfs[548]:         [--pbkdf-parallel=threads] [--perf-no_read_workqueue]
[   13.727726] coreos-cryptfs[548]:         [--perf-no_write_workqueue] [--perf-same_cpu_crypt]
[   13.737608] coreos-cryptfs[548]:         [--perf-submit_from_crypt_cpus] [--persistent] [--priority=STRING]
[   13.748853] coreos-cryptfs[548]:         [--progress-frequency=secs] [-r|--readonly]
[   13.757867] coreos-cryptfs[548]:         [--reduce-device-size=bytes] [--refresh] [--resilience=STRING]
[   13.768626] coreos-cryptfs[548]:         [--resilience-hash=STRING] [--resume-only] [--sector-size=INT]
[   13.779534] coreos-cryptfs[548]:         [--serialize-memory-hard-pbkdf] [--shared] [-b|--size=SECTORS]
[   13.790857] coreos-cryptfs[548]:         [-p|--skip=SECTORS] [--subsystem=STRING] [--tcrypt-backup]
[   13.801307] coreos-cryptfs[548]:         [--tcrypt-hidden] [--tcrypt-system] [--test-args]
[   13.811019] coreos-cryptfs[548]:         [--test-passphrase] [-t|--timeout=secs] [--token-id=INT]
[   13.821849] coreos-cryptfs[548]:         [--token-only] [--token-type=STRING] [-T|--tries=INT]
[   13.839333] coreos-cryptfs[548]:         [-M|--type=STRING] [--unbound] [--use-random] [--use-urandom]
[   13.850034] coreos-cryptfs[548]:         [--uuid=STRING] [--veracrypt] [--veracrypt-pim=INT]
[   13.859933] coreos-cryptfs[548]:         [--veracrypt-query-pim] [-v|--verbose] [-y|--verify-passphrase]
[   13.871968] coreos-cryptfs[548]:         [OPTION...] <action> <action-specific>
[   13.880560] coreos-cryptfs[548]: cryptsetup: invalid numeric value
[   13.900533] coreos-cryptfs[520]: coreos-cryptfs: /dev/vda4 is configured for Clevis pin ''
[   13.910192] coreos-cryptfs[520]: coreos-cryptfs: Checking for default route.
[   13.918490] coreos-cryptfs[520]: coreos-cryptfs: Waiting for DNS resolver to appear.
[   13.927548] systemd[1]: Starting CoreOS LUKS Opener...
[   13.934267] systemd[1]: nm-initrd.service was skipped because of a failed condition check (ConditionPathExists=/run/NetworkManager/initrd/neednet).
[   13.949546] systemd[1]: Reached target Network.
[   13.955404] systemd[1]: nm-wait-online-initrd.service was skipped because of a failed condition check (ConditionPathExists=/run/NetworkManager/initrd/need.
[   13.971993] systemd[1]: Starting dracut initqueue hook...
[   13.978689] systemd[1]: Finished dracut initqueue hook.
[   13.985257] systemd[1]: Reached target Preparation for Remote File Systems.
[   13.993726] systemd[1]: Reached target Remote Encrypted Volumes.
[   14.000938] systemd[1]: Reached target Remote File Systems.
[   14.007708] systemd[1]: Acquire live PXE rootfs image was skipped because of a failed condition check (ConditionPathExists=/run/ostree-live).
[   14.023742] systemd[1]: Persist osmet files (PXE) was skipped because of a failed condition check (ConditionPathExists=/run/ostree-live).
[   14.037921] systemd[1]: Starting dracut pre-mount hook...
[   14.044619] systemd[1]: Finished dracut pre-mount hook.
[**    ] (2 of 2) A start job is running for…isk/by-label/root (33s / 1min 30s)
[FAILED] Failed to start CoreOS LUKS Opener.
See 'systemctl status coreos-luks-open.service' for details.
[DEPEND] Dependency failed for Igni…on Subsequent Boot Disk Setup.
[DEPEND] Dependency failed for Subs… (Not Ignition) boot complete.
[DEPEND] Dependency failed for Initrd Default Target.
travier commented 2 years ago

Pushed 255ec69 (#905), let's see if that's the root of the issue.

HuijingHei commented 2 years ago

Thanks @travier for the fix! Close this issue.

=== RUN   rhcos.upgrade.luks/setup
Imported: acf4cb815eb9c9f03bd3aa7b147ec0b2f38b706e4a3e90797763638bdc85e1ff
0 metadata, 0 content objects imported; 0 bytes content written
=== RUN   rhcos.upgrade.luks/upgrade-from-previous
=== RUN   rhcos.upgrade.luks/verify
--- PASS: rhcos.upgrade.luks (420.38s)
    --- PASS: rhcos.upgrade.luks/setup (153.28s)
    --- PASS: rhcos.upgrade.luks/upgrade-from-previous (90.91s)
    --- PASS: rhcos.upgrade.luks/verify (0.39s)