fedora iot-simplified-installer fdo re-encryption failed

yih-redhat commented 1 year ago

Describe the bug provision edge vm with iot-simplified-installer, install a failing health check unit and rollback, then check fdo re-encryption by command "cryptsetup luksDump /dev/vda3". The expected result is there is no "cipher_null-ecb" in output, but actually "cipher_null-ecb" is in the output. The same test passed on rhel and centos-stream, the difference is we check /dev/vda4 on these os, with command "cryptsetup luksDump /dev/vda4", so I guess maybe the root reason is that fedora image only has /dev/vda3, and rhel/centos has /dev/vda4.

Environment

OS version (/etc/os-release and /etc/redhat-release):
osbuild-composer version (rpm -qi osbuild-composer)

To Reproduce Steps to reproduce the behavior:

build iot-simplified-installer and provision vm.
install sanely failing health check unit and reboot "rpm-ostree install --cache-only https://s3.amazonaws.com/org.osbuild.test-dependencies/greenboot-failing-unit-1.0-1.el8.noarch.rpm --reboot"
vm failed to boot and then rollback automatically.
check fdo re-encryption status. [simple@localhost ~]$ sudo cryptsetup luksDump /dev/vda3 [sudo] password for simple: LUKS header information Version: 2 Epoch: 6 Metadata area: 16384 [bytes] Keyslots area: 16744448 [bytes] UUID: 05ad1795-54bc-4a57-bb01-9082f86a774d Label: crypt_root Subsystem: (no subsystem) Flags: (no flags)

Data segments: 0: crypt offset: 16777216 [bytes] length: (whole device) cipher: cipher_null-ecb sector: 512 [bytes]

Keyslots: 1: luks2 Key: 256 bits Priority: normal Cipher: aes-xts-plain64 Cipher key: 512 bits PBKDF: pbkdf2 Hash: sha256 Iterations: 1000 Salt: 6c 19 c0 8e 05 f0 05 21 42 70 98 5a 07 c9 19 8a d0 7a d8 ef 16 14 95 be 94 9e d2 d8 46 bf 16 0f AF stripes: 4000 AF hash: sha256 Area offset:163840 [bytes] Area length:131072 [bytes] Digest ID: 0 Tokens: 0: clevis Keyslot: 1 Digests: 0: pbkdf2 Hash: sha256 Iterations: 1000 Salt: 75 96 82 66 56 55 02 a1 0a 63 58 db b2 c9 60 fd 3b cd 8d fe ef cf 39 76 73 7d 68 8e b0 6f f7 aa Digest: b9 fb 7b a8 6f 2b 91 20 e2 8f b7 b4 2a 6f 67 09 7e bf b3 2b 45 2b c1 1c be 23 d9 dc e0 54 f2 48

Expected behavior there is no "cipher_null-ecb" in output

Additional context Add any other context about the problem here.

yih-redhat commented 1 year ago

@runcom @7flying Could you please take a look of this bug? And I found a similar bug was filed before https://bugzilla.redhat.com/show_bug.cgi?id=2220851

yih-redhat commented 1 year ago

fdo client log:

[simple@localhost home]$ journalctl -u fdo-client-linuxapp.service Display all 177 possibilities? (y or n) [simple@localhost home]$ journalctl -u fdo-client-linuxapp.service Oct 09 13:49:10 localhost.localdomain systemd[1]: Starting fdo-client-linuxapp.service - FDO client... Oct 09 13:49:10 localhost.localdomain fdo-client-linuxapp[1077]: 2023-10-09T13:49:10.386Z INFO fdo_client_linuxapp > No usable device> Oct 09 13:49:10 localhost.localdomain systemd[1]: fdo-client-linuxapp.service: Deactivated successfully. Oct 09 13:49:10 localhost.localdomain systemd[1]: Finished fdo-client-linuxapp.service - FDO client. -- Boot cd6c4a0071004e97bf4bae580ac5d3d1 -- Oct 09 13:50:43 localhost.localdomain systemd[1]: Starting fdo-client-linuxapp.service - FDO client... Oct 09 13:50:43 localhost.localdomain fdo-client-linuxapp[956]: 2023-10-09T13:50:43.400Z INFO fdo_client_linuxapp > No usable device > Oct 09 13:50:43 localhost.localdomain mv[976]: /usr/bin/mv: cannot stat '/boot/device-credentials': No such file or directory Oct 09 13:50:43 localhost.localdomain systemd[1]: fdo-client-linuxapp.service: Deactivated successfully. Oct 09 13:50:43 localhost.localdomain systemd[1]: Finished fdo-client-linuxapp.service - FDO client.

fdo aio serviceinfo config file: [root@yih-f38 ~]# cat /etc/fdo/aio/configs/serviceinfo_api_server.yml service_info: initial_user: username: fdouser sshkeys:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkMAjwQdrK0LBzcyQxEsJQIxlRNQaKbH/eGI3lPzoFumY+HdxoktJKPDZCibB2LyIfOHu5qFj4HSeLSFy18TzydrgQiZxIgrZOvQyfaEutHxfNLbDxU5GSFqss43Nv3qnCCvac8K62m/nWmhQvyuGXZuRs/VFA/RxzjH8TYYQXLv0c66LXX2cH6Xs92gkL1YKjW0EEkySb+MsMGSl8CzFLlFHPNNRO9neAreLHAVL0xzPsHFDYti4Hb2xnL7LEy7iTB5e0gmcZ1/jyfIOc70dkq1a+/AXxCSGsDcdWahwBoKOYDRG0JJeYG3cw2zJXYp2H4/H3wVE0wxirejwkIwY+2BGOcO7W3LUHuWouPQ4idv+gZznDIz+Vt14IJyiiVz5ADNM9zeqHAYwQHvoj1/xokiLRbsH+zC+QzHsXHx4g7LNfyDKsX7rKmV9AKRQi6z0ypwAozKF3eTFHaUJNnEv4F5DJ5qiZiNtMv1jyJ+p4c2QZwSPEeSivCkegkIIuBWU= fedora@yih-f38 files:
- path: /etc/sudoers.d/fdouser source_path: /var/lib/fdo/fdouser commands: null diskencryption_clevis:
- disk_label: /dev/vda3 reencrypt: true binding: pin: tpm2 config: '{}' additional_serviceinfo: null after_onboarding_reboot: false bind: 0.0.0.0:8083 service_info_auth_token: c3iJ+YAIT7PzmCA90z9bLISn2/OdJwrXVm9e4C258ig= admin_auth_token: pfQHrBgMVvPwzctW439fsqtF4urNi/Mxc10pDcdp+Rc= device_specific_store_driver: Directory: path: /etc/fdo/aio/stores/serviceinfo_api_devices

7flying commented 1 year ago

@yih-redhat we need the logs from the manufacturing-client.service, if you have them

@runcom @7flying Could you please take a look of this bug? And I found a similar bug was filed before https://bugzilla.redhat.com/show_bug.cgi?id=2220851

that one was a TPM issue, don't know yet if this will also be the case, but we'll need the manufacturing logs.

Thanks

henrywang commented 1 year ago

This might be selinux issue, I think. @yih-redhat Could you please check the selinux fix in RHEL 9 in Fedora? Thanks.

yih-redhat commented 1 year ago

This might be selinux issue, I think. @yih-redhat Could you please check the selinux fix in RHEL 9 in Fedora? Thanks.

sure, could you please let me know the steps to check the selinux fix?

yih-redhat commented 1 year ago

I do found below denied avc log for /tmp/fdouser, but after I changed to use /var/lib/fdo/fdouser, I didn't see it anymore.

type=AVC msg=audit(10/09/2023 08:35:03.580:9660) : avc: denied { open } for pid=232024 comm=fdo-serviceinfo path=/tmp/fdouser dev="tmpfs" ino=251 scontext=system_u:system_r:fdo_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=file permissive=0

henrywang commented 1 year ago

It's selinux issue. The selinux fdo fix in RHEL should be landed in Fedora.

7flying commented 1 year ago

This is what I've got:

ct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:  2023-10-19T14:54:53.232Z INFO  fdo_client_linuxapp > Got TO2 addresses: ["http://192.168.122.180:8081", "http://fe80::97e2:1716:6aa8:88ba:8081"]
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:  2023-10-19T14:54:53.233Z INFO  fdo_client_linuxapp > Performing TO2 protocol, URL: "http://192.168.122.180:8081"
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:  2023-10-19T14:54:53.277Z INFO  fdo_client_linuxapp::serviceinfo > Initiating disk re-encryption, disk-label: /dev/vda3, pin: tpm2, config: {}, reencrypt: true
Oct 19 14:54:53 localhost.localdomain audit[1392]: AVC avc:  denied  { search } for  pid=1392 comm="pwmake" name="cracklib" dev="dm-1" ino=37010 scontext=system_u:system_r:fdo_t:s0 tcontext=system_u:object_r:crack_db_t:s0 tclass=dir permissive=0
Oct 19 14:54:53 localhost.localdomain audit[1392]: AVC avc:  denied  { search } for  pid=1392 comm="pwmake" name="cracklib" dev="dm-1" ino=37010 scontext=system_u:system_r:fdo_t:s0 tcontext=system_u:object_r:crack_db_t:s0 tclass=dir permissive=0
Oct 19 14:54:53 localhost.localdomain audit[1392]: AVC avc:  denied  { search } for  pid=1392 comm="pwmake" name="cracklib" dev="dm-1" ino=37010 scontext=system_u:system_r:fdo_t:s0 tcontext=system_u:object_r:crack_db_t:s0 tclass=dir permissive=0
Oct 19 14:54:53 localhost.localdomain audit[1392]: AVC avc:  denied  { search } for  pid=1392 comm="pwmake" name="cracklib" dev="dm-1" ino=37010 scontext=system_u:system_r:fdo_t:s0 tcontext=system_u:object_r:crack_db_t:s0 tclass=dir permissive=0
Oct 19 14:54:53 localhost.localdomain audit[1392]: AVC avc:  denied  { search } for  pid=1392 comm="pwmake" name="cracklib" dev="dm-1" ino=37010 scontext=system_u:system_r:fdo_t:s0 tcontext=system_u:object_r:crack_db_t:s0 tclass=dir permissive=0
Oct 19 14:54:53 localhost.localdomain audit[1392]: AVC avc:  denied  { search } for  pid=1392 comm="pwmake" name="cracklib" dev="dm-1" ino=37010 scontext=system_u:system_r:fdo_t:s0 tcontext=system_u:object_r:crack_db_t:s0 tclass=dir permissive=0
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:  2023-10-19T14:54:53.413Z ERROR fdo_client_linuxapp              > ServiceInfo failed, error: Error processing returned serviceinfo
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]: Caused by:
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:     0: Error executing clevis
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:     1: Error executing disk encryption for disk label /dev/vda3
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:     2: Error rebinding clevis
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:     3: Error binding clevis
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:     4: Failed to bind clevis: ExitStatus(unix_wait_status(256)), stdout: , stderr:
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:        /usr/share/cracklib/pw_dict.pwd.gz: Permission denied
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:        /usr/share/cracklib/pw_dict.pwd.gz: Permission denied
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:        /usr/share/cracklib/pw_dict.pwd.gz: Permission denied
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:        Error: Password generation failed - required entropy too low for settings
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:        Unable to generate a new key
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:        Error adding new binding to /dev/vda3

So, it is selinux, but not our typical case!

7flying commented 1 year ago

Linked the issue to our issue tracker: https://github.com/fedora-iot/iot-distro/issues/8

yih-redhat commented 1 year ago

Verifying this bug with build https://koji.fedoraproject.org/koji/buildinfo?buildID=2320649

yih-redhat commented 1 year ago

Verified this bug with build https://koji.fedoraproject.org/koji/buildinfo?buildID=2320649, fixed, the fdo re-encryption works as expected.

osbuild / osbuild-composer

fedora iot-simplified-installer fdo re-encryption failed #3726