Feature Request

Luks PIN for IBM CEX (Cryptograpic Express Card) similar to TPM for Encrypting root volume LUKS with secureKeys by using IBM CEX card co-processor CCA . I am in process of enabling CEX cryptograhic card in S390x for Luks Disk Encryption specifically for root volume.

The main aim is to implement in OCP to utilize protected key generated from utility zkey where each CEX (CCA)cards will be assigned to each OCP nodes with Master key loaded. For that this feature requires in RHEL. The secure key get generated from the master key in the cryptographic co-processor and the effective key in securekey can be only derived inside cryptogrpahic cards with master key residing in the card.

Environment

IBM S390x.

What hardware/cloud provider/hypervisor is being used to run Ignition?

IBM S390x.

Desired Feature

A CEX (Hardware Security Module) in LUKS PIN for root device encryption. Similar to TPM2.

Other Information

Have tested by creating volume for pervasive encryption in s390x. similarly using for root device LUKS encryption with protected key.

https://www.ibm.com/docs/en/linux-on-systems?topic=volumes-creating-volume-pervasive-encryption

From my understanding, this looks like something that should be added to Clevis, not Ignition.

Hi @travier , As recommened by @cgwalters i raised an issue in upstream ignition. to discuss further on this. If it is not here i'll raise it in Clevis then.

Hi @travier @cgwalters ,

Since there is no response from clevis side. So I decided to do some research on with Rhcos qcow2 image.

I am using CEX card for luks encryption. Here is the command I am using to spin up the kvm guest.

qemu-system-s390x -machine s390-ccw-virtio,accel=kvm -m 4096 -smp 2 -nographic -drive if=none,id=hda,file=/tmp/rhcos.qcow2,auto-read-only=off,cache=unsafe -device virtio-blk,iommu_platform=on,drive=hda -netdev bridge,br=virbr0,id=mynet0 -device virtio-net-ccw,iommu_platform=on,netdev=mynet0 -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/ca3dcf2d-1efb-486e-9022-ed81c96f5829

here is my ignition file which I infuse on the qcow2 image before the above command. The key_file I am using for encryption is the secure key generated from CEX card.

The guest vm spins up and I am able to login and the root device get luks encrypted. However whilst rebooting I am getting passphrase prompt.

I think the passphrase is getting generated by luks itself. Is there anyway we can derive or pass as an argument in ignition?.

[  OK  ] Finished Wait for udev To Complete Device Initialization.
[    2.298276] systemd[1]: Finished Wait for udev To Complete Device Initialization.
[    2.299346] systemd[1]: Device-Mapper Multipath Default Configuration was skipped because of an unmet condition check (ConditionKernelCommandLine=rd.multipath=default).
         Starting Device-Mapper Multipath Device Controller...
[    2.300508] systemd[1]: Starting Device-Mapper Multipath Device Controller...
[    2.313272] multipathd[612]: --------start up--------
[    2.313305] multipathd[612]: read /etc/multipath.conf
[    2.313327] multipathd[612]: /etc/multipath.conf does not exist, blacklisting all devices.
[    2.313339] multipathd[612]: You can run "/sbin/mpathconf --enable" to create
[    2.313351] multipathd[612]: /etc/multipath.conf. See man mpathconf(8) for more details
[    2.315726] multipathd[612]: path checkers start up
[    2.316617] multipathd[612]: /etc/multipath.conf does not exist, blacklisting all devices.
[    2.316642] multipathd[612]: You can run "/sbin/mpathconf --enable" to create
[    2.316656] multipathd[612]: /etc/multipath.conf. See man mpathconf(8) for more details
[  OK  ] Started Device-Mapper Multipath Device Controller.
[    2.338474] systemd[1]: Started Device-Mapper Multipath Device Controller.
[    2.389610] systemd[1]: Reached target Preparation for Local File Systems.
[    2.389705] systemd[1]: Reached target Local File Systems.
[    2.389876] systemd[1]: Started Forward Password Requests to Clevis.
[    2.389920] systemd[1]: Started Dispatch Password Requests to Console.
🔐 Please enter passphrase for disk root: (press TAB for no echo) [    2.403666] systemd-tty-ask-password-agent[638]: Starting password query on /dev/ttysclp0.

IGNITION FILE

variant: fcos
version: 1.4.0
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-ed25519 NzaC1lZDI1NTE5AAAAINQvDF2S5sGktl88+DEMhb6jAf8gdAZi0+sifheG47ov  
storage:
  files:
    - path: /etc/sshd/sshd_config.d/20-enable-passwords.conf
      mode: 0644
      contents:
        inline: |
          PasswordAuthentication yes
    - path: /etc/zkey/repository/secure_xtskey1.skey
      mode: 0644
      contents:
        source: "data:;base64,AQAAAAQAwHRx7PRqNb2JJPhly8r54bb6SDszPi+e2yvIqx1xMK4TB4RBnRTRsJ5OAAAAAAAAAAABAAAgZxg7KQEAAAAEAMD3cez0ajW9iSRzg4weEfotBdyIMNw2sGTPialfnUPOpyR2HxZvbpgGbwAAAAAAAAAAAQAAIPiQsRI="
  disks:
    - device: /dev/vda
      partitions:
        - label: root
          number: 4
          resize: true
          size_mib: 0
  luks:
    - name: root
      label: luks-root
      device: /dev/disk/by-partlabel/root
      wipe_volume: true
      key_file:
        source: "data:;base64,AQAAAAQAwHRx7PRqNb2JJPhly8r54bb6SDszPi+e2yvIqx1xMK4TB4RBnRTRsJ5OAAAAAAAAAAABAAAgZxg7KQEAAAAEAMD3cez0ajW9iSRzg4weEfotBdyIMNw2sGTPialfnUPOpyR2HxZvbpgGbwAAAAAAAAAAAQAAIPiQsRI="
      options:
         - "--pbkdf"
         - "pbkdf2"
         - "--cipher"
         - "paes-xts-plain64"
         - "--key-size"
         - "512"

  filesystems:
    - device: /dev/mapper/root
      format: xfs
      label: root
      wipe_filesystem: true

The SecureKey Generated from the CEX card .

[root@bastion ~]# lszcrypt -VVV
CARD.DOM TYPE  MODE        STATUS     REQUESTS  PENDING HWTYPE QDEPTH FUNCTIONS  DRIVER     
--------------------------------------------------------------------------------------------
00       CEX6C CCA-Coproc  online           32        0     12     08 S--D--NF-  cex4card   
00.0047  CEX6C CCA-Coproc  online           32        0     12     08 S--D--NF-  cex4queue  
[root@bastion ~]# zkey list
Key                          : secure_xtskey1                                        
-------------------------------------------------------------------------------------
        Description          : 
        Secure key size      : 128 bytes
        Clear key size       : 512 bits
        XTS type key         : Yes
        Key type             : CCA-AESDATA
        Volumes              : (none)
        APQNs                : 00.0047
        Key file name        : /etc/zkey/repository/secure_xtskey1.skey
        Sector size          : 4096 bytes
        Volume type          : LUKS2
        Verification pattern : 755e7875ececc3cd8a898262717cf109
                               05e29a802a4e6fd5582ac963aea2f827
        KMS                  : (local)
        KMS key label        : (local)
        Dummy passphrase     : /etc/zkey/repository/secure_xtskey1.pass
        Created              : 2023-09-14 06:38:35
        Changed              : (never)
        Re-enciphered        : (never)

The key file encrypted in the Guest system.

[root@localhost ~]# zkey validate /etc/zkey/repository/secure_xtskey1.skey 
Validation of secure key in file '/etc/zkey/repository/secure_xtskey1.skey':
  Status:                Valid
  Secure key size:       128 bytes
  Key type:              CCA-AESDATA
  Clear key size:        512 bits
  XTS type key:          Yes
  Enciphered with:       CURRENT master key (MKVP: 71ecf46a35bd8924)
  Verification pattern:  755e7875ececc3cd8a898262717cf109
                         05e29a802a4e6fd5582ac963aea2f827

[root@localhost ~]# zkey validate /etc/luks/root 
Validation of secure key in file '/etc/luks/root':
  Status:                Valid
  Secure key size:       128 bytes
  Key type:              CCA-AESDATA
  Clear key size:        512 bits
  XTS type key:          Yes
  Enciphered with:       CURRENT master key (MKVP: 71ecf46a35bd8924)
  Verification pattern:  755e7875ececc3cd8a898262717cf109
                         05e29a802a4e6fd5582ac963aea2f827

The guest vm spins up and I am able to login and the root device get luks encrypted. However whilst rebooting I am getting passphrase prompt.

I think the passphrase is getting generated by luks itself. Is there anyway we can derive or pass as an argument in ignition?.

It's asking for a password because the keyfile is not accessible since it's on the encrypted rootfs. (The key data provided is written to a file and stored in /etc/luks/ and referenced in crypttab.)

I'd agree with @travier; this looks like support needs to go in Clevis instead, and then it could be exposed in Ignition via something like:

luks:
  name: root,
  label: luks-root
  device: /dev/disk/by-partlabel/root
  clevis:
    cex: true

(This isn't strictly necessary, since Ignition does support custom Clevis pins too, i.e. luks[].clevis.custom.)

Adding it to Clevis means it would also be supported by traditional RHEL and in theory any other environment where Clevis is supported.

Hi @jlebon and @travier , Need a help here. I am stuck at the creating the logic to encrypt and decrypt using the PIN specific to CEX card. I've written the concept of clevis encrypt CEX and clevis decrypt CEX. There is problem here

Whilst using the CEX card for encryption the administrator are limited to issue few commands generating the secure keys, reencipher etc with certain parameters. As there is no specific command we can use to interact to decrypt or encrypt via the device. It is taken care by the dmcrypt that pass to PAES kernel module which internally generate a protected key for data encryption. The following images shows the work flow.

For opening a device

As I understood from the TPM2 is the encryption and decryption can be done with TPM2 device where we can seal and unseal generating keys etc can be interacted with tpm device with commands. Similarly to Tang (NDBE) We get the thp we generate jws from that we extract the payload -> then perform the enc generate the jwk from the enc

However when we do similar concept to create CEX PIN for encryption, the master key resides in the CEX controller which we do not have a control. we can create the secure keys use that keys as the master key to format luks device eg here .

Generate the secure key zkey generate --name secure_xtskey1 --keybits 256 --xts --volumes /dev/mapper/disk1:enc-disk1 --volume-type LUKS2 --sector-size 4096 --apqns 03.0039,04.0039

Format the volume zkey cryptsetup --volumes /dev/mapper/disk1 - #This generates the command to run for luks encryption or by add ing the arguments --run will execute the following command.

cryptsetup luksFormat --type luks2 --pbkdf pbkdf2 --master-key-file '/etc/zkey/repository/secure_xtskey1.skey' --key-size 1024 --cipher paes-xts-plain64 --sector-size 4096 /dev/mapper/disk1

zkey-cryptsetup setvp /dev/mapper/disk1 #It ask for passphrase

Similarly we can open the device with cryptsetup luksOpen command.

There is no specific command or method we can use these keys to encrypt or decrypt a plaintext that we expected to do with PIN concept. The usage is of key is limited to cryptsetup

The other problem is the secure key is getting generated from a single command which is XTS key that can only be reside in a file because it contain null bytes (gibberish). Something like below this is getting wrapped by CRYPTO master key in the Crypto Card Controller.

[root@fedora ~]# cat secure_xtskey1.skey ??q??j5??$Z#?#? ?j?[??FX??ґO?nwtr?"? .xT+??q??j5??$???yS۽????? ?2??U?A?| ܼ?? ??H=

Certainly, we can convert that to base64 encrypted but decryption must be to a file not on a variable. But the issue here is secure key file processing happened at kernel level as explained in the work flow.

Looking for your valuable input.

Hi @jlebon , Here is the procedure we use to luks format the volume using IBM CryptoAdapter in Rhel with manual command.

The prerequisites are

[ root@linux ] # lsmod | grep pkey
pkey                   36864  1 paes_s390
zcrypt                126976  2 pkey,zcrypt_cex4

IBM Crypto Card must be present in the system.

[root@linux ] #lszcrypt -V
CARD.DOM TYPE  MODE        STATUS     REQUESTS  PENDING HWTYPE QDEPTH FUNCTIONS  DRIVER     
--------------------------------------------------------------------------------------------
00       CEX6C CCA-Coproc  online           21        0     12     08 S--D--NF-  cex4card   
00.0011  CEX6C CCA-Coproc  online           21        0     12     08 S--D--NF-  cex4queue

Generate the Secure Keys with following commands

 [root@linux ] # zkey generate --name secure_xtskeys --keybits 256 --xts  --volume-type LUKS2 --sector-size 4096 --apqns 00.0011
INFO: APQN 00.0011: The OLD master key register contains the same master key as
the CURRENT master key register.

CARD.DOMAIN NEW MK           CURRENT MK       OLD MK           TYPE  
---------------------------------------------------------------------
00.0011     -                71ecf46a35bd8924 71ecf46a35bd8924 CEX6C

Listing the key

[root@linux ~]# zkey list
Key                          : secure_xtskeys                                        
-------------------------------------------------------------------------------------
        Description          : 
        Secure key size      : 128 bytes
        Clear key size       : 512 bits
        XTS type key         : Yes
        Key type             : CCA-AESDATA
        Volumes              : (none)
        APQNs                : 00.0011
        Key file name        : /etc/zkey/repository/secure_xtskeys.skey
        Sector size          : 4096 bytes
        Volume type          : LUKS2
        Verification pattern : b7e6c63a1b89fb31e77bf03bea518eff
                               313f2d13f997fc0f5b8146026051c3d1
        KMS                  : (local)
        KMS key label        : (local)
        Dummy passphrase     : (none)
        Created              : 2024-01-19 18:31:29
        Changed              : (never)
        Re-enciphered        : (never

Formatting the luks volume with the secure key generated above. On debug for granularity.

[root@linux ~]# cryptsetup luksFormat --type luks2 --pbkdf pbkdf2 --master-key-file '/etc/zkey/repository/secure_xtskeys.skey' --key-size 1024 --cipher paes-xts-plain64 --sector-size 4096 /dev/sdc1 --debug
# cryptsetup 2.4.3 processing "cryptsetup luksFormat --type luks2 --pbkdf pbkdf2 --master-key-file /etc/zkey/repository/secure_xtskeys.skey --key-size 1024 --cipher paes-xts-plain64 --sector-size 4096 /dev/sdc1 --debug"
# Running command luksFormat.
# Locking memory.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating context for crypt device /dev/sdc1.
# Trying to open and read device /dev/sdc1 with direct-io.
# Initialising device-mapper backend library.

WARNING!
========
This will overwrite data on /dev/sdc1 irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
# Interactive passphrase entry requested.
Enter passphrase for /dev/sdc1: 
Verify passphrase: 
# Checking new password using default pwquality settings.
# New password libpwquality score is 21.
# Crypto backend (OpenSSL 3.0.1 14 Dec 2021 [default][legacy]) initialized in cryptsetup library version 2.4.3.
# Detected kernel Linux 5.14.0-162.6.1.el9_1.s390x s390x.
# PBKDF pbkdf2-sha256, time_ms 2000 (iterations 0).
# Formatting device /dev/sdc1 as type LUKS2.
# dm version   [ opencount flush ]   [16384] (*1)
# dm versions   [ opencount flush ]   [16384] (*1)
# Detected dm-ioctl version 4.46.0.
# Device-mapper backend running with UDEV support enabled.
# Topology: IO (32768/0), offset = 0; Required alignment is 1048576 bytes.
# Formatting LUKS2 with JSON metadata area 12288 bytes and keyslots area 16744448 bytes.
# Creating new digest 0 (pbkdf2).
# Setting PBKDF2 type key digest 0.
# Running pbkdf2(sha256) benchmark.
# PBKDF benchmark: memory cost = 0, iterations = 455111, threads = 0 (took 72 ms)
# PBKDF benchmark: memory cost = 0, iterations = 443560, threads = 0 (took 591 ms)
# Benchmark returns pbkdf2(sha256) 443560 iterations, 0 memory, 0 threads (for 1024-bits key).
# Segment 0 assigned to digest 0.
# Wiping LUKS areas (0x000000 - 0x1000000) with zeroes.
# Wiping keyslots area (0x008000 - 0x1000000) with random data.
# Reusing open rw fd on device /dev/sdc1
# Device size 21473787904, offset 16777216.
# Acquiring write lock for device /dev/sdc1.
# Opening lock resource file /run/cryptsetup/L_8:33
# Verifying lock handle for /dev/sdc1.
# Device /dev/sdc1 WRITE lock taken.
# Trying to write LUKS2 header (16384 bytes) at offset 0.
# Reusing open rw fd on device /dev/sdc1
# Checksum:b2d8f84634bcede8709014727cfd3267c0d8c494f8264eb9bba103d39e5765cb (in-memory)
# Trying to write LUKS2 header (16384 bytes) at offset 16384.
# Reusing open rw fd on device /dev/sdc1
# Checksum:00f77b973edcb077822a215432bc859d993742133aef0c284b0579676957f85e (in-memory)
# Device /dev/sdc1 WRITE lock released.
# Adding new keyslot -1 using volume key.
# Adding new keyslot -1 with volume key assigned to a crypt segment.
# Selected keyslot 0.
# Keyslot 0 assigned to digest 0.
# Trying to allocate LUKS2 keyslot 0.
# Found area 32768 -> 544768
# Running pbkdf2(sha256) benchmark.
# PBKDF benchmark: memory cost = 0, iterations = 1092266, threads = 0 (took 30 ms)
# PBKDF benchmark: memory cost = 0, iterations = 891646, threads = 0 (took 588 ms)
# Benchmark returns pbkdf2(sha256) 891646 iterations, 0 memory, 0 threads (for 512-bits key).
# Calculating attributes for LUKS2 keyslot 0.
# Acquiring write lock for device /dev/sdc1.
# Opening lock resource file /run/cryptsetup/L_8:33
# Verifying lock handle for /dev/sdc1.
# Device /dev/sdc1 WRITE lock taken.
# Checking context sequence id matches value stored on disk.
# Opening locked device /dev/sdc1
# Verifying locked device handle (bdev)
# Running keyslot key derivation.
# Updating keyslot area [0x8000].
# Reusing open rw fd on device /dev/sdc1
# Device size 21473787904, offset 16777216.
# Device /dev/sdc1 WRITE lock already held.
# Trying to write LUKS2 header (16384 bytes) at offset 0.
# Reusing open rw fd on device /dev/sdc1
# Checksum:8013d462f87b880f37bad54fd5c7581666abfafe83e2f029072aec752f53b64e (in-memory)
# Trying to write LUKS2 header (16384 bytes) at offset 16384.
# Reusing open rw fd on device /dev/sdc1
# Checksum:18efc830069d27255c84baf9201324af76493e50ca38f21695776e112962feb1 (in-memory)
# Device /dev/sdc1 WRITE lock released.
Key slot 0 created.
# Releasing crypt device /dev/sdc1 context.
# Releasing device-mapper backend.
# Closing read only fd for /dev/sdc1.
# Closing read write fd for /dev/sdc1.
# Unlocking memory.
Command successful.

Before running the zkey-cryptsetup to set the verification pattern, cryptsetup luksDump /dev/sdc1.

[root@linux ~]# cryptsetup luksDump /dev/sdc1
LUKS header information
Version:        2
Epoch:          3
Metadata area:  16384 [bytes]
Keyslots area:  16744448 [bytes]
UUID:           a21ac728-1896-4380-87f0-0856a4522079
Label:          (no label)
Subsystem:      (no subsystem)
Flags:          (no flags)

Data segments:
  0: crypt
    offset: 16777216 [bytes]
    length: (whole device)
    cipher: paes-xts-plain64
    sector: 4096 [bytes]

Keyslots:
  0: luks2
    Key:        1024 bits
    Priority:   normal
    Cipher:     aes-xts-plain64
    Cipher key: 512 bits
    PBKDF:      pbkdf2
    Hash:       sha256
    Iterations: 1783292
    Salt:       1a 1e 95 97 4c 12 59 a7 c1 83 0e 13 b0 2a ae 48 
                b5 ca f7 48 bc 53 35 b4 73 5f d9 ba 6d 68 b0 48 
    AF stripes: 4000
    AF hash:    sha256
    Area offset:32768 [bytes]
    Area length:512000 [bytes]
    Digest ID:  0
Tokens:
Digests:
  0: pbkdf2
    Hash:       sha256
    Iterations: 55445
    Salt:       d0 e9 11 8c bf de a4 89 e6 0e 3c 1c 8b 91 fe 71 
                eb 09 d4 1d 1f 1c f6 e7 08 04 c1 3a c1 ee e9 3a 
    Digest:     80 19 e5 09 65 91 01 df f6 3b 47 70 ba e9 d1 8b 
                a8 c2 58 04 cb 95 8f b9 39 29 91 93 08 4f 72 33

Setting the verification pattern.

[root@linux ~]# zkey-cryptsetup setvp /dev/sdc1
Enter passphrase for '/dev/sdc1':

[root@linux ~]# cryptsetup luksDump /dev/sdc1
LUKS header information
Version:        2
Epoch:          4
Metadata area:  16384 [bytes]
Keyslots area:  16744448 [bytes]
UUID:           a21ac728-1896-4380-87f0-0856a4522079
Label:          (no label)
Subsystem:      (no subsystem)
Flags:          (no flags)

Data segments:
  0: crypt
    offset: 16777216 [bytes]
    length: (whole device)
    cipher: paes-xts-plain64
    sector: 4096 [bytes]

Keyslots:
  0: luks2
    Key:        1024 bits
    Priority:   normal
    Cipher:     aes-xts-plain64
    Cipher key: 512 bits
    PBKDF:      pbkdf2
    Hash:       sha256
    Iterations: 1783292
    Salt:       1a 1e 95 97 4c 12 59 a7 c1 83 0e 13 b0 2a ae 48 
                b5 ca f7 48 bc 53 35 b4 73 5f d9 ba 6d 68 b0 48 
    AF stripes: 4000
    AF hash:    sha256
    Area offset:32768 [bytes]
    Area length:512000 [bytes]
    Digest ID:  0
Tokens:
  0: paes-verification-pattern
Digests:
  0: pbkdf2
    Hash:       sha256
    Iterations: 55445
    Salt:       d0 e9 11 8c bf de a4 89 e6 0e 3c 1c 8b 91 fe 71 
                eb 09 d4 1d 1f 1c f6 e7 08 04 c1 3a c1 ee e9 3a 
    Digest:     80 19 e5 09 65 91 01 df f6 3b 47 70 ba e9 d1 8b 
                a8 c2 58 04 cb 95 8f b9 39 29 91 93 08 4f 72 33

Opening the luks Device on debug mode.

[root@linux ~]# cryptsetup luksOpen /dev/sdc1 enc1 --debug
# cryptsetup 2.4.3 processing "cryptsetup luksOpen /dev/sdc1 enc1 --debug"
# Running command open.
# Locking memory.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating context for crypt device /dev/sdc1.
# Trying to open and read device /dev/sdc1 with direct-io.
# Initialising device-mapper backend library.
# Trying to load any crypt type from device /dev/sdc1.
# Crypto backend (OpenSSL 3.0.1 14 Dec 2021 [default][legacy]) initialized in cryptsetup library version 2.4.3.
# Detected kernel Linux 5.14.0-162.6.1.el9_1.s390x s390x.
# Loading LUKS2 header (repair disabled).
# Acquiring read lock for device /dev/sdc1.
# Opening lock resource file /run/cryptsetup/L_8:33
# Verifying lock handle for /dev/sdc1.
# Device /dev/sdc1 READ lock taken.
# Trying to read primary LUKS2 header at offset 0x0.
# Opening locked device /dev/sdc1
# Verifying locked device handle (bdev)
# LUKS2 header version 2 of size 16384 bytes, checksum sha256.
# Checksum:20f6ca39fb3b8519720b9298ee52abb2e8c7eb531543b3c1a5984b244a9b3448 (on-disk)
# Checksum:20f6ca39fb3b8519720b9298ee52abb2e8c7eb531543b3c1a5984b244a9b3448 (in-memory)
# Trying to read secondary LUKS2 header at offset 0x4000.
# Reusing open ro fd on device /dev/sdc1
# LUKS2 header version 2 of size 16384 bytes, checksum sha256.
# Checksum:ef1254dfda71d83b0b3650b7bf3c04b39039073e4de5984685e2641dea07419e (on-disk)
# Checksum:ef1254dfda71d83b0b3650b7bf3c04b39039073e4de5984685e2641dea07419e (in-memory)
# Device size 21473787904, offset 16777216.
# Device /dev/sdc1 READ lock released.
# PBKDF argon2id, time_ms 2000 (iterations 0), max_memory_kb 1048576, parallel_threads 4.
# Activating volume enc1 using token (any type) -1.
# dm version   [ opencount flush ]   [16384] (*1)
# dm versions   [ opencount flush ]   [16384] (*1)
# Detected dm-ioctl version 4.46.0.
# Device-mapper backend running with UDEV support enabled.
# dm status enc1  [ opencount noflush ]   [16384] (*1)
# Token 0 unusable for segment 0 with desired keyslot priority 2.
# Token 0 unusable for segment 0 with desired keyslot priority 1.
No usable token is available.
# Interactive passphrase entry requested.
Enter passphrase for /dev/sdc1: 
# Activating volume enc1 [keyslot -1] using passphrase.
# dm versions   [ opencount flush ]   [16384] (*1)
# dm status enc1  [ opencount noflush ]   [16384] (*1)
# Keyslot 0 priority 1 != 2 (required), skipped.
# Trying to open LUKS2 keyslot 0.
# Running keyslot key derivation.
# Reading keyslot area [0x8000].
# Acquiring read lock for device /dev/sdc1.
# Opening lock resource file /run/cryptsetup/L_8:33
# Verifying lock handle for /dev/sdc1.
# Device /dev/sdc1 READ lock taken.
# Reusing open ro fd on device /dev/sdc1
# Device /dev/sdc1 READ lock released.
# Verifying key from keyslot 0, digest 0.
# dm target-version crypt  [ opencount flush ]   [16384] (*1)
# dm versions   [ opencount flush ]   [16384] (*1)
# Detected dm-crypt version 1.24.0.
# Loading key (128 bytes, type logon) in thread keyring.
# dm versions   [ opencount flush ]   [16384] (*1)
# dm status enc1  [ opencount noflush ]   [16384] (*1)
# Calculated device size is 41908224 sectors (RW), offset 32768.
# DM-UUID is CRYPT-LUKS2-a21ac7281896438087f00856a4522079-enc1
# Udev cookie 0xd4db24c (semid 0) created
# Udev cookie 0xd4db24c (semid 0) incremented to 1
# Udev cookie 0xd4db24c (semid 0) incremented to 2
# Udev cookie 0xd4db24c (semid 0) assigned to CREATE task(0) with flags DISABLE_LIBRARY_FALLBACK         (0x20)
# dm create enc1 CRYPT-LUKS2-a21ac7281896438087f00856a4522079-enc1 [ opencount flush ]   [16384] (*1)
# dm reload   (253:2) [ opencount flush securedata ]   [16384] (*1)
# dm resume enc1  [ opencount flush securedata ]   [16384] (*1)
# enc1: Stacking NODE_ADD (253,2) 0:6 0660 [trust_udev]
# enc1: Stacking NODE_READ_AHEAD 1024 (flags=1)
# Udev cookie 0xd4db24c (semid 0) decremented to 1
# Udev cookie 0xd4db24c (semid 0) waiting for zero
# Udev cookie 0xd4db24c (semid 0) destroyed
# enc1: Skipping NODE_ADD (253,2) 0:6 0660 [trust_udev]
# enc1: Processing NODE_READ_AHEAD 1024 (flags=1)
# enc1 (253:2): read ahead is 256
# enc1 (253:2): Setting read ahead to 1024
Key slot 0 unlocked.
# Releasing crypt device /dev/sdc1 context.
# Releasing device-mapper backend.
# Closing read only fd for /dev/sdc1.
# Unlocking memory.
Command successful.

Create filesystem

[root@linux ~]# mke2fs /dev/mapper/enc1
mke2fs 1.46.5 (30-Dec-2021)
Creating filesystem with 5238528 4k blocks and 1310720 inodes
Filesystem UUID: 01131d05-b8a7-49e8-bc28-f28909bc5598
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
    4096000

Allocating group tables: done                            
Writing inode tables: done                            
Writing superblocks and filesystem accounting information: done

Mount the volume

[root@linux ~]# mount /dev/mapper/enc1 /mnt
[root@m1317022 ~]# df -h
Filesystem           Size  Used Avail Use% Mounted on
devtmpfs             4.0M     0  4.0M   0% /dev
tmpfs                1.9G     0  1.9G   0% /dev/shm
tmpfs                747M  8.3M  739M   2% /run
/dev/mapper/mpatha1  120G   27G   94G  22% /
tmpfs                374M  4.0K  374M   1% /run/user/0
/dev/mapper/enc1      20G   24K   19G   1% /mnt

[root@linux ~]# blkid
/dev/mapper/mpatha1: UUID="d467b383-a0db-4bcd-a3c2-338ebce06a3a" TYPE="xfs" PARTUUID="8470d393-01"
/dev/sdb: PTUUID="8470d393" PTTYPE="dos"
/dev/mapper/enc1: UUID="01131d05-b8a7-49e8-bc28-f28909bc5598" TYPE="ext2"
/dev/mapper/mpatha: PTUUID="8470d393" PTTYPE="dos"
/dev/sdc1: UUID="a21ac728-1896-4380-87f0-0856a4522079" TYPE="crypto_LUKS" PARTUUID="3b9b01ee-01"
/dev/sda: PTUUID="8470d393" PTTYPE="dos"

Closing the luks Device.

[root@linux ~]# cryptsetup luksClose /dev/mapper/enc1 --debug
# cryptsetup 2.4.3 processing "cryptsetup luksClose /dev/mapper/enc1 --debug"
# Running command close.
# Locking memory.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating crypt device context by device /dev/mapper/enc1.
# Initialising device-mapper backend library.
# dm version   [ opencount flush ]   [16384] (*1)
# dm versions   [ opencount flush ]   [16384] (*1)
# Detected dm-ioctl version 4.46.0.
# Detected dm-crypt version 1.24.0.
# Device-mapper backend running with UDEV support enabled.
# dm status enc1  [ opencount noflush ]   [16384] (*1)
# Releasing device-mapper backend.
# Trying to open and read device /dev/sdc1 with direct-io.
# Allocating context for crypt device /dev/sdc1.
# Trying to open and read device /dev/sdc1 with direct-io.
# Initialising device-mapper backend library.
# dm versions   [ opencount flush ]   [16384] (*1)
# dm table enc1  [ opencount flush securedata ]   [16384] (*1)
# Trying to open and read device /dev/sdc1 with direct-io.
# dm versions   [ opencount flush ]   [16384] (*1)
# dm deps enc1  [ opencount flush ]   [16384] (*1)
# Crypto backend (OpenSSL 3.0.1 14 Dec 2021 [default][legacy]) initialized in cryptsetup library version 2.4.3.
# Detected kernel Linux 5.14.0-162.6.1.el9_1.s390x s390x.
# Reloading LUKS2 header (repair disabled).
# Acquiring read lock for device /dev/sdc1.
# Opening lock resource file /run/cryptsetup/L_8:33
# Verifying lock handle for /dev/sdc1.
# Device /dev/sdc1 READ lock taken.
# Trying to read primary LUKS2 header at offset 0x0.
# Opening locked device /dev/sdc1
# Verifying locked device handle (bdev)
# LUKS2 header version 2 of size 16384 bytes, checksum sha256.
# Checksum:20f6ca39fb3b8519720b9298ee52abb2e8c7eb531543b3c1a5984b244a9b3448 (on-disk)
# Checksum:20f6ca39fb3b8519720b9298ee52abb2e8c7eb531543b3c1a5984b244a9b3448 (in-memory)
# Trying to read secondary LUKS2 header at offset 0x4000.
# Reusing open ro fd on device /dev/sdc1
# LUKS2 header version 2 of size 16384 bytes, checksum sha256.
# Checksum:ef1254dfda71d83b0b3650b7bf3c04b39039073e4de5984685e2641dea07419e (on-disk)
# Checksum:ef1254dfda71d83b0b3650b7bf3c04b39039073e4de5984685e2641dea07419e (in-memory)
# Device size 21473787904, offset 16777216.
# Device /dev/sdc1 READ lock released.
# PBKDF argon2id, time_ms 2000 (iterations 0), max_memory_kb 1048576, parallel_threads 4.
# Deactivating volume /dev/mapper/enc1.
# dm versions   [ opencount flush ]   [16384] (*1)
# dm status enc1  [ opencount noflush ]   [16384] (*1)
# dm versions   [ opencount flush ]   [16384] (*1)
# dm table enc1  [ opencount flush securedata ]   [16384] (*1)
# Trying to open and read device /dev/sdc1 with direct-io.
# dm versions   [ opencount flush ]   [16384] (*1)
# dm deps enc1  [ opencount flush ]   [16384] (*1)
# dm versions   [ opencount flush ]   [16384] (*1)
# dm table enc1  [ opencount flush securedata ]   [16384] (*1)
# dm versions   [ opencount flush ]   [16384] (*1)
# Udev cookie 0xd4d2e4b (semid 1) created
# Udev cookie 0xd4d2e4b (semid 1) incremented to 1
# Udev cookie 0xd4d2e4b (semid 1) incremented to 2
# Udev cookie 0xd4d2e4b (semid 1) assigned to REMOVE task(2) with flags DISABLE_LIBRARY_FALLBACK         (0x20)
# dm remove enc1  [ opencount flush retryremove ]   [16384] (*1)
# Udev cookie 0xd4d2e4b (semid 1) decremented to 0
# Udev cookie 0xd4d2e4b (semid 1) waiting for zero
# Udev cookie 0xd4d2e4b (semid 1) destroyed
# Requesting keyring logon key for revoke and unlink.
# Releasing crypt device /dev/sdc1 context.
# Releasing device-mapper backend.
# Closing read only fd for /dev/sdc1.
# Unlocking memory.
Command successful.

To validate the device that is luks encrypted with crypto card.

[root@linux ~]# zkey-cryptsetup val /dev/sdc1
Enter passphrase for '/dev/sdc1': 
Validation of secure volume key of device '/dev/sdc1':
  Status:                Valid
  Secure key size:       128 bytes
  XTS type key:          Yes
  Key type:              CCA-AESDATA
  Clear key size:        512 bits
  Enciphered with:       CURRENT master key (MKVP: 71ecf46a35bd8924)
  Verification pattern:  b7e6c63a1b89fb31e77bf03bea518eff
                         313f2d13f997fc0f5b8146026051c3d1

============================================================================================ OR

============================================================================================

With Single command we can execute above procedure till verification pattern.

[root@linux ~]# zkey cryptsetup --volumes /dev/sdc1 --run -V
zkey: Keystore in directory '/etc/zkey/repository' opened successfully
zkey: Process_filtered: name_filter = '(null)', volume_filter = '/dev/sdc1', apqn_filter = '(null)'
zkey: File names for key 'secure_xtskeys': '/etc/zkey/repository/secure_xtskeys.skey' and '/etc/zkey/repository/secure_xtskeys.info'
zkey: Key 'secure_xtskeys' filtered out due to volumes filter
zkey: File names for key 'secure_xtskeys1': '/etc/zkey/repository/secure_xtskeys1.skey' and '/etc/zkey/repository/secure_xtskeys1.info'
zkey: 128 bytes read from file '/etc/zkey/repository/secure_xtskeys1.skey'
 00000000: 01000000 0400c0cc 71ecf46a 35bd8924
 00000010: b3975811 a6fb51f1 ee054540 4e76e99a
 00000020: 98301eca 5cfed7c7 3460fff4 ea82f3ec
 00000030: 00000000 00000000 01000020 58cd01c7
 00000040: 01000000 0400c097 71ecf46a 35bd8924
 00000050: c0cc8807 dbc8e664 bf33984c a9203a90
 00000060: 5fd13009 19d57bdf 087c0eba 6b07eec9
 00000070: 00000000 00000000 01000020 9fbf28f7
Executing: cryptsetup luksFormat -v --type luks2 --master-key-file '/etc/zkey/repository/secure_xtskeys1.skey' --key-size 1024 --cipher paes-xts-plain64 --pbkdf pbkdf2 --sector-size 4096 /dev/sdc1
WARNING: Device /dev/sdc1 already contains a 'crypto_LUKS' superblock signature.

WARNING!
========
This will overwrite data on /dev/sdc1 irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
Enter passphrase for /dev/sdc1: 
Verify passphrase: 
Existing 'crypto_LUKS' superblock signature on device /dev/sdc1 will be wiped.
Existing 'crypto_LUKS' superblock signature on device /dev/sdc1 will be wiped.
Key slot 0 created.
Command successful.
Executing: zkey-cryptsetup setvp /dev/sdc1 -V
zkey-cryptsetup: Device '/dev/pkey' has been opened successfully
zkey-cryptsetup: Installing SIGINT/SIGTERM handler
zkey-cryptsetup: Volume key size: 128
zkey-cryptsetup: Integrity: 'none'
zkey-cryptsetup: Integrity key size: 0
Enter passphrase for '/dev/sdc1': 
zkey-cryptsetup: Volume key obtained from key slot 0
zkey-cryptsetup: Build a list of APQNs for the key
zkey-cryptsetup: 1 APQNs found
zkey-cryptsetup:   APQN: 00.0011
zkey-cryptsetup: Secure key validation completed successfully
zkey-cryptsetup: Volume key is currently enciphered with CURRENT master key
zkey-cryptsetup: 'paes-verification-pattern' token not found
zkey-cryptsetup: Key verification pattern:  58729dfd2ece6f16e6e286f425fc6abb87d40004be2518727f1cf849c6f643c1
zkey-cryptsetup: Verification-pattern: 58729dfd2ece6f16e6e286f425fc6abb87d40004be2518727f1cf849c6f643c1
zkey-cryptsetup: Verification-pattern token put to token slot 0
zkey: Process_filtered rc = 0
zkey: Successfully generated cryptsetup commands

Cryptsetup dump after the luksformat.

[root@linux ~]# cryptsetup luksDump /dev/sdc1
LUKS header information
Version:        2
Epoch:          4
Metadata area:  16384 [bytes]
Keyslots area:  16744448 [bytes]
UUID:           ac6b0ab8-a478-4f41-b56a-be47c54bdd49
Label:          (no label)
Subsystem:      (no subsystem)
Flags:          (no flags)

Data segments:
  0: crypt
    offset: 16777216 [bytes]
    length: (whole device)
    cipher: paes-xts-plain64
    sector: 4096 [bytes]

Keyslots:
  0: luks2
    Key:        1024 bits
    Priority:   normal
    Cipher:     aes-xts-plain64
    Cipher key: 512 bits
    PBKDF:      pbkdf2
    Hash:       sha256
    Iterations: 1789378
    Salt:       6c 71 52 2c bb ea f0 8a 6c dc b9 7e 6c 34 55 de 
                03 ae ad 9a a8 c4 6a 41 8e 2f 13 10 23 cb 92 69 
    AF stripes: 4000
    AF hash:    sha256
    Area offset:32768 [bytes]
    Area length:512000 [bytes]
    Digest ID:  0
Tokens:
  0: paes-verification-pattern
Digests:
  0: pbkdf2
    Hash:       sha256
    Iterations: 53542
    Salt:       10 da 5f 5e 70 45 09 b9 2f ce 99 0e fa 00 64 4a 
                d5 ba d2 38 56 7e cd 17 76 e7 9a 1a fd cd 1a 46 
    Digest:     1f ca 60 9a 31 60 c9 b5 42 7e 33 63 de eb 36 9a 
                a6 c4 c1 4d 38 eb 66 95 fe 3e 62 d6 1a cd 70 7f

Validation

[root@m1317022 ~]# zkey-cryptsetup val /dev/sdc1
Enter passphrase for '/dev/sdc1': 
Validation of secure volume key of device '/dev/sdc1':
  Status:                Valid
  Secure key size:       128 bytes
  XTS type key:          Yes
  Key type:              CCA-AESDATA
  Clear key size:        512 bits
  Enciphered with:       CURRENT master key (MKVP: 71ecf46a35bd8924)
  Verification pattern:  58729dfd2ece6f16e6e286f425fc6abb
                         87d40004be2518727f1cf849c6f643c1

Hi @jlebon, I've given the details above for the data volume encryption process with IBM CEX card.

Could you direct me how can i achieve to test this in FCOS qcow image with crytpsetup luksFormat --master-key ? Is it something i need to make changes in the ignition to accept the args as master key?

Summary of the discussion:

LUKS is encrypted with a protected key (cipher shows up in dump as paes-xts-plain64).
Kernel knows to reach out to CEX card to read it.
Passphrase/key file is required but doesn't need to be kept securely since the master key is stored in the CEX card. So we can use a dummy/hardcoded one.
Example butane config:

luks:
  name: root,
  label: luks-root
  device: /dev/disk/by-partlabel/root
  cex: true

(Translates in Ignition to e.g. .storage.luks[].cex: true.)

Use an hardcoded key in the initramfs/rootfs as keyfile.
Add logic to Ignition to format the LUKS device using the zkey tools / cryptsetup commands, using an hardcoded key.
Enhance rdcore rootmap to add rd.luks.key with a path to the keyfile with the hardcoded key in the initrd

Hi,

I've two questions,

Should the new feature to be added in the base_experiemental or fcos_experimental? If it is in base_experimental does the translation and validation to be done at ignition level? Because the luks.* parameters are getting validated at ignition luks.go. Any examples would be helpful.
To generate the masterkey and running the cryptsetup, user can pass options like key-type (default is CCAAES-DATA), but if requires they can pass CCAES-CIPHER) etc. can this be added from the butane side?

So Something like.

luks:
  name: root,
  label: luks-root
  device: /dev/disk/by-partlabel/root
  cexmode:
      cex:true
      options:                                    #optional
           key_type: ccaaes-cipher.  #to generate the keydefault to ccaes-data 
           key_size: 1024                    #requires by cryptsetup 
           sector_size: 512                 #default to 4096
          cipher: paes-xts-plain64.

type CexMode struct {
    Cex     *bool       `yaml:"cex"`
    Options     CexCustom   `yaml:"custom"`
}

type CexCustom struct {
    Key_Size        *string     `yaml:"key_size"`
    Key_Type        *string     `yaml:"key_type"`
    Cipher          *string     `yaml:"cipher"`
    Sector_Size     *int        `yaml:"sector_size"`
}

Hi,

I've two questions,

Should the new feature to be added in the base_experiemental or fcos_experimental? If it is in base_experimental does the translation and validation to be done at ignition level? Because the luks.* parameters are getting validated at ignition luks.go. Any examples would be helpful.

For this issue, I would start off implementing the Ignition side of this. We can look at adding Butane sugar afterwards (but to answer your question, yes I think this would belong in the base spec which models the Ignition spec closely, but then we'd probably also want to support it in the boot_device sugar too, which is specific to the fcos and openshift variants).

To generate the masterkey and running the cryptsetup, user can pass options like key-type (default is CCAAES-DATA), but if requires they can pass CCAES-CIPHER) etc. can this be added from the butane side?

So Something like.

luks:
  name: root,
  label: luks-root
  device: /dev/disk/by-partlabel/root
  cexmode:
      cex:true
      options:                                    #optional
           key_type: ccaaes-cipher.  #to generate the keydefault to ccaes-data 
           key_size: 1024                    #requires by cryptsetup 
           sector_size: 512                 #default to 4096
          cipher: paes-xts-plain64.

type CexMode struct {
  Cex     *bool       `yaml:"cex"`
  Options     CexCustom   `yaml:"custom"`
}

type CexCustom struct {
  Key_Size        *string     `yaml:"key_size"`
  Key_Type        *string     `yaml:"key_type"`
  Cipher          *string     `yaml:"cipher"`
  Sector_Size     *int        `yaml:"sector_size"`
}

This would need to be done in Ignition first, but yes we could mirror it as needed in the base Butane spec when we get there. That said, I'm not even sure we need to expose these to start since options already provides a way to customize the luksFormat arguments (or do some of these fields affect the zdev setup step too?). The sector size and cipher especially should have good enough defaults and I don't think warrant dedicated keys to make it easier than via options.

Do you have more information about CCAES-CIPHER vs CCAAES-DATA?

Thanks @travier ,

Here is the snippet from IBM regards to CCA-AESDATA, CCA-AESCIPHER or EP11 is another crypto controller where both comes in same CEX card( It may use in future).

Using the --key-type parameter of the zkey command, you can decide between various types of secure keys: a CCA AES DATA key, a CCA AES CIPHER key, or an EP11 AES key for use with pervasive volume encryption. An AES DATA secure key is generated as the default, if you omit this parameter.

With an AES CIPHER secure key, certain attributes are cryptographically bound to the key. These attributes may limit the usage of the key, for example, restrict the export or the usability scope. So this key type is assumed to be even more secure than the default AES DATA key. To generate an AES CIPHER secure key, a CEX6C or later coprocessor is required.

In Bytes

CCA AES DATA: 64, 128 for XTS
CCA AES CIPHER: 136, 272 for XTS
EP11 AES: 320, 640 for XTS

--key-type CCA-AESCIPHER can be passed as a argument during the master key generation

eg:

#  zkey generate 
                  --name secure_xtskey1 
                 --keybits 256 
                 --xts
                 --volumes /dev/mapper/disk1:enc-disk1 
                 --key-type CCA-AESCIPHER 
                 --volume-type LUKS2 
                --sector-size 4096 
                --apqns 03.0039,04.0039

Hi, a) I have done the frontend config part and able to generate the schema and docs. b) Have added in the distro.go for zkey tools command required for cex luks encryption. c) Done validation for cexmode that not to use with clevis and options specific to use only --key-type CCA-AESCIPHER || CCA-AESDATA.

I am bit stuck of following.

How to achieve to hardcoded the passphrase in initramfs? is it through coreos-installer with ignition? i tried to add in the files section in the ignition, looks like it is only configuring in the installed root not in the initramfs.

For the luks encryption: Is it ok to add separate function under ignition.config.internal.exec.stages.disks.luks.go -> createLuks() specific to cex for using zkey ? because i am trying to achieve the following.

  a) Verify the Cex Card available and obtain the domain number for key generation. `lszcrypt` command to obtain the domain number.

 b)  Generate the Secure keys.
   `# zkey generate --name secure_rootCipherkey --xts --volumes <devAlias>:<device-mapper> -K <keytype> --apqns <card domain number>`
          This command generate `Secure keys` and `Secure keys.info` (details about the key)

 c)  `zkey cryptsetup --volume <devAlias>:device-mapper --run --key-file <keyfile> -q`

The above command automatically run the cryptsetup luksFormat and zkey setvp command from the securekey.info file.

How to achieve to hardcoded the passphrase in initramfs? is it through coreos-installer with ignition? i tried to add in the files section in the ignition, looks like it is only configuring in the installed root not in the initramfs.

We can create it in module-setup.sh and install it in e.g. /etc/luks/cex.key.

For the luks encryption: Is it ok to add separate function under ignition.config.internal.exec.stages.disks.luks.go -> createLuks() specific to cex for using zkey ? because i am trying to achieve the following.
      a) Verify the Cex Card available and obtain the domain number for key generation. `lszcrypt` command to obtain the domain number.

     b)  Generate the Secure keys.
       `# zkey generate --name secure_rootCipherkey --xts --volumes <devAlias>:<device-mapper> -K <keytype> --apqns <card domain number>`
              This command generate `Secure keys` and `Secure keys.info` (details about the key)

     c)  `zkey cryptsetup --volume <devAlias>:device-mapper --run --key-file <keyfile> -q`  
The above command automatically run the cryptsetup luksFormat and zkey setvp command from the securekey.info file.

I think for this level of detail, it would be better to open a PR with what you came up with and we can discuss there.

Hi,

I've completed the required changes in ignition for CEX card configuration. I am getting panic error while running the unit test ./test.

what i have done is updated config/doc/ignition.yaml for cex config. Then updated config/v3_5_experimental/schema/ignition.json. First i done the ./generate. which is successfully create config/v3_5_experimental/types/schema.go. Then done the required changes in backend. then finally run the ./test.

The following is the translate ignition_test.txt from types.config to types.config

Any hint where to troubleshoot or any per-requisite i am missing?

It'd be easier to debug this in a PR.

Hi @jlebon , the types.config test failed Looks like i did not add the translate.go for the Cexmode struct under the v3_5_experiemental. However i added but i am bit lost on how it will convert the old_types to types if the old types version is v3_4. Because v3_4 does not have struct details of CEX.

Hi, I am able to test CEX with luks encryption in RHCOS and FCOS qcow2 image. I noticed that the securekey files created in disk stages are not getting saved on the real root filesystem, i can see only the empty directory /etc/zkey/repository.

Is it something we need to create in files stages?

RHCOS test.

  OK  ] Finished Ignition OSTree: Save Partitions.
         Starting Ignition (disks)...
[   22.470124] systemd[1]: Starting Ignition (disks)...
[   22.491771] ignition[870]: Ignition 98be9e17-dirty
[   22.491875] ignition[870]: Stage: disks
[   22.492195] ignition[870]: reading system config file "/usr/lib/ignition/base.d/00-core.ign"
[   22.494234] ignition[870]: no config dir at "/usr/lib/ignition/base.platform.d/qemu"
[   22.503276] ignition[870]: disks: createPartitions: op(1): [started]  waiting for devices [/dev/vda]
[   22.508311] ignition[870]: disks: createPartitions: op(1): [finished] waiting for devices [/dev/vda]
[   22.508414] ignition[870]: disks: createPartitions: created device alias for "/dev/vda": "/run/ignition/dev_aliases/dev/vda" -> "/dev/vda"
[   22.508450] ignition[870]: disks: createPartitions: op(2): [started]  partitioning "/run/ignition/dev_aliases/dev/vda"
[   22.508480] ignition[870]: disks: createPartitions: op(2): op(3): [started]  reading partition table of "/run/ignition/dev_aliases/dev/vda"
[   22.520271] ignition[870]: disks: createPartitions: op(2): op(3): [finished] reading partition table of "/run/ignition/dev_aliases/dev/vda"
[   22.520388] ignition[870]: disks: createPartitions: op(2): running sgdisk with options: [--pretend --delete=4 --new=4:788480:+0 --info=4 /run/ignition/dev_aliases/dev/vda]
[   22.540194]  vda: vda3 vda4
[   22.559722] ignition[870]: disks: createPartitions: op(2): resizing partition 4
[   22.564104] ignition[870]: disks: createPartitions: op(2): running sgdisk with options: [--delete=4 --new=4:788480:+32765919 --change-name=4:root --typecode=4:0FC63DAF-8483-4772-8E79-3D69D8477DE4 --partition-guid=4:245A0423-912D-4A88-ACB5-4B9EE39F73A1 /run/ignition/dev_aliases/dev/vda]
[   22.564277] ignition[870]: disks: createPartitions: op(2): op(4): [started]  deleting 1 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/vda"
[   23.852773]  vda: vda3 vda4
[   23.854229] ignition[870]: disks: createPartitions: op(2): op(4): [finished] deleting 1 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/vda"
[   23.854306] ignition[870]: disks: createPartitions: op(2): op(5): [started]  waiting for triggered uevent
[   23.986049] ignition[870]: disks: createPartitions: op(2): op(5): [finished] waiting for triggered uevent
[   23.986122] ignition[870]: disks: createPartitions: op(2): [finished] partitioning "/run/ignition/dev_aliases/dev/vda"
[   23.986147] ignition[870]: disks: createLuks: op(6): [started]  waiting for devices [/dev/vda4]
[   23.991246] ignition[870]: disks: createLuks: op(6): [finished] waiting for devices [/dev/vda4]
[   23.991282] ignition[870]: disks: createLuks: created device alias for "/dev/vda4": "/run/ignition/dev_aliases/dev/vda4" -> "/dev/vda4"
[   23.991445] ignition[870]: disks: createLuks: op(7): [started]  wiping filesystem signatures from "/run/ignition/dev_aliases/dev/vda4"
[   23.998275] ignition[870]: disks: createLuks: op(7): [finished] wiping filesystem signatures from "/run/ignition/dev_aliases/dev/vda4"
[   23.998422] ignition[870]: disks: createLuks: op(8): [started]  generating cex secure keys
[   24.027452] ignition[870]: disks: createLuks: op(8): [finished] generating cex secure keys
[   24.027528] ignition[870]: disks: createLuks: op(9): [started]  creating "root"
[  *** ] A start job is running for Ignition (disks) (34s / no limit)
[   37.367630] ignition[870]: disks: createLuks: op(9): [finished] creating "root"
[   ***] A start job is running for Ignition (disks) (35s / no limit)
[   39.318719] ignition[870]: disks: createLuks: op(a): [finished] Setting verification pattern for device: "/run/ignition/dev_aliases/dev/vda4"
[    **] A start job is running for Ignition (disks) (39s / no limit)
[   41.682931] ignition[870]: disks: createLuks: op(b): [finished] opening luks device root
[   41.683118] ignition[870]: disks: createLuks: op(c): [started]  waiting for triggered uevent
[   41.749009] ignition[870]: disks: createLuks: op(c): [finished] waiting for triggered uevent
[   41.751767] ignition[870]: disks: createFilesystems: op(d): [started]  waiting for devices [/dev/mapper/root]
[   41.769531] ignition[870]: disks: createFilesystems: op(d): [finished] waiting for devices [/dev/mapper/root]
[   41.770051] ignition[870]: disks: createFilesystems: created device alias for "/dev/mapper/root": "/run/ignition/dev_aliases/dev/mapper/root" -> "/dev/dm-0"
[   41.770649] ignition[870]: disks: createFilesystems: op(e): [started]  determining filesystem type of "/dev/mapper/root"
[   41.902954] ignition[870]: disks: createFilesystems: op(e): [finished] determining filesystem type of "/dev/mapper/root"
[   41.903063] ignition[870]: disks: createFilesystems: found  filesystem at "/dev/mapper/root" with uuid "" and label ""
[   41.903147] ignition[870]: disks: createFilesystems: op(f): [started]  wiping filesystem signatures from "/run/ignition/dev_aliases/dev/mapper/root"
[   41.992972] ignition[870]: disks: createFilesystems: op(f): [finished] wiping filesystem signatures from "/run/ignition/dev_aliases/dev/mapper/root"
[   41.993118] ignition[870]: disks: createFilesystems: op(10): [started]  creating "xfs" filesystem on "/run/ignition/dev_aliases/dev/mapper/root"
[   42.140405] ignition[870]: disks: createFilesystems: op(10): [finished] creating "xfs" filesystem on "/run/ignition/dev_aliases/dev/mapper/root"
[  OK  ] Finished Ignition (disks).
[   42.362376] ignition[870]: disks: createFilesystems: op(11): [finished] waiting for triggered uevent
[   42.362456] ignition[870]: disks: disks passed
[   42.362477] ignition[870]: Ignition finished successfully
[   42.368278] systemd[1]: Finished Ignition (disks).
[  OK  ] Reached target Initrd Root Device.

I guess for the root disk we don't really need to save it, but for data volumes we do and it'd be awkward to try to special-case the root case.

Is it something we need to create in files stages?

Yes. Search for s.State.LuksPersistKeyFiles in the code.

Hi @jlebon , i understood. if needed we can export the --volume-key in the real root. so i'll leave it in that case. Even i tried --dump-key-file the volume key and verified the secure-key validity and the key has all the attributes.

Sorry, I think I got confused earlier.

Once the LUKS device is created with the secure volume key, is there any need to have the key in /etc/zkey/ anymore? It doesn't actually need to be persisted right?

Oops, I lately see this message. i did not include any method added to save the secure key. Because the important key is the master key resides in the crypto controller which is tamper resistant opaque to operating system. The securekey get unwrapped by the master key and that effective key get transported to secure hardware channel to CPACF to generate the protected key (the effective key again get wrapped) which is called protected key which is specific to VM. with that protected key the blocks of data get decrypted inside the CPACF. So there is noway these keys are exposed to operating system.

In production mode the crypto controller master key gets managed by TKE (key management workstation) using different smart cards.

Note: CCA-AESDATA can be exported to the zkey repository, but in case of CCA-AESCIPHER is not possible, only we can dump the volume key from the Luks Header.

I will check with IBM crypto team whether the backup of secure key in the repository has any use for recovery at any stage for root volume.

Sorry, I think I got confused earlier.

Once the LUKS device is created with the secure volume key, is there any need to have the key in /etc/zkey/ anymore? It doesn't actually need to be persisted right?

Here is the update from IBM crypto team, Technically the key in the luks header is good enough. But we think that having the key in the zkey repository as well is a good idea for backup and key management reasons. The zkey repository has information about which key is used for which volume (if you associated the key to a volume properly). So its also kind of a documentation which key is used where. If you ever forget to reencipher the key in the luks header after a master key change can still apply the key from the repository to the luks header afterwards. A key reencipherment of the keys in the key repository is just one single command, and zkey knows which keys to reencipher after a MK change of a set of APQNs. So its much easier to handle it that way as to manually know which luks keys to reencipher.... you can even reencipher the volume key directly with "zkey-cryptsetup reencipher"

OK, that's fine. We can persist it into the real root. In that case yes, you'll want to use the s.State facility to do this.

coreos / ignition

Support encrypting LUKS devices using CEX secure keys on s390x #1693

Feature Request

Environment

Desired Feature

Other Information