Creation of encrypted zpool during CN setup broken

[teutat3s]

Reproduce (on a Triton headnode with a new computenode, YubiKey inserted): PI: 20220118T183559Z

sdc-server setup -s $UUID encryption_enabled=true

Logs:

...
[2022-03-23T11:18:22Z] ./joysetup.sh:969: main(): create_zpool zones /tmp/pool.json
[2022-03-23T11:18:22Z] ./joysetup.sh:458: create_zpool(): SYS_ZPOOL=zones
[2022-03-23T11:18:22Z] ./joysetup.sh:459: create_zpool(): POOL_JSON=/tmp/pool.json
[2022-03-23T11:18:22Z] ./joysetup.sh:460: create_zpool(): local e_flag=
[2022-03-23T11:18:22Z] ./joysetup.sh:461: create_zpool(): local bootable=no
[2022-03-23T11:18:22Z] ./joysetup.sh:463: create_zpool(): [[ -n '' ]]
[2022-03-23T11:18:22Z] ./joysetup.sh:472: create_zpool(): [[ -n 1 ]]
[2022-03-23T11:18:22Z] ./joysetup.sh:472: create_zpool(): e_flag='-e '
[2022-03-23T11:18:22Z] ./joysetup.sh:474: create_zpool(): /usr/sbin/zpool list -H -o name zones
cannot open 'zones': no such pool
[2022-03-23T11:18:22Z] ./joysetup.sh:475: create_zpool(): printf %-56s 'creating pool: zones'
[2022-03-23T11:18:22Z] ./joysetup.sh:480: create_zpool(): mkzpool -B -f -e zones /tmp/pool.json
fatal error: pool creation failed: create-zpool: illegal option -- B
Unknown option -B
Usage: kbmadm create-zpool [-g guid] [-t template] -- <zpool create args>...
       kbmadm recover [-c cfgnum] dataset
       kbmadm unlock [-r] dataset...
       kbmadm recovery add [-a] [-t template] [-r recovery_token] dataset
       kbmadm recovery list [-p] dataset
       kbmadm recovery activate dataset
       kbmadm recovery cancel dataset

[2022-03-23T11:18:23Z] ./joysetup.sh:483: create_zpool(): mkzpool -f -e zones /tmp/pool.json
fatal error: pool creation failed: create-zpool: illegal option -- f
Unknown option -f
Usage: kbmadm create-zpool [-g guid] [-t template] -- <zpool create args>...
       kbmadm recover [-c cfgnum] dataset
       kbmadm unlock [-r] dataset...
       kbmadm recovery add [-a] [-t template] [-r recovery_token] dataset
       kbmadm recovery list [-p] dataset
       kbmadm recovery activate dataset
       kbmadm recovery cancel dataset

[2022-03-23T11:18:23Z] ./joysetup.sh:484: create_zpool(): printf '%6s\n' failed
[2022-03-23T11:18:23Z] ./joysetup.sh:485: create_zpool(): fatal 'failed to create pool'
[2022-03-23T11:18:23Z] ./joysetup.sh:172: fatal(): echo 'Error: failed to create pool'
[2022-03-23T11:18:23Z] ./joysetup.sh:173: fatal(): exit 1

[teutat3s]

Testing notes:

• on the computenode to be encrypted:

  
  cp /usr/node/node_modules/zfs.js /var/tmp/zfs.js
  chmod u+w /var/tmp/zfs.js
  # make change in line 176
  vim /var/tmp/zfs.js
  mount -O -F lofs /var/tmp/zfs.js /usr/node/node_modules/zfs.js

on the headnode

sdc-server setup -s $UUID encryption_enabled=true

back on the computenode

tail -f /tmp/joysetup.*

... [2022-03-24T16:03:23Z] ./joysetup.sh:969: main(): create_zpool zones /tmp/pool.json [2022-03-24T16:03:23Z] ./joysetup.sh:458: create_zpool(): SYS_ZPOOL=zones [2022-03-24T16:03:23Z] ./joysetup.sh:459: create_zpool(): POOL_JSON=/tmp/pool.json [2022-03-24T16:03:23Z] ./joysetup.sh:460: create_zpool(): local e_flag= [2022-03-24T16:03:23Z] ./joysetup.sh:461: create_zpool(): local bootable=no [2022-03-24T16:03:23Z] ./joysetup.sh:463: create_zpool(): [[ -n '' ]] [2022-03-24T16:03:23Z] ./joysetup.sh:472: create_zpool(): [[ -n 1 ]] [2022-03-24T16:03:23Z] ./joysetup.sh:472: create_zpool(): e_flag='-e ' [2022-03-24T16:03:23Z] ./joysetup.sh:474: create_zpool(): /usr/sbin/zpool list -H -o name zones cannot open 'zones': no such pool [2022-03-24T16:03:23Z] ./joysetup.sh:475: create_zpool(): printf %-56s 'creating pool: zones' [2022-03-24T16:03:23Z] ./joysetup.sh:480: create_zpool(): mkzpool -B -f -e zones /tmp/pool.json [2022-03-24T16:03:37Z] ./joysetup.sh:481: create_zpool(): printf '\n%-56s (as potentially bootable)' '' [2022-03-24T16:03:37Z] ./joysetup.sh:482: create_zpool(): bootable=yes [2022-03-24T16:03:37Z] ./joysetup.sh:497: create_zpool(): get_bootparams [2022-03-24T16:03:37Z] ./joysetup.sh:497: create_zpool(): [2022-03-24T16:03:37Z] ./joysetup.sh:21: get_bootparams(): grep '^triton_installer' case $OSTYPE in [2022-03-24T16:03:37Z] ./joysetup.sh:23: get_bootparams(): bootparams [2022-03-24T16:03:37Z] ./joysetup.sh:576: create_zpool(): zfs set atime=off zones [2022-03-24T16:03:37Z] ./joysetup.sh:581: create_zpool(): printf '%4s\n' done [2022-03-24T16:03:37Z] ./joysetup.sh:583: create_zpool(): [[ SunOS == \S\u\n\O\S ]] [2022-03-24T16:03:37Z] ./joysetup.sh:584: create_zpool(): svccfg -s svc:/system/smartdc/init setprop config/zpool=zones [2022-03-24T16:03:37Z] ./joysetup.sh:585: create_zpool(): svccfg -s svc:/system/smartdc/init:default refresh [2022-03-24T16:03:37Z] ./joysetup.sh:588: create_zpool(): export CONFDS=zones/config [2022-03-24T16:03:37Z] ./joysetup.sh:588: create_zpool(): CONFDS=zones/config [2022-03-24T16:03:37Z] ./joysetup.sh:589: create_zpool(): export COREDS=zones/cores [2022-03-24T16:03:37Z] ./joysetup.sh:589: create_zpool(): COREDS=zones/cores [2022-03-24T16:03:37Z] ./joysetup.sh:590: create_zpool(): export OPTDS=zones/opt [2022-03-24T16:03:37Z] ./joysetup.sh:590: create_zpool(): OPTDS=zones/opt [2022-03-24T16:03:37Z] ./joysetup.sh:591: create_zpool(): export VARDS=zones/var [2022-03-24T16:03:37Z] ./joysetup.sh:591: create_zpool(): VARDS=zones/var [2022-03-24T16:03:37Z] ./joysetup.sh:592: create_zpool(): export SWAPVOL=zones/swap [2022-03-24T16:03:37Z] ./joysetup.sh:592: create_zpool(): SWAPVOL=zones/swap [2022-03-24T16:03:37Z] ./joysetup.sh:598: create_zpool(): touch /zones/.system_pool ...

[danmcd]

The bug of not feeding "--" was always there since encrypted-CN-pools happened. The tickling of the bug was introduced in two parts:

The -B (bootable) path of mkzpool(1M->8) with -f came in with TRITON-2188, and the comment above it explains why. That came in with Triton release 20210128. After that it would fail, but then the non-bootable attempt would get tried.

The non-bootable path was introduced with TRITON-2233, solving a similar problem specific to Linux CNs (but would be a NOP for SmartOS ones). That came in with Triton release 20210715, and it's where @teutat3s 's full bug first appeared.