Can't define/use a bootstrap-osd keyring?

So initially when I created this latest cluster, I only had the client.admin keyring defined:

cephadm_keys:
  - name: client.admin
    caps:
      mon: "allow *"
      osd: "allow *"
      mgr: "allow *"
  - name: client.bootstrap-osd
    caps:
      mon: "profile bootstrap-osd"
      mgr: "allow r"

Now I've gone and defined that other keyring and re-run the cephadm_key role, that should enable OSD hosts to setup/replace OSDs. (I believe.)

But for some reason my cephadm containers on non-monitor hosts can't access this key yet:

mcollins1@storage-14-09022:~$ sudo cephadm shell -- ceph-volume lvm batch /dev/sdbo --db-devices /dev/ceph-f91899ea-2922-430e-abd3-41201822d059/osd-db-ca821d97-fbb0-4004-955b-b488e37341fd --osd-id 404
...

  Type            Path                                                    LV Size         % of device
----------------------------------------------------------------------------------------------------
  OSD id          404
  data            /dev/sdbo                                               20.01 TB        100.00%
  block_db        ceph-f91899ea-2922-430e-abd3-41201822d059/osd-db-ca821d97-fbb0-4004-955b-b488e37341fd 99.36 GB        10000.00%
--> do you want to proceed? (yes/no) yes
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd tree -f json
 stderr: 2024-09-26T04:54:39.103+0000 7f7013cab640 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory
 stderr: 2024-09-26T04:54:39.103+0000 7f7013cab640 -1 AuthRegistry(0x7f700c064348) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
 stderr: 2024-09-26T04:54:39.103+0000 7f7013cab640 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory
 stderr: 2024-09-26T04:54:39.103+0000 7f7013cab640 -1 AuthRegistry(0x7f700c067c30) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
 stderr: 2024-09-26T04:54:39.103+0000 7f7013cab640 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory
 stderr: 2024-09-26T04:54:39.103+0000 7f7013cab640 -1 AuthRegistry(0x7f7013caa0c0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
 stderr: 2024-09-26T04:54:39.103+0000 7f7011246640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
 stderr: 2024-09-26T04:54:39.103+0000 7f7011a47640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
 stderr: 2024-09-26T04:54:39.103+0000 7f7012248640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
 stderr: 2024-09-26T04:54:39.103+0000 7f7013cab640 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication
 stderr: [errno 13] RADOS permission denied (error connecting to the cluster)
-->  RuntimeError: Unable check if OSD id exists: 404

I feel there's something obvious I'm not getting here. 😅 Does anyone know how to get a bootstrap-osd keyring working with this collection?

stackhpc / ansible-collection-cephadm

Can't define/use a bootstrap-osd keyring? #154