search

NearNodeFlash / NearNodeFlash.github.io

View this document https://nearnodeflash.github.io/
Apache License 2.0
3 stars 3 forks source link

Lustre FS name is being recycled #147

Open roehrich-hpe opened 3 months ago

roehrich-hpe commented 3 months ago

We've seen a case where the Lustre FS name is being recycled.

Code in createNnfStorage() is using the first 7 chars from the Servers resource UID to create the unique FS name: https://github.com/NearNodeFlash/nnf-sos/blob/master/internal/controller/nnf_workflow_controller_helpers.go#L638

Unfortunately, after creating 1758 filesystems k8s created a Servers resource with a UID that shares those same first 7 chars from the earlier Servers resource UID and we collided with an old name.

The Lustre MGS does not forget a name, and can't be told to forget a name. So after we teardown an ephemeral Lustre filesystem its name will live forever in the MGS. There may be a mount option that tells the MGS to overwrite existing records that use the name, but we'd have to be certain the previous filesystem is really gone.

behlendorf commented 3 weeks ago

The Lustre MGS does not forget a name, and can't be told to forget a name.

It turns out this can be done. lctl erase_lcfg {fsname} deletes all the config llogs for the named file system (see LCTL-LCFG(8)). It must be run on the node where the MGS is running. There are no sanity checks to prevent one from deleting llogs while someone is still using the file system.