Investigate graceful termination of DriverContainers

[dagrayvid]

When creating a recipe for the lustre-client kernel module based off of the simple-kmod recipe I found the following issue. The method used in several recipes for the entrypoint.sh did not consistently unmount the kernel module when the SpecialResource CR was deleted.

#!/bin/bash -x
    unmount() {
        modprobe -r lustre
        ...
    }
    ...
    modprobe -v lustre
    trap "echo 'Caught signal'; exit 1" HUP INT QUIT PIPE TERM
    trap "unmount" EXIT
    sleep infinity

How can we improve graceful termination of kernel modules / driver containers?

[dagrayvid]

@zvonkok I believe I found a good solution to this. The problem with the above, which I got from the mofed driver container entrypoint.sh, is that on pod deletion, SIGTERM is sent to PID 1 in the container. However because sleep infinity is running, the entrypoint script won't receive SIGTERM until sleep is stopped (never).

Please take a look at the working solution below, and if you agree with it I will add it to the lustre-client recipe.

  entrypoint.sh: |-
    #!/bin/bash -x

    unmount() {
        modprobe -r lustre
        ...
        # Any cleanup code
    }

    modprobe -v lnet
    ...
    # Any startup code

    sleep infinity & PID=$!
    trap "kill $PID; unmount" HUP INT QUIT PIPE TERM EXIT
    wait

With this, when the DriverContainer pod is deleted for any reason (deleting the SR), the kernel modules can be unmounted cleanly. As previously mentioned, another option is using a preStop hook for the cleanup code, but handling SIGTERM is the "standard" k8s way.

This is important for e.g. the lustre-client recipe where left over kernel modules can cause a problem when mounting a lustre FS, that I have only been able to fix by unloading and reloading the kernel modules. I assume similar problems may exist for other kernel modules, or with the cleanup code for the mofed driver.