[Bug] Layer lock with tri layers

[EdenEast]

Describe the bug

The layer lock feature does not seem to handle layers that have been activated by update_tri_layer_state function. When one of the layers that are used to activate the tri layer is released the layer is moved back to the layer that is still being held. This makes it not possible to use a layer lock key to keep the layer on the tri activated layer.

Information

Here is the userspace code that I use to set the tri layers:

/**
 * @brief Tri state handler
 *
 * This handles activating a layer by holding down two other layers.
 *
 * @param state Curret layer state that will be updated
 */
layer_state_t layer_state_set_user(layer_state_t state) {
  return update_tri_layer_state(state, _RAISE, _LOWER, _ADJ);
}

[getreuer]

@EdenEast thank you for reporting this. That's an interesting edge case.

Here is a method to lock the tri layer. In keymap.c, change layer_state_set_user() as

layer_state_t layer_state_set_user(layer_state_t state) {
  if (!is_layer_locked(_ADJ)) {
    state = update_tri_layer_state(state, _RAISE, _LOWER, _ADJ);
  }
  return state;
}

With tri layer "layer RAISE + layer LOWER = layer ADJ," what's happening is:

1. Layer switches are held to activate layers RAISE and LOWER.
2. update_tri_layer_state activates the tri layer ADJ.
3. Layer Lock is pressed while layer ADJ is active. The behavior of the Layer Lock key is to toggle the lock on the highest active layer, locking ADJ as intended. Layers RAISE and LOWER are not locked.
4. Layer switches for RAISE and LOWER are released. When this happens, update_tri_layer_state deactivates ADJ overriding the lock. Layer Lock notes that its lock was overridden and unlocks ADJ.

That last point is working as intended: Even when a layer is nominally "locked," it is possible and expected that other features may nevertheless turn the layer off. It is arguable a good thing that the lock is easily broken, since this avoids a situation where it is impossible to switch layers due to a layer being stuck on.

The solution is to adjust layer_state_set_user a bit to take into account when the ADJ layer is locked.

[EdenEast]

Thanks @getreuer, for the explanation and help with this. This solved my issue.

[getreuer]

Wonderful, that's great to hear. Thank you for confirming the fix and reporting this issue!