Consider userinit instead of, or in addition to, shell replacement.

Overview

It can be tough to get the Libki client configured so it loads after user (auto) login, but before the user shell starts. It would be nice if there were a cleaner way to start the Libki client between those two events. I think userinit may be a decent solution to consider and, based on my testing, it seems to work well as-is.

Problem

The shell replacement option in the installer doesn't work well in some cases. It looks like Libki client is running explorer.exe to launch the shell. This StackOverflow question describes some of the conditions that need to be met to get that to work. Specifically:

Explorer must see it is the actual shell - hence you need to replace that value before launching explorer.exe (could change it back a few seconds later)

One of the other answers in that StackOverflow question claims that C:\Windows\explorer.exe will always launch the shell, but that's incorrect based on my testing. It doesn't change anything.

The above is saying that HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon must have Shell=explorer.exe for the normal shell to launch when running explorer.exe. Based on my testing, that's correct. With that value set to C:\Program Files (x86)\Libki\libkiclient.exe the normal shell won't launch when explorer.exe is run. Based on my testing all that happens is the file manager launches.

I tried updating the run_on_login option in the Libki client .ini config with a script that would modify the above registry value prior to launching explorer.exe, but I ran into two issues.

The value needs to be set back to libkiclient.exe. This isn't impossible and would probably be tolerable if it were the only drawback.
It doesn't work with guest accounts. The user logging in needs to have permission to edit HKEY_LOCAL_MACHINE and guest accounts don't.

Solution

Based on my testing, the Userinit value in that same registry key may be a better solution. I've only tested the use case that fits my needs, but it seems to work really well as-is. This is what I did.

Prep PC

Create two user accounts; an administrator named Tech and a guest (aka limited) account.

Install Libki Client

Install the client. When prompted for Client Information:

Run only for this user: Leave this blank.
Run for all users but this one: Tech (or the name of the admin account).

When prompted for Startup Mode:

Do not start Libki client automatically.

Post Install Config

As the administrator, open a PowerShell prompt and run the following command to modify the Userinit registry value.

Set-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon' -Name 'Userinit' -Value 'C:\Program Files (x86)\Libki\libkiclient.exe,C:\Windows\system32\userinit.exe,'

Explanation

I couldn't find much documentation for Userinit, so this is based on what I've observed. I think the programs run sequentially and the first one (libkiclient.exe) will block the second one (userinit.exe) from running. I'm basing that on the way it behaves when I login. The guest account blocks with the Libki client UI until a successful authentication occurs. The administrator account continues right away. I assume that's exiting here because of the Run for all users but this one option.

I don't know enough about C++ and QT to step my way through the rest of the client to see how / where it exits. I assume the main process needs to exit before userinit.exe will run, but I don't know for sure. It's worth paying attention to because the current startup behavior seems to be good as-is and, since libkiclient.exe is running before userinit.exe, care would need to be taken to ensure nothing is ever changed in a way that blocks userinit.exe from loading.

If I understand everything you wrote correctly, UserInit would be more like the older method that uses the registry key HKLM/Software\Microsoft\Windows\CurrentVersion\Run to launch the client, but sooner in the startup chain. I'm reading Userinit.exe is the file responsible for executing the logon scripts, re-establishing the network connection, and then starting Explorer.exe. from https://www.minitool.com/news/userinit-exe.html but I'm not sure that can be correct if the client is working for you, because the client requires networking in order for user logins to work!

This definitely sounds like a great option in addition to the existing two options. I think at least for most use cases it's probably the best option available. I'll do some testing and see if I can add this as an option to the installer!

@kylemhall I tested this a bit more, specifically the (non) blocking behavior and the networking behavior.

I was wrong about the blocking. The first entry in Userinit does not block subsequent entries. I think what I was likely seeing is the Libki client loading quickly enough that it was full screen and blocking input before userinit.exe got anything loaded. I verified this by putting cmd.exe, a simple program I wrote called gowait.exe, and userinit.exe in the Userinit entry. My gowait.exe executable blocks until it sees SIGINT or SIGTERM and I had a full desktop before I gave it a signal to exit.

Based on that, I'm confident everything in Userinit loads simultaneously.

For the network behavior, I'm not sure if that linked doc is correct based on what I observed, at least for Win10. I tested two different Win10 machines. One was a VM with a wired connection and one was a laptop with a WiFi connection. I tested by replacing the C:\Windows\System32\userinit.exe value in Userinit with C:\Windows\System32\cmd.exe. It loads a command prompt and nothing else after login. On both devices I had network connectivity and was able to ping google.com.

I was also able to manually run userinit.exe from the command prompt and it loaded the shell / desktop as expected. I would say there are a couple methods that would probably work well for Libki, assuming the Libki client would be the only entry in Userinit and would become responsible for running userinit.exe.

Start Libki and put up the modal screen that blocks input. Then run userinit.exe. This is probably real close to what I saw when I was testing, but has the advantage of guaranteeing the Libki client loads before anything else. As long as nothing but the Libki client can be interacted with, this has the advantage of having the desktop instantly available after a successful authentication. It's also probably a bit safer if there are Windows versions that rely on userinit.exe for networking like that linked doc suggests.
Start Libki and wait until a successful authentication happens before loading userinit.exe. This has the advantage of keeping less stuff running until a user provides valid credentials, which means less opportunity to bypass the Libki client.

Those are very similar. Either would be great IMO. Let me know if there's anything else that would be helpful for me to test.

I think overriding userinit could have unknown side effects (not running scheduled tasks for the user, etc). Of course, this is just speculation, as there is little to none information about what it actually does.

There could be other alternatives to solve the problem without resorting to the use of userinit. The original problem is that explorer.exe needs to see itself as the default shell to load the desktop. To make that happen, the Libki client should somehow be able to change the Shell registry entry back to explorer.exe, run it and switch the setting back. For Libki to be able to do that, the Shell entry to replace should be under HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon, instead of HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon (as it is now), so the Windows user running the Libki client does have the permissions to make that change.

For that to work, the installer would have to know in advance the name of the OS user running Libki. Currently, the installer is replacing HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\Shell because it doesn't know which Windows user will be running Libki, so that all users get the setting, but as @ryanjaeb has found out, it doesn't actually work, as the client cannot later revert it back. One solution could be to have the installer apply the replacement of the shell to the Windows Default User, so that all new users created after that would get the setting when their profiles are created. That could be done by locating the NTUSER.DAT registry file for the default user (should be on %SYSTEMDRIVE%\Users\Default), loading it, modify the Shell entry and unloading it again. All of this should be doable from Inno Setup unsing Exec and the OS reg.exe command. I will try to make a test, and create a PR if it works.

I have performed additional tests on a Windows 10 21H2 machine:

Inside regedit, loaded default user registry hive from C:\Users\Default\NTUSER.DAT
For that default user, changed SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\Shell to C:\Windows\system32\cmd.exe and unloaded the registry hive.
Created a new user, and logged on. cmd.exe now appears instead of explorer.exe.
From that cmd, run explorer.exe. The full desktop appears.
As administrator, changed HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\Shell to C:\Windows\system32\cmd.exe.
Log in again with the new user. cmd.exe appears instead of explorer.exe (same as before).
From that cmd, run explorer.exe. This time, only the file explorer window appears.

From that test, it seems explorer.exe only checks if it is the defaul shell under the Shell key from HKEY_LOCAL_MACHINE, not the current user key. So, if that setting is modified only for the default user (instead of the machine), it will be applied to all users created afterwards, and there is no need to revert the registry key for Libki to start the full desktop via explorer.exe.

Libki / libki-client