Prevent terminal windows from popping up during start on Windows

[gerritsangel]

Is your feature request related to a problem? Please describe.

It's really "ugly" that Darktable opens multiple terminal windows on startup, which close immediately. After all, Darktable is a GUI application, and no other GUI (which are not geared towards developers or really really special purpose) application opens a terminal screen on startup.

This is even worse, because you think that the Ui is opened, and then a terminal window quickly pops up directly in focus, thus preventing being able to click around in the UI.

Describe the solution you'd like

Just start Darktable without any terminal screens :)

Alternatives

I think there is no alternative. This behaviour is just completely unnecessary on Windows and it does not make any sense.

Additional context

I have this issue on Windows 11 Home. I think it might have something to do with the lua interpreter being loaded, and for each lua script loaded there is yet another terminal screen opening? Probably the lua interpreters should be started as GUI process?

Anyway, I cannot imagine how this was not yet discussed, but this is really one of the weirdest UI things I find with Darktable :)

I would also try to give it a go on fixing it myself, although not sure where to start. Any pointers would be appreciated :)

[wpferguson]

This is a "feature" of windows any time you call a system command from inside a program. If you search you'll find lots of threads about the problem with no solution. The command prompt is a gui application, so any time you run a command, the command prompt has to appear.

You could run WSL and run the linux version of darktable, which can launch a system command without flashing a window at you.

[ralfbrown]

I took a look at what the darktable code uses to run external programs. The Lua code uses system(), while printing and email use Glib functions in the g_spawn_* family. I wonder if g_spawn_command_line_sync (the equivalent of system) might be able to avoid the terminal window.... if not, g_spawn_sync should, but it's more work to setup.

[gerritsangel]

This is a "feature" of windows any time you call a system command from inside a program. If you search you'll find lots of threads about the problem with no solution. The command prompt is a gui application, so any time you run a command, the command prompt has to appear.

You could run WSL and run the linux version of darktable, which can launch a system command without flashing a window at you.

Hmm okay, but:

I have seen this behaviour in no other programs, even in the ones which use for example Lua. Let's say Davinci Resolve, it has a Lua and a Python scripting engine built in. It does not open any shell window. So there must be some solution, or probably the approach which Darktable uses is not appropriate.

And normal users do not know what WSL is, and even then I would have a Linux application running inside Windows , which then probably breaks most of the Windows integration (let's say file piker etc).

Maybe the issue is that there is a system command called? I mean, why does the Lua interpreter have to be called as a command, instead of just load the interpreter in C code? Is this some security feature?

[parafin]

It’s not the lua interpreter that runs separately. It’s programs run by it and maintenance things like updating lua scripts using git (if I’m not mistaken).

[gerritsangel]

Okay, thanks for the hint, i'll try and do some investigation.

[wpferguson]

The Lua interpreter is part of darktable, thus it has access to the darktable internals and can be used to extend the functionality. When a script executes an os.system() orio.popen() instruction the operating system command interpreter (bash on linux/macos, cmd on windows) is invoked to run the argument. Since CMD is a GUI application, the window has to open in order to run the argument (command).

See https://stackoverflow.com/questions/6362841/use-lua-os-execute-in-windows-to-launch-a-program-with-out-a-flash-of-cmd

[gerritsangel]

Okay, so I did some tests with a rust program:

#![windows_subsystem = "windows"]
use mlua::prelude::*;

fn main() -> LuaResult<()> {
    let lua = Lua::new();

    lua.load(r#"os.execute("notepad")"#).exec()?;
    Ok(())
}

So, indeed it behaves the same as Darktable. It will compile as a windows subsystem binary, it will open a terminal window and then notepad appears.

But if I start a command with only rust APIs, it only starts notepad:

#![windows_subsystem = "windows"]
use std::process::Command;
use std::io::Result;

fn main() -> Result<()> {
    Command::new("notepad").output()?;
    Ok(())
}

So, basically this tells me:

• It's not a general problem on Windows
• The problem is only Lua's os.execute() and io.popen().

But when lua's function is deliberately kept simple (which in general is probably fine)... Why not then replace it with a custom function which is OS dependent, or probably better, only on Windows replace it with a Windows specific API call which behaves the same as io.popen and os.execute?

I mean if Rust's standard library can execute a process under Windows with a windows subsystem application without opening a terminal, a C program should be able to do that as well.

[wpferguson]

@gerritsangel run your tests with MSYS2 and gcc and see if that works.

[gerritsangel]

@wpferguson Yes, still same behaviour. The standard Rust Command api does not open a terminal window, only the Lua one.

I used mingw-w64-ucrt-x86_64-gcc as written on the msys2 homepage, and for Rust i used toolchain stable-x86_64-pc-windows-gnu.

[wpferguson]

So what solution are you proposing/desiring?

darktable uses lua as supplied by lua.org. If there are issues with the way os.execute and io.popen work on windows, then the issue should be raised with the lua developers.

[parafin]

I think what is being proposed is to create a darktable-specific functions implementing os.execute and io.popen, that could be called from lua scripts, and then migrating all scripts to make use of them. Not sure if it’s a good idea. Maybe indeed it would be better to enhance lua’s original functions instead upstream…

[Dannny1]

This may be related also to the issue of incorrect colors on windows. #3619 If i remember correctly it appeared at the same time and also it appears always on primary monitor (similarly primary monitor profile is always used).

[wpferguson]

@Dannny1 this is a lua issue and has nothing to do with colors or multiple monitors.

[Dannny1]

It may not be only lua related as sometimes i manage to notice the quick pre-window appearing and i'm not using lua. If it' used to determine the profile, then it may explain the color management issue as it always opens on primary screen even when main window appears after on different one.

[wpferguson]

i manage to notice the quick pre-window appearing and i'm not using lua

You're not using Lua, but darktable is. On startup it checks if you have git installed and if the scripts are installed so that it knows if it can/should offer you the choice to install the lua-scripts.

[wpferguson]

Apologies in advance, this is going to be long

TLDR; Lua in darktable uses shell commands to interact with the operating system. On windows the shell is the command prompt which is a GUI application. Therefore, every shell command creates a window to run the command then closes it when the command finishes.

Purpose of Lua in darktable

darktable uses the lua-scripts to extend darktable's functionality both internally (no popping windows) and externally (popping windows). For example:

Internal

• add a button to the selection module to select untagged images
• add a button to the actions on selection module to clear the GPS information from the selected images.
• manipulate the GUI by hiding and showing modules
• manipulate the GUI process images by selecting processing modules and setting options, moving sliders, etc.
• add shortcuts for actions and processes
• color profile management
• persistent grouping across darktable instances (WIP)
• workflow automation (WIP)
• etc, etc, etc...

external

• external editor (GIMP, proprietary...)
• panorama stitching (hugin)
• HDR blending (enfuse, HDRmerge, ...)
• image stacking (imagemagick)
• focus stacking (enfuse)
• facial recognition
• smart tagging (photils)
• post export sharpening (imagemagick)
• etc, etc, etc...

How Lua works in darktable (why the windows pop up)

Lua needs information from the operating system in order to do the above listed processes. It's lightweight and only provides basic file operations and 2 methods of communicating with the operating system, os.execute() and io.popen(). In order to get/transfer information from/to the operating system "shell" commands are used. On Linux/MacOS this is bash|sh|csh|zsh|etc. which can be spawned and run headless. On windows this is the command prompt which can't run headless, so a window pops up.

When darktable starts up here's what happens in the Lua part

• _scripts_install runs as part of the Lua initialization process. It checks a preferences to see if the scripts are installed. If the preference is false, then _scripts_install_ checks for the lua-scripts directory (window pops up)
• If the user decides to install the scripts then _scripts_install checks for the git executable (window pops up)
• If the scripts are not installed, then a check is made to see if the luarc file exists (window pops up)
• If the luarc file exists, it is renamed to save it (window pops up)
• The scripts are installed (window pops up)
• The luarc file is created (window pops up)
• script_manager is started
• if check for updates is enabled then the current branch of the lua-scripts installation is checked (window pops up)
• if check for updates is enabled then the github repository is checked for branches (window pops up)
• if check for updates is enabled and a newer branch exists in the repository then the branches are pulled from the repository (window pops up)
• if check for updates is enabled and we are not on the correct branch, then the correct branch is checked out (window pops up)
• the scripts directory is scanned for all the scripts (window pops up).

So the worst case on startup is 11 windows pop up (you don't have the scripts installed and you want them). The best case (scripts installed and check for updates off) is one window pops up.

When internal (don't do anything with the OS) scripts are started no windows pop up.

When external scripts are started they may or may not pop windows depending on whether they want information when the start or they wait until they are used to determine the information. When they actually perform the action they are written for, then one or more windows is going to pop up.

Head spinning yet? It gets worse...

On windows if you run an external command it starts the process then returns so you don't know if the process exited normally or had an error and you don't know when the process ended so that your script can continue. So on windows a batch file is created with the command and the batch file is run. It holds the process open until the command completes and the return code is sent to the batch file. The batch file then sends the return code back to the caller and closes so that the script can continue.

Possible solutions...

User windows terminal instead of command prompt

Pros:

• Supposedly can run headless
• may support a command set closer to the Linux/MacOS command line

Cons:

• Only default on Win 11. darktable also runs on 8, 8.1 and 10 which have command prompt as the default.
• Doesn't support all the commands the command prompt does

---

Add external Lua packages to provide a richer O/S interface

Pros:

• Lots of Lua packages exist to provide more functionality

Cons:

• Users would have to install them. Luarocks provides a tool to do that, but it depends on other tools. Also, installing git was/is a huge issue for the users, so I don't see using Luarocks as a real possibility.

---

Rely more on what Lua "knows" about darktable and the OS (WIP)

We currently store information about executables (or at least we try to), but we confirm the information (pop windows) before we use it.

Pros:

• Fewer windows popping

Cons:

• if the situation on disk is different the situation that Lua "knows" then things will crash.
• it will be more difficult to troubleshoot user issues because it will be harder to determine the "truth"

---

Code up windows user interface routines to bypass the command prompt

Pros:

• could work

Cons:

• we don't have any windows devs
• extra code to maintain
• more bugs to fix
• harder to troubleshoot scripts (interface issue or script issue)
• integrating into the scripts will be a PITA (either we hide it in the libraries or we have lots of if statements or we have a windows set of scripts and an all others set of scripts)
• we might have to code a function for each query we make to the operating system (i.e. all the git interaction would be a separate module, search functions, test functions, etc.)

---

I was playing last night launching GIMP from a C program using system(). I could launch it without a window popping but I noticed that between the splash screen and the UI opening 2 windows flashed. I then started GIMP from the windows shortcut and saw the same issue. I'm pretty sure it was the script engines reading the directories to see what scripts were installed. So it appears that darktable isn't the only cross platform program that experiences this issue. darktable is much worse due to the power of the extensions. As I was playing with spawning GIMP I came up with another possibility:

Open a window with io.popen() and use it to communicate with the shell

Pros:

• No "popping windows". One window opens and stays open for the duration that darktable runs. Depending on when it starts it'll probably end up behind the UI so the user may not even notice.

Cons:

• I don't know.... I have several ideas about how to do it, so I'll play with it and see what I can figure out.

EDIT: This wont work, io.popen is read only. However, spawning a process and attaching a couple of pipes to it might be doable.

[gerritsangel]

Thanks for all the replies everyone :)

So the worst case on startup is 11 windows pop up (you don't have the scripts installed and you want them). The best case (scripts installed and check for updates off) is one window pops up.

This is exactly what is happening on my machine. Might not be 11, but it's definitely about 5 or more. The problem is also, it's not even in rapid succession. You think everything is done, and then yet another terminal pops up. It gets the focus, it disrupts the workflow. I cannot imagine how new users feel when they see an application like this and don't know of the reasons of it., I anticipate they will uninstall Darktable right away. As there is also no explanation why this is happening, I can't even know that this is related to Lua.

So what solution are you proposing/desiring?

darktable uses lua as supplied by lua.org. If there are issues with the way os.execute and io.popen work on windows, then the issue should be raised with the lua developers.

If the Lua concept is to be a lightweight library by default, then I can see their decision to not code much OS specific stuff inside. Probably they just want to rely on the standard C library. That might be fine for their use case. Might be an idea to check up with them, though.

But as far as I understood it, Lua is customizable and you can just replace normal Lua functions with custom ones by overwriting the table (?) where all these functions are replaced. Maybe this is even their goal: You can use the default interpreter as is, but if you want to integrate it better, you need to adjust it. In my experience, this is normal usecase in software development. Many libraries allow customization, so why not do that here as well? Nothing really works 100% out of the box flawlessly.

So for me it's just a different/more specific implementation of an interface. Replace os.execute() and io.popen() with either custom written Windows functions or check if there is already something available.

From what I anticipate, no scripts would need to be changed , because the signature and the name of os.execute() and io.popen() stay the same.

---

Users would have to install them. Luarocks provides a tool to do that, but it depends on other tools. Also, installing git was/is a huge issue for the users, so I don't see using Luarocks as a real possibility.

Which normal (non-dev) knows what even Lua is, what Luarocks is, and even more: Why need to install this to fix some application bug? I think this is not good. Yes, these users might have wanted to use some Lua extensions, but it could also well be from being told in the Internet "yeah you can use this by installing these extension" without even knowing what they are doing. Especially if some really handy features are hidden inside the extensions.

Pros:

* Fewer windows popping

Fewer windows is still more than zero, so imho still not good user experience.

Cons:

* we don't have any windows devs

* extra code to maintain

* more bugs to fix

Of course there might be bugs, but look at the upsides:

• No flashy terminal windows at every start of the application anymore
• New users are not distracted by this and don't immediately discard Darktable anymore (I've seen people telling me "Why does it load a terminal on startup 😆 " when I recommend them Darktable.)

[wpferguson]

So we are stuck at

• we don't have any windows devs

unless you want to take it on.

[gerritsangel]

Yes, I can give it a try. No guarantees, and I probably need some help, but I would be willing to try :)

[wpferguson]

I can write the swapping functions and create a lua module for you to drop the code into. You could probably just grab the io.popen and os.execute functions from the lua source and just work on the part where it calls the operating system.

I've attached a file with the stuff you need. Unzip it, then untar it from the top darktable directory.

winstuff.zip

[wpferguson]

Data point: Just tested #17141 on win 11. When I run it in a terminal popping windows go away. As a further test, I ran it from a command prompt and got all the flashing windows. So, the ultimate (easiest) fix may be to change the shortcut to run darktable in a terminal (maybe).

EDIT: Installed Terminal on win 10 and tried running darktable from there. Windows still popped. I'm guessing because Terminal isn't the default command processor on win 10.

[gerritsangel]

Does this mean that the user has to start Darktable in the terminal or does this workaround also work when starting DT normally? I mean if it solves the issue completely, I guess it's better than trying to replace the Lua code. But the entire existence of the command line should be hidden on Windows, because this is just not normal use case for a normal Windows person, who is not a developer or system administrator. (Also, I don't see why it would be even necessary to have a console open).

Currently I have started a bit to play with the Lua code. I'm currently checking CreateProcessW function, which is also used by the Rust runtime, so I thought it's a good starting point. Especially to do just like system() behaves should not be so problematic, haven't yet checked with the io.popen yet, though...

[wpferguson]

If you start darktable normally, then you get popping windows. If you start darktable from the command line in terminal, no popping windows. I was thinking that maybe we could hijack the shortcut and force it to run a terminal in background (headless) and the start darktable from that. May work, may not, haven't tried...

Whatever solution you come up with has to compile with msys2, since that's the build environment.

[ptilopteri]

• wpferguson @.***> [08-12-24 10:51]:

  If you start darktable normally, then you get popping windows. If you start darktable from the command line in terminal, no popping windows. I was thinking that maybe we could hijack the shortcut and force it to run a terminal in background (headless) and the start darktable from that. May work, may not, haven't tried...

  Whatever solution you come up with has to compile with msys2, since that's the build environment.

is it not possible to set the window app to start within a terminal?

make the app icon/shortcut reflect the same

-- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri facebook/ptilopteri Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet oftc