Generating a command-line executable without the GUI features

[kakyoism]

Is your feature request related to a problem? Please describe.

The Beeware tutorials only mention how to build and distribute a GUI app. It's unclear how to generate a command-line program.

Describe the solution you'd like

I wonder if I could opt out of the GUI requirements and build a standalone CLI executable that includes all the implicit Python dependencies, i.e., a single-file executable. Then with the package command this executable can be included in an installer.

[freakboy3742]

Thanks for the suggestion!

Supporting command line apps is definitely something we'd like to do; and conceptually, there's nothing about Briefcase that is inherently "GUI" about what it generates. The sticking point is working out how to wrap the Python app so that it can be accessed as a command line app.

Briefcase specifically doesn't use the "Single file executable" model, because of problems I've historically seen with that approach. A lot of Python code in the wild includes implicit assumptions about how it is deployed - that's why, for example, the egg format includes the 'zip-safe' flag.

If you're particularly enthused about the single file executable model, there are tools (like pyinstaller) that can support that. You could probably even write a Briefcase platform backend that would serve as a wrapper of that tool using Briefcase's plugin system. However, we're unlikely to merge a single-file executable model into Briefcase itself.

[freakboy3742]

After thinking about this some more, I think supporting command-line apps may not be as hard as I first thought.

Linux apps are now command-line compatible. System packages install are installed into /usr/bin; if you only use stdout/err rather than a GUI toolkit, you can write a command line app.

So - this leaves macOS and Windows. On these platforms, the actual binaries would work fine - if only they could be put on the path. If you manually invoke myapp.exe from the command line, it will run. The challenge becomes how to put the executable on the command line.

I think we may be able to do this with installers.

On Windows, we already have an MSI installer; modifying %PATH% shouldn't be too difficult. We will also need to name the executable with the app name, rather than the formal name, and disable the generation of Start menu items. This may require addressing #1014.

On macOS, we'd need to introduce a formal installer, rather than .app or .dmg (see #1184) - but once that installer exists, a post-install script can add the binary to the path. This is effectively what Python.org does with its macOS installer. To make this work, we'd need to:

• Rename the bundle to .framework, rather than .app.
• Change the binary to use the app name, rather than the formal name
• Install into /Library/Frameworks, rather than /Applications.

There may be some other internal changes that would be advisable; this may warrant using an entirely different template for both macOS app and Xcode builds.

In both the Windows and macOS case, these changes could be enabled by an app-level command_line_app = True setting. This setting would be a no-op on Linux, and would raise an error on Android, iOS and web.

Lastly, it would be worth auditing the command line stubs to ensure that they are fully isolated from the runtime environment. We need to ensure that if the command line app is run inside an active virtualenv, the packages in the virtualenv (or PYTHONPATH, or any other python environment settings) don't alter the operation of the app.

[brendan-simon-indt]

Forgetting a single file executable for now, briefcase should be able to use the zip package format (for distribution), then the user can unpack it wherever they like and just run the executable manually from the command line.

I did that (uncompressed the zip and run the .exe executable from the command line) but it "didn't work". My guess is it is a stdin/stdout/stderr issue - not attached to the console.

My use case is to quickly package something up for a small number of users (maybe 1 or 2) to quickly test, get some results, and that's basically it. There might be some quick turn-around of versions to try - the installer part is just a pain for this use case. NOTE: this use case is also applicable to an app with a GUI. i.e. my use case is to produce a "portable/stand-alone" distribution, whether it's zip (or single-file exe), that can be placed anywhere an executed, without "installation".

[JPHutchins]

Hello BeeWare team & @freakboy3742 !

I have a repository that demonstrates universal python packaging and I've created a wix template that accomplishes the CLI features required. It assumes PyInstaller (directory, not one file) but I think that could be worked around.

Add to PATH: https://github.com/JPHutchins/python-distribution-example/blob/51c7914e4d99e2b4f48dbb88ef1062448ea0293a/distribution/windows/Package.wxs#L44-L49

For Linux I agree with the assessment above. I was using FPM for packaging to deb and rpm and simply add a "after-install" script. Seems a hack, but hey, Linux?

https://github.com/JPHutchins/python-distribution-example/blob/51c7914e4d99e2b4f48dbb88ef1062448ea0293a/distribution/linux/after-install.sh#L6-L7

Afraid I don't know much about MacOS! I was sorta hoping it would "just work". Speaking of which, will briefcase build me universal binaries on the GitHub macos runner?

In addition to adding to PATH, I think another feature that's helpful for Windows users is having a double-clickable version of the CLI that simply launches a terminal that's running the program. This is accomplished by having Wix create a shortcut:

https://github.com/JPHutchins/python-distribution-example/blob/51c7914e4d99e2b4f48dbb88ef1062448ea0293a/distribution/windows/Package.wxs#L87-L97

It works because the CLI app has a flag -i meaning "interactive" or "shell" mode.

I'm working on a series of articles about python application distribution with a focus on security and I'd like to include the briefcase environment as I'm working through it, especially for GUI apps. I experimented with pyoxidizer and got it working but abandoned it due to bloated sizes, poor/wrong documentation, and it seems to be abandoned. PyInstaller is "OK"... but you know. 🤣 Everything flags it as a virus. Which I suppose will happen to briefcase as well once it takes off unless it's mitigated somehow by executing signed python interpreters?

https://dev.to/jphutchins/building-a-universally-portable-python-app-2gng

[freakboy3742]

I have a repository that demonstrates universal python packaging and I've created a wix template that accomplishes the CLI features required. It assumes PyInstaller (directory, not one file) but I think that could be worked around.

Add to PATH: https://github.com/JPHutchins/python-distribution-example/blob/51c7914e4d99e2b4f48dbb88ef1062448ea0293a/distribution/windows/Package.wxs#L44-L49

Ah - that's a lot easier than adding WiX features usually ends up being :-) That should be a relatively straightforward change to add to the briefcase-windows-app-template and briefcase-windows-visualstudio-template.

I think Windows may need one more change in addition to this WiX configuration. The stub binary that Briefcase uses bakes in some assumptions about the app being a GUI app; those assumptions need to be relaxed if running from the command line. As an example, package an app where the Python code deliberately includes a syntax or import error; the app will pop up a dialog, rather than outputting the stack trace to the console. Strictly, I guess that change isn't essential, but I suspect most console-based Windows apps would prefer console-based behavior.

For Linux I agree with the assessment above. I was using FPM for packaging to deb and rpm and simply add a "after-install" script. Seems a hack, but hey, Linux?

https://github.com/JPHutchins/python-distribution-example/blob/51c7914e4d99e2b4f48dbb88ef1062448ea0293a/distribution/linux/after-install.sh#L6-L7

I think this might be an artefact of your usage of FPM, rather than Briefcase. Briefcase is able to generate DEB, RPM and PKG files; in my testing, installing a briefcase-generated package results in a binary in /usr/bin.

Afraid I don't know much about MacOS! I was sorta hoping it would "just work".

It "just works" for a GUI app, but not for a command line app; as with Windows, the issue is putting an item on the PATH. The complication is that a macOS App doesn't have an installer - it's a self-contained executable. As described above, the fix will be to create .pkg installer, which would allow for a post-install script that adds the binary to the user's path.

Speaking of which, will briefcase build me universal binaries on the GitHub macos runner?

Yes. The binaries produced by macOS are always universal binaries (unless explicitly disabled with universal_build=False in the app config); the only place that will fall down is if you have a binary dependency that doesn't provide one of the required architectures (in which case, there's not much Briefcase can do).

In addition to adding to PATH, I think another feature that's helpful for Windows users is having a double-clickable version of the CLI that simply launches a terminal that's running the program. This is accomplished by having Wix create a shortcut:

https://github.com/JPHutchins/python-distribution-example/blob/51c7914e4d99e2b4f48dbb88ef1062448ea0293a/distribution/windows/Package.wxs#L87-L97

It works because the CLI app has a flag -i meaning "interactive" or "shell" mode.

I can see how that could be useful. If this were to be added to Briefcase, I'd suggest it should be controllable with an option flag so it can be opt in/out.

I'm working on a series of articles about python application distribution with a focus on security and I'd like to include the briefcase environment as I'm working through it, especially for GUI apps.

Any publicity is good publicity :-)

I experimented with pyoxidizer and got it working but abandoned it due to bloated sizes, poor/wrong documentation, and it seems to be abandoned. PyInstaller is "OK"... but you know. 🤣 Everything flags it as a virus. Which I suppose will happen to briefcase as well once it takes off unless it's mitigated somehow by executing signed python interpreters?

Briefcase includes binary and installer signing for Windows, so this shouldn't be an issue (provided you've got a certificate that is trusted by Windows itself). If you find any evidence to the contrary, let us know.

[JPHutchins]

Briefcase includes binary and installer signing for Windows, so this shouldn't be an issue (provided you've got a certificate that is trusted by Windows itself). If you find any evidence to the contrary, let us know.

I am in touch with the Azure Code Signing team: https://techcommunity.microsoft.com/t5/security-compliance-and-identity/azure-code-signing-democratizing-trust-for-developers-and/ba-p/3604669

Generally, FOSS isn’t thrilled to pay for MS’s certificates. And FOSS should not be distributed without a means of authentication. Hoping that MS steps up to the plate here.

I will investigate the wix system in briefcase and see what works. Agreed re: popups and opt-in for a shortcut.

[freakboy3742]

Briefcase includes binary and installer signing for Windows, so this shouldn't be an issue (provided you've got a certificate that is trusted by Windows itself). If you find any evidence to the contrary, let us know.

I am in touch with the Azure Code Signing team: https://techcommunity.microsoft.com/t5/security-compliance-and-identity/azure-code-signing-democratizing-trust-for-developers-and/ba-p/3604669

Generally, FOSS isn’t thrilled to pay for MS’s certificates. And FOSS should not be distributed without a means of authentication. Hoping that MS steps up to the plate here.

Good luck with that :-) I'd be happy to be proven wrong, but personally, I wouldn't hold my breath on Microsoft changing their mind or approach on this. I'd posit that an approach much more likely to succeed would be a Let's Encrypt-style community project that decides to offer a usable certificates at no cost to FLOSS projects.

[JPHutchins]

@freakboy3742 So far I have no luck getting print() stdout from the built app on Windows. To address this, I need to understand better what the generated "exe" actually is. I ask because I can launch the GUI app from the console but the console returns immediately instead of remaining open.

I see this entry:

self.tools.subprocess.run(
    [
        self.tools.rcedit.rcedit_path,
        self.binary_path(app).relative_to(self.bundle_path(app)),
        "--set-version-string",
        "CompanyName",
        app.author,
        # Although "FileDescription" sounds like it should be a... description,
        # this is the label that appears as a grouping in the Task Manager
        # when the application runs.
        "--set-version-string",
        "FileDescription",
        app.formal_name,
        "--set-version-string",
        "FileVersion",
        app.version,
        "--set-version-string",
        "InternalName",
        app.module_name,
        "--set-version-string",
        "OriginalFilename",
        self.binary_path(app).name,
        "--set-version-string",
        "ProductName",
        app.formal_name,
        "--set-version-string",
        "ProductVersion",
        app.version,
        "--set-icon",
        "icon.ico",
    ],
    check=True,
    cwd=self.bundle_path(app),
)

but I do not see where the exe is created.

The run_app method of WindowsRunCommand is not run by the built executable as evidenced by the app continuing to work after altering it. I was looking here because I saw the stdout redirection and thought this was the issue.

Long story short, I'm looking for the entry point and it feels like there's some Windows magic happening that I'm missing! Like, it's executing the application detached from the shell that it was called from.

Thanks, JP

[freakboy3742]

There are two backends that you can use for Briefcase on Windows- the app backend, and the VisualStudio backend. The difference between the two is that the VisualStudio backend is a full Visual Studio project that compiles an exe; the app backend includes a pre-compiled version of the output of the default VisualStudio project.

So - if you want to tinker with the actual executable, you probably want to use the VisualStudio backend, by invoking briefcase create windows visualstudio (and similar for build, run etc). This uses the template from https://github.com/beeware/briefcase-windows-visualstudio-template.

The https://github.com/beeware/briefcase-windows-app-template is what is used by default if you run briefcase create on Windows. The pre-compiled exe is included in that template; it is generated as part of a CI action. The pre-compiled option exists because most users won't need to modify the executable; and it removes the need to have a Visual Studio install to use Briefcase.

[JPHutchins]

Excellent, I see the Stub-<>.exes in my fork now. I will continue my investigation from there.

[rmartin16]

@JPHutchins, when I originally evaluated sending stdout to a console, I came up with this:

AllocConsole();
FILE* fDummy;
freopen_s(&fDummy, "CONOUT$", "w", stdout);
freopen_s(&fDummy, "CONOUT$", "w", stderr);
freopen_s(&fDummy, "CONIN$", "r", stdin);

This replaces the if (!AttachConsole(ATTACH_PARENT_PROCESS)) { block. Just FYI.

[JPHutchins]

Interesting, lots of boiler plate, but that's the whole point I suppose!

Feels like it was already considered: https://github.com/beeware/briefcase-windows-VisualStudio-template/blob/5e349f51e8b5f2d1c1a98806934e541182a04ffa/%7B%7B%20cookiecutter.format%20%7D%7D/%7B%7B%20cookiecutter.formal_name%20%7D%7D/Main.cpp#L52-L54

Or is the "default" at the Visual Studio template not what was used to create the stubs at https://github.com/beeware/briefcase-windows-app-template?

[JPHutchins]

Where is the source for the stub binaries located here https://github.com/beeware/briefcase-windows-app-template/tree/main/%7B%7B%20cookiecutter.format%20%7D%7D/src?

[freakboy3742]

Those stub binaries are the ones generated from the Main.cpp you linked to.

[freakboy3742]

The binaries in the -app template are created using this workflow, triggered by tagging the repo.

[JPHutchins]

Interesting, the AllocConsole(); creates a new terminal, which is cool, but not necessary!

[JPHutchins]

Hello, World!

./build\hellocli\windows\visualstudio\x64\Release\hellocli.exedio\x64\Release\hellocli.exe
Hello, from main!

It looks like the main switch will be changing the subsystem from "Window" to "Console": https://github.com/JPHutchins/briefcase-windows-VisualStudio-template/commit/a4c1c276fe37d827847a03f7029f72df6bb5a410

It's already a template so that's easy enough. After further testing I can look into creating stubs since "installing Visual Studio" could generate some friction for users 🤣

[freakboy3742]

To be clear - if we make a change to the Visual Studio template, we pretty much automatically re-tag the app template as well to generate new stubs. If it's a change that will enable a fix to this ticket, we'll definitely re-tag the binary.

Having the visual studio template is mostly a convenience for people working on the stub, with a very minor additional benefit that if you want to link in complex third-party libraries, you've got a way to do that.

[JPHutchins]

Sorry, I wasn't clear! None of this is intended to modify the existing system. New stubs would be added, postfixed with "-cli" for example, to provide the config that a terminal app needs.

[freakboy3742]

As a guide - I'd vastly prefer to not have a separate stub binary for CLI apps. I'd much rather work out a way to make the two use cases co-exist in a single binary.

A related piece of work is the removal of binaries from the templates discussed as part of #1523. Binaries don't fit well into the templates; if introducing a second binary is unavoidable, that might accelerate the need to do the 'binary-ectomy" part of that ticket.

[JPHutchins]

Does #1523 suggest that once the templates are consolidated, then it would be desirable for the stub sources to get their own repos with tagged release artifacts? I can state that it would have simplified development. Potentially simplifies unit testing for the stub binaries, though I have not dug into the test systems yet.

[JPHutchins]

I might backtrack on another point. If a developer is building the Windows executable locally, it is reasonable for them to install Visual Studio. If they develop from some other OS, then they can build their binaries in a GitHub workflow. Perhaps the concern about binary variants is a moot point and the user should be forced to use the “windows VisualStudio” build.

[freakboy3742]

1523 is still an open discussion, especially the extent to which "consolidation" actually happens. If I had to use my crystal ball, I'd say that extracting the binaries as a separate artefact is the most likely, and possibly the only part of that ticket that will actually happen.

I wouldn't expect to see a separate repo, though. What I'd expect to see is that the binary artefact that is built out of the visual studio template being published to an S3 bucket, rather than a GitHub commit. The issue that exists is entirely about using Git as a distribution channel for binaries - a task for which it isn't well suited, because all historical binaries are included in the commit history.

We're unlikely to ever require visual studio to package a Windows binary, except possibly as a stop-gap measure to enable CLI-based apps before we've had a chance to split out binaries from the template.

[mhsmith]

What I'd expect to see is that the binary artefact that is built out of the visual studio template being published to an S3 bucket

Wouldn't it be better to use a GitHub release artifact, like we do for cpython-apple-source-deps? Keeping everything on one site would make it easier to automate and manage.

[freakboy3742]

So - once upon a time, we did use GitHub artefacts. But then we managed to hit download limits during a PyCon sprint when we had 25 or so developers simultaneously downloading the same artefact. Moving to S3 hosting avoided this problem.

That doesn't impact the cpython-source-deps because it's not downloaded by many people, whereas the support packages are downloaded by every user of Briefcase.

As a side effect, using S3 means we're also in the position to track download metrics, which GitHub doesn't give us.

That said - I haven't revisited this in several years, so it's possible (even probable) that the download limits have been lifted. The download location ultimately doesn't really matter, as long as it's reliable.

[JPHutchins]

Well, things ended up being a bit trickier than I expected - like, learning that C++/CLI exists - but I finally have an MVP.

You can clone this repo to build a briefcase CLI app for Windows: https://github.com/JPHutchins/test-briefcase-cli

It relies on Visual Studio building this fork of the Windows template: https://github.com/JPHutchins/briefcase-windows-VisualStudio-template/blob/main/%7B%7B%20cookiecutter.format%20%7D%7D/%7B%7B%20cookiecutter.formal_name%20%7D%7D/Main.cpp

Most of the complexity is introduced by challenges in capturing stdout/stderr. As far as I understand it, the briefcase application has 3 phases:

1. Initialization. Logs should go to log file unless a critical error occurs in which case it should go to console + logs (the crash dialog, now in the console).
2. App is running. Everything from the app's stdout stderr should go to the console.
3. App has exited. If it exits without an error code (python exception?) then the console should be quiet and the return code should be forwarded from the python app. Otherwise use the custom traceback formatter.

So, I think this is accomplished but of course needs testing in the field. I've tested three scenarios:

1. Initialization error logging. Screw up args to the various initializers to make them fail and make sure that stderr was captured to the logs and a message printed via crash dialog.
2. App runs successfully without error. Console should output what the python app wants.
3. App runs and exits with exception. Console outputs up to the exception then prints the custom formatted exception to console and the log file.

Problems

1. Could not find a way to redirect stderr to an open log file + console. I tried lots of approaches. This would have simplified things. Every call that may have the side effect of writing to stderr should be surrounded by log->StartStdErrCapture(); and log->StopStdErrCapture();. The deficiency now is that the log file will contain the stderr output that was captured during calls like Py_InitializeFromConfig(&config); but the console will not. As a compromise, the console output directs the user to the log file for more details.

The needs around console vs log output are different enough from a GUI app that I think it's justified for this to be its own template. Even if there was a way around the subsystem:windows vs subsystem:console thing.

Once we've decided on the approach I'd like to help draft a roadmap for this feature.

[freakboy3742]

Well, things ended up being a bit trickier than I expected - like, learning that C++/CLI exists - but I finally have an MVP.

🎉

You can clone this repo to build a briefcase CLI app for Windows: https://github.com/JPHutchins/test-briefcase-cli

It relies on Visual Studio building this fork of the Windows template: https://github.com/JPHutchins/briefcase-windows-VisualStudio-template/blob/main/%7B%7B%20cookiecutter.format%20%7D%7D/%7B%7B%20cookiecutter.formal_name%20%7D%7D/Main.cpp

Most of the complexity is introduced by challenges in capturing stdout/stderr. As far as I understand it, the briefcase application has 3 phases:

1. Initialization. Logs should go to log file unless a critical error occurs in which case it should go to console + logs (the crash dialog, now in the console).
2. App is running. Everything from the app's stdout stderr should go to the console.
3. App has exited. If it exits without an error code (python exception?) then the console should be quiet and the return code should be forwarded from the python app. Otherwise use the custom traceback formatter.

I'm not sure this is entirely correct.

You're correct that there's 3 different "phases" of output - pre-startup, runtime, and exit.

The pre-startup output is entirely debugging content, mostly useful to the developers of briefcase to ensure that PYTHONPATH etc has been configured correctly. It's not part of "normal" app output; it could be considered vaguely analogous to the output you get with python -v -m myapp - "verbose mode" output to help diagnosing the Python interpreter itself.

The "runtime" output is anything that is usually printed to stdout and stderr.

The exit output will be nothing in a well behaved app; but if the python code crashes, a stack trace needs to be displayed.

The underlying problem here is that a GUI application, when started from the start menu or an icon, doesn't have a stdout that is visible. As a result, if there's a crash, or some other issue with app startup, it's completely invisible to the user - they click on a link, and it does nothing. For these cases, it's essential that errors are surfaced graphically. It's also helpful if "normal" runtime output is written to a log for diagnostic purposes - if an app crashes, you can ask the user to send you the log file.

When you're running a GUI application from inside Briefcase (or, I guess, starting a GUI app from a command shell), you do have a console, and it's lot more helpful to see console output in the actual console, rather than needing to wrangle a log file. The "verbose pre-start" output isn't a problem, but isn't really needed either.

Briefcase currently differentiating between these two cases by checking for the existence of an attached console, and disabling log handling if the attached console exists.

You're adding a third case - a pure console app. For this case, console output is preferred in all cases, because GUI popups are problematic. Log files are possibly nice to have, but are essentially superfluous because you could also generate them by redirecting console output to a file. The "verbose pre-start" output is a distraction that you don't want to see unless you're actually debugging.

So, I think this is accomplished but of course needs testing in the field. I've tested three scenarios:

1. Initialization error logging. Screw up args to the various initializers to make them fail and make sure that stderr was captured to the logs and a message printed via crash dialog.
2. App runs successfully without error. Console should output what the python app wants.
3. App runs and exits with exception. Console outputs up to the exception then prints the custom formatted exception to console and the log file.

Problems

1. Could not find a way to redirect stderr to an open log file + console. I tried lots of approaches. This would have simplified things. Every call that may have the side effect of writing to stderr should be surrounded by log->StartStdErrCapture(); and log->StopStdErrCapture();. The deficiency now is that the log file will contain the stderr output that was captured during calls like Py_InitializeFromConfig(&config); but the console will not. As a compromise, the console output directs the user to the log file for more details.

I'm glad I'm not the only person who had fun wrestling this :-)

However, from what I can make out, a lot of the changes you've made to the template revolve around finding a clean abstraction of logging so that log file output is preserved. Unless I'm missing something, adding handling for logging in addition to console output isn't really needed here - you can get the same logs by capturing the console output into a file.

Have I misunderstood the nature of the changes you've made here?

The needs around console vs log output are different enough from a GUI app that I think it's justified for this to be its own template. Even if there was a way around the subsystem:windows vs subsystem:console thing.

Templates can contain logic, so switching between subsystem:windows and subsystem:console based on a Briefcase configuration flag won't be hard to add. Even enabling or disabling blocks of logic internal to the Main.cpp based on the same flag won't be especially difficult. I'd prefer to see a large block of common code for starting an app, with a couple of context-sensitive modifications for console vs GUI apps, rather than completely standalone template.

However, the use of a different subsystem suggests that a different binary will be needed in the app template - if that's the case, I'd vastly prefer to extract the binary from the template as described by #1523, and then use that same capability for the macOS app template as well.

[JPHutchins]

starting a GUI app from a command shell), you do have a console

In my limited testing, a briefcase app started from a windows console will not attach to that console. As I understand it, the reason is the use of the subsystem:window instead of subsystem:console. A GUI app using subsystem:console will attach to the console if launched from one. I am not sure what happens when it's double-clicked - we wouldn't want it to open a background terminal.

rather than needing to wrangle a log file

CLI applications that I work on typically have a log file in addition to console output. It's really no different than a website or native application. Storing verbose log files allows for a clean UX + clues for helping users.

The "verbose pre-start" output isn't a problem, but isn't really needed either.

IMO, "meta messages from the briefcase wrapper" can 1) go to a log file or 2) be removed from "release builds" entirely. Developers of CLI applications expect that the UX of their application is identical during development and release. Briefcase should be transparent to users; that is, regardless of whether the user installs the application via pip, pipx, clones the repo, or uses a briefcase package, the experience should be the same. Though I am a big fan of how briefcase removes the full paths from the back trace 😎!

console output is preferred in all cases, because GUI popups are problematic

Agreed - the crash dialog abstraction is replaced by an in-console crash dialog.

I'm glad I'm not the only person who had fun wrestling this :-)

I still don't understand why I can't get it to work. 😢

However, from what I can make out, a lot of the changes you've made to the template revolve around finding a clean abstraction of logging so that log file output is preserved. Unless I'm missing something, adding handling for logging in addition to console output isn't really needed here - you can get the same logs by capturing the console output into a file.

Have I misunderstood the nature of the changes you've made here?

This is accurate, but I would summarize the choices as 1) use a log file for briefcase wrapper "meta logs" or 2) suppress logging from the briefcase wrapper entirely. In order to preserve the developers intended UX.

Templates can contain logic, so switching between subsystem:windows and subsystem:console based on a Briefcase configuration flag won't be hard to add. Even enabling or disabling blocks of logic internal to the Main.cpp based on the same flag won't be especially difficult. I'd prefer to see a large block of common code for starting an app, with a couple of context-sensitive modifications for console vs GUI apps, rather than completely standalone template.

It's out of scope, but I think that templates are not the right tool in this case. I like CMake for managing cross-platform C/C++ build systems. Well, "like" might not be the right word... 🤣 but it is the lingua franca. One approach would be to define an Operating System Abstraction Layer, "OSAL", header which would be implemented by source files for each OS. In this way, one common "main.c" could be used for Windows/MacOS/Linux. The Python C API seems to be OS agnostic, so no issue there 🤞.

However, the use of a different subsystem suggests that a different binary will be needed in the app template - if that's the case, I'd vastly prefer to extract the binary from the template as described by #1523, and then use that same capability for the macOS app template as well.

Agreed! But I think we might be headed towards the GUI apps using subsystem:console if the current behavior is not intended?

[freakboy3742]

starting a GUI app from a command shell), you do have a console

In my limited testing, a briefcase app started from a windows console will not attach to that console.

I'm not sure if we're using different terminology here (or if I'm massively mis-remembering what happens here), but that contradicts my understanding of what the existing stub app does. AFAIR, the reason the current console management code exists is so that briefcase run will display the app logs in the terminal window. Without it, the app goes silent when run under briefcase.

As I understand it, the reason is the use of the subsystem:window instead of subsystem:console. A GUI app using subsystem:console will attach to the console if launched from one. I am not sure what happens when it's double-clicked - we wouldn't want it to open a background terminal.

It's been a long time since I've worked on this code - but it was my understanding that this was the reason the windows subsystem is selected - if you use the console subsystem, the app will create a console if there isn't one visible. We should definitely confirm this, though.

rather than needing to wrangle a log file

CLI applications that I work on typically have a log file in addition to console output. It's really no different than a website or native application. Storing verbose log files allows for a clean UX + clues for helping users.

Ok - this might be a Windows vs Unix discrepancy. I haven't used Windows as a daily driver since the early 2000s, and even then it was essentially being used as a mechanism to start cygwin, so my understanding of "expected" Windows idiom may be off. From the Unix experience, I wouldn't expect a console-based app to generate a log - if you want the logs, you tee/concat the console output.

The "verbose pre-start" output isn't a problem, but isn't really needed either.

IMO, "meta messages from the briefcase wrapper" can 1) go to a log file or 2) be removed from "release builds" entirely.

Completely agreed - My original thought about this was that the stub app would honour a BRIEFCASE_VERBOSE environment variable that would enable/disable this pre-start output; Briefcase can easily turn this option on by default, but it won't be there for a normal user.

Though I am a big fan of how briefcase removes the full paths from the back trace 😎!

Yeah - that was a bit of basic developer ergonomics. On macOS in particular, the paths become unmanageable really quickly; and given that Python interpreter is isolated, you're not gaining any extra information by replicating the app prefix as part of stack traces etc.

Templates can contain logic, so switching between subsystem:windows and subsystem:console based on a Briefcase configuration flag won't be hard to add. Even enabling or disabling blocks of logic internal to the Main.cpp based on the same flag won't be especially difficult. I'd prefer to see a large block of common code for starting an app, with a couple of context-sensitive modifications for console vs GUI apps, rather than completely standalone template.

It's out of scope, but I think that templates are not the right tool in this case. I like CMake for managing cross-platform C/C++ build systems. Well, "like" might not be the right word... 🤣 but it is the lingua franca. One approach would be to define an Operating System Abstraction Layer, "OSAL", header which would be implemented by source files for each OS. In this way, one common "main.c" could be used for Windows/MacOS/Linux. The Python C API seems to be OS agnostic, so no issue there 🤞.

The problem is that the Python C API is literally the only part that is OS agnostic. The rest of the stub apps for iOS, Android, macOS, Windows and GTK are all radically different. There's almost nothing re-usable, other than the raw Python startup code. Even then, there's differences in language and data types - C, Objective C, and Managed C++ al lhave very different representations of String and wide-char data types, so there's very little that can be shared beyond the basic "shape" of the stub app.

The use of templates also overlaps with a piece of intended functionality - we present a "native" IDE experience for projects that actually want to develop a complex app that integrates with the rest of the platform developer ecosystem. Think an app developer that wants to drop a complex macOS framework or Windows assembly into their app so that their Python code can access it. It might be possible to use CMake to do this, but it won't be a "native" developer experience.

Lastly, CMake isn't an option at all for Android or iOS - they're pretty much forced into Gradle and Xcode, respectively.

However, the use of a different subsystem suggests that a different binary will be needed in the app template - if that's the case, I'd vastly prefer to extract the binary from the template as described by #1523, and then use that same capability for the macOS app template as well.

Agreed! But I think we might be headed towards the GUI apps using subsystem:console if the current behavior is not intended?

I'm always up for simplification - if we can use a single stub app, that would be great. However, we need to confirm whether we can use the console subsystem for Windows apps first.

[JPHutchins]

It's been a long time since I've worked on this code - but it was my understanding that this was the reason the windows subsystem is selected - if you use the console subsystem, the app will create a console if there isn't one visible. We should definitely confirm this, though.

I think that you'll need to test this - it sounds like it may be behaving as intended and I am confused by differences between briefcase run vs running the executable from the console. Also, no reason to support launching GUIs from command line in Windows, that is not a common occurrence - and I don't think users would necessarily expect stdout - IDK what they'd expect🤣.

Ok - this might be a Windows vs Unix discrepancy. I haven't used Windows as a daily driver since the early 2000s, and even then it was essentially being used as a mechanism to start cygwin, so my understanding of "expected" Windows idiom may be off. From the Unix experience, I wouldn't expect a console-based app to generate a log - if you want the logs, you tee/concat the console output.

I think it must be a "me" thing. I like logs, but I don't like cluttering up UX. Briefcase shouldn't care since the developer can choose where/how to implement application logs. For a CLI app, I would support dropping all "briefcase meta logs" other than stderr. It would vastly simplify the code.

Describing why it would be hard to make generic source code for the OS templates is a great reminder of the utility of projects like Briefcase! And yes I am scared off from attempting anything, thanks for helping me help myself 😭.

[freakboy3742]

It's been a long time since I've worked on this code - but it was my understanding that this was the reason the windows subsystem is selected - if you use the console subsystem, the app will create a console if there isn't one visible. We should definitely confirm this, though.

I think that you'll need to test this - it sounds like it may be behaving as intended and I am confused by differences between briefcase run vs running the executable from the console.

briefcase run is effectively a wrapper around subprocess.Popen(), with stderr piped to stdout, and stdout piped to the Briefcase console. So - there's a subtle difference in handling of stderr, but output to either from the underlying process should be displayed.

I've just done an audit of the current behavior. The existing template (using the Windows subsystem), unmodified, will:

1. If started from Briefcase:
   • No log file is produced
   • stdout and stderr go to the console
   • Exceptions appear in a GUI, and the console.
2. If started from the command line as en executable (i.e., path/to/binary.exe)
   • No log file is produced
   • stdout and stderr don't appear anywhere
   • Exceptions appear in a GUI
3. If started by double clicking on icon
   • A log file is produced that contains the preamble
   • Stdout and stderr appear in the log file
   • Exceptions appear in the log file, and in the log file.

Of these, case (2) is a bit of an edge case we don't really care about for GUI apps. The real use cases are (1) (debugging in Briefcase) and (3) (a packaged and deployed app); and those are currently behaving as intended.

(There's maybe a bug to be resolved with the final exception content being printed twice - I think that's crash_dialog() and PyErr_Print() double handling the crash condition - but I'd call that a cosmetic issue - the log information is there, it's just duplicated)

However, for a CLI app, cases (1) and (2) are what we care about. In those cases:

• stdout and stderr should always be visible in the console
• the GUI popup should never appear
• The use of a log file is debatable; I could go either way, depending on how complex the implementation is.
• If a log file is produced, it should ideally only be generated when not running through Briefcase (although it's not obvious how you'd confirm that condition, other than an environment variable that is only set by Briefcase)

If you modify the existing template to just use the console subsystem, the behavior is the same for all three:

• stdout only ever goes to the log file
• stderr only ever goes to the console
• exception output is displayed in both the console and with a GUI popup
• If the app is started by double clicking on an icon, a popup console window is displayed.

However, if you make the following three changes to the template:

• Change to using the Console subsystem
• Comment out the AttachConsole() handling (L55-94)
• Comment out the dialog display code in crash_dialog() (L331-339)

I think you get the desired behavior, with the "no log file" option. Those changes could easily be accommodated with {% if cookiecutter.console_app %} clauses in the Main.cpp template.

I also tried using your template to get a comparison. There's a bug in the main.cpp that is hard coding Python 3.12; but once I fixed that, in every mode of execution (briefcase, command line, icon):

• A log file is always produced that contains the preamble
• stdout and stderr go to the console, but not the log file
• Exceptions go to the console and the log file

That works, but means the log file isn't really all that useful, as it only ever contains the preamble, and just the crash exception (but not any logging context leading up to that crash).

Also, no reason to support launching GUIs from command line in Windows, that is not a common occurrence - and I don't think users would necessarily expect stdout - IDK what they'd expect🤣.

Agreed - this is effectively the case (2) with the use of the Windows subsystem from above.

Ok - this might be a Windows vs Unix discrepancy. I haven't used Windows as a daily driver since the early 2000s, and even then it was essentially being used as a mechanism to start cygwin, so my understanding of "expected" Windows idiom may be off. From the Unix experience, I wouldn't expect a console-based app to generate a log - if you want the logs, you tee/concat the console output.

I think it must be a "me" thing. I like logs, but I don't like cluttering up UX. Briefcase shouldn't care since the developer can choose where/how to implement application logs. For a CLI app, I would support dropping all "briefcase meta logs" other than stderr. It would vastly simplify the code.

I think the important differentiation here is "Briefcase meta logs" and "normal operation logs". I agree that the briefcase meta logs shouldn't be visible. The fact that they're visible is not an issue with GUI apps because you're either debugging (running from Briefcase) where it's not a problem to see them; or you're activating from a GUI, where you can't see them because there's no console, but they're in the log file where they might be useful diagnostic detail.

The console app case makes it slightly more difficult, because it might be helpful to see them if you're debugging a problem with the stub, but you definitely don't want to see them in the console during normal operation.

I'd suggest that outputting any meta content is something that should be controlled by an environment variable; if you set BRIEFCASE_VERBOSE=1, you see the meta content (and Briefcase can set that value as part of starting the app in run, or in run if verbosity is enabled); if you don't, you just see the app's output as defined by the developer.

Beyond that, I agree that the developer can choose what to do with logs. The stub's logging behaviour is only required because for GUI apps, stdout/stderr is a black hole without them. However, for a CLI app, stdout/stderr does exist. If the developer wants to augment that with a log file, then they can; but the default behavior is entirely usable for diagnostic purposes.

[JPHutchins]

However, if you make the following three changes to the template:

Change to using the Console subsystem Comment out the AttachConsole() handling (L55-94) Comment out the dialog display code in crash_dialog() (L331-339)

This seems like the way to go!

[Jzhenli]

So briefcase will support console app for windows? @freakboy3742

[freakboy3742]

@Jzhenli It doesn't currently; this ticket is open because someone has requested this feature. Recent comments on this thread seem to have resolved many of the technical issues with a Windows implementation of this feature, but someone still needs to implement the feature.

[Jzhenli]

ok, got it, thanks for your effort on this feature.@freakboy3742 @JPHutchins

[rmartin16]

So - once upon a time, we did use GitHub artefacts. But then we managed to hit download limits during a PyCon sprint when we had 25 or so developers simultaneously downloading the same artefact. Moving to S3 hosting avoided this problem.

a relevant datapoint I hit today.... 😞

Run actions/setup-python@v5.1.0
Installed versions
  Version 3.10 was not found in the local cache
  Version 3.10 is available for downloading
  Download from "https://github.com/actions/python-versions/releases/download/3.10.11-4626646535/python-3.10.11-darwin-arm64.tar.gz"
  Unexpected HTTP response: 429
  Waiting 13 seconds before trying again
  Unexpected HTTP response: 429
  Waiting 15 seconds before trying again
  Received HTTP status code 429.  This usually indicates the rate limit has been exceeded
  Error: Unexpected HTTP response: 429

source: https://github.com/beeware/briefcase/actions/runs/8928248580/job/24523637866?pr=1768

[freakboy3742]

Oh good... a rate limit on obtaining GitHub Actions. That's not going to cause problems at all...