search

beeware / briefcase

Tools to support converting a Python project into a standalone native application.
https://briefcase.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
2.61k stars 366 forks source link

Add support for console apps #1781

Closed freakboy3742 closed 4 months ago

freakboy3742 commented 4 months ago

Adds support for packaging console apps with Briefcase.

In order to support console apps, it is also necessary to introduce a .pkg app packaging format for macOS. This generates an installer that is able to perform the post-installation step of adding a symlink to /usr/local/bin. The PKG format is the default and only packaging format allowed for macOS console apps.

This PR also renames the macOS app packaging format to zip - this is for consistency with Windows, to match the actual output format, and to avoid confusion about "packaging an app as an app that is a zip" descriptions.

To facilitate easier refactoring of the PKG format, the handling of signing identities on macOS has been modified to introduce a single SigningIdentity object that incorporates the id, identity name and team ID into a single object.

To enable generation of product GUIDs on Windows, a new dns_uuid5 template filter has been added. This will convert a domain name like feature.appname.example.com into a GUID using the UUID5 algorithm in the DNS namespace.

When running console apps (with dev or run), any output streaming is disabled. This is so that the user gets raw, unbuffered stdout - and especially stdin - access. However, when running a console app in test mode, streaming is enabled, as we need to be able to process test output looking for the sentinel.

This branch also adds a BRIEFCASE_DEBUG environment variable to the runtime environment if running in debug (i.e., -vv mode. One of the major changes to the stub binaries is that they no longer generate the "preamble" console output (detailing the PYTHONPATH etc) unless BRIEFCASE_DEBUG is enabled. This allows us to generate console apps that only generate output that the user actually requests, while retaining the ability to get deep debugging information if necessary.

Lastly, a new bootstrap (and a testing bootstrap) for console apps has been added.

This code can be tested with the macOS Xcode backend, the windows VisualStudio backend, the linux System backend the Flatpak backend. It requires template modifications to generate console-compatible apps. To test this branch, add:

template_branch = "console-app"
console_app = True

to your pyproject.toml, and run briefcase with:

$ briefcase run macOS Xcode
$ briefcase package macOS Xcode

The macOS App template and Windows App template have been partially updated. The logic to generate and handle 2 stub binaries has been added; however, those PRs cannot be completed until the macOS Xcode and Windows VisualStudio template changes have been merged. Inadvertently, this also fixes #1729; by moving the process of renaming the stub from the create step to the build step, we're able to clearly identify if the a build has been performed.

The landing strategy for this:

As an aside: This PR increases the need to address #993, as the app repos have just doubled the number of binaries that are stored. See #1849 for the in-progress work to extract the stub binaries.

Update: 4 Jun

The Xcode template updates have been landed, so briefcase run macOS Xcode and briefcase run Windows VisualStudio will work without the template_branch setting.

The app templates have been updated with new stub binaries, so briefcase run will work as long as the template_branch = "console-app" setting exists.

PR Checklist:

michelcrypt4d4mus commented 4 months ago

Just poking around here for a bit and sorry if this is a dumb question as i've never really looked at the briefcase code base before but let's say i was trying to create an app with a pre or post-install script - it's not immediately clear to me how to do that. do I just create a preinstall and/or postinstall script somewhere (where?) in my project?

freakboy3742 commented 4 months ago

let's say i was trying to create an app with a pre or post-install script - it's not immediately clear to me how to do that. do I just create a preinstall and/or postinstall script somewhere (where?) in my project?

This isn't a customisation point that this PR supports - but if you were to use a custom app template (e.g.,, a derivative of the Xcode template that is required to use this branch), you could customise the existing postinstall, or add an additional preinstall script. Those two names are "special" to the pkgbuild tool; the location in the scripts folder is hard-coded by Briefcase.

michelcrypt4d4mus commented 4 months ago

ah ok, i hadn't seen that repo before. why not let the user configure their own postinstall script location in pyproject.toml? seems like that could be a useful feature.

michelcrypt4d4mus commented 4 months ago

i tried installing this branch of briefcase as per the docs and added

template_branch = "console-app"
console_app = True

to my pyproject.toml. briefcase run macOS Xcode worked fine but briefcase package macOS Xcode failed:

[myapp] Building PKG...
Installing license... errored

Log saved to /Users/uzor/workspace/my_app/briefcase_app/myapp/logs/briefcase.2024_05_21-17_35_53.package.log

Traceback (most recent call last):
  File "/Users/uzor/workspace/my_app/briefcase_app/myapp/.venv/bin/briefcase", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/uzor/workspace/my_app/lib_repos/briefcase/src/briefcase/__main__.py", line 29, in main
    command(**options)
  File "/Users/uzor/workspace/my_app/lib_repos/briefcase/src/briefcase/commands/package.py", line 151, in __call__
    state = self._package_app(
            ^^^^^^^^^^^^^^^^^^
  File "/Users/uzor/workspace/my_app/lib_repos/briefcase/src/briefcase/commands/package.py", line 99, in _package_app
    state = self.package_app(app, **full_options(state, options))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/uzor/workspace/my_app/lib_repos/briefcase/src/briefcase/platforms/macOS/__init__.py", line 879, in package_app
    self.package_pkg(
  File "/Users/uzor/workspace/my_app/lib_repos/briefcase/src/briefcase/platforms/macOS/__init__.py", line 932, in package_pkg
    (installer_path / "resources").mkdir(exist_ok=True)
  File "/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/pathlib.py", line 1311, in mkdir
    os.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/uzor/workspace/my_app/briefcase_app/myapp/build/myapp/macos/xcode/installer/resources'

because there's no installer/ dir in macos/xcode/. is it an issue that i didn't originally briefcase create with these settings or something?

and while i'm here - what does it mean to do briefcase run macOS Xcode vs. briefcase run macOS app? I would have assumed that the Xcode option just tells briefcase to create a scaffolding xcodeproj etc, not actually run anything.

michelcrypt4d4mus commented 4 months ago

i think you may just need to change the line

(installer_path / "resources").mkdir(exist_ok=True)

to add the parents=True option?

edit: changing that line fixed my issue.

(installer_path / "resources").mkdir(exist_ok=True, parents=True)
freakboy3742 commented 4 months ago

If you generated the project before you added the template_branch setting, then that will be the cause of the problem. The template is only used by the create command; once the template has been rolled out, the template isn't used again.

The difference between macOS Xcode and macOS app is that the Xcode template contains the full Xcode definition for building the stub app, and the app template contains a pre-compiled binary. Most projects can just use the pre-compiled binary, but obviously you need to generate the pre-compiled binary from somewhere - the Xcode project is also used to generate that binary.

michelcrypt4d4mus commented 4 months ago
  1. what's the best practice for recreating my app with a different template? Right now I'm looking at moving pyproject.toml out of the way temporarily, running briefcase create, and then moving it back... but that seems a bit fraught.
  2. re: macOS Xcode and macOS app is the right way to think about it that they are basically the same but macOS Xcode leaves behind more artifacts in the form of the full xcodeproj etc?
michelcrypt4d4mus commented 4 months ago

nevermind - i sorted this out with briefcase create macOS Xcode. might want to update the PR description to add that step if other people might be testing this.

michelcrypt4d4mus commented 4 months ago

is there any mechanism for injecting custom build options into project.pbxproj? for instance i have customized the template with INFOPLIST_KEY_LSBackgroundOnly = YES; in the buildSettings:

0D354FDD2551BFBD119178D1 /* Debug */ = {
            isa = XCBuildConfiguration;
            buildSettings = {
                ARCHS = "$(ARCHS_STANDARD)";
                ASSETCATALOG_COMPILER_APPICON_NAME = "App Server";
                ASSETCATALOG_COMPILER_GLOBAL_ACCENT_COLOR_NAME = AccentColor;
                CLANG_CXX_LANGUAGE_STANDARD = "gnu++0x";
                CLANG_CXX_LIBRARY = "libc++";
                CODE_SIGN_ENTITLEMENTS = AppServer/app.entitlements;
                CODE_SIGN_IDENTITY = "-";
                CODE_SIGN_STYLE = Automatic;
                COMBINE_HIDPI_IMAGES = YES;
                ENABLE_HARDENED_RUNTIME = YES;
                GCC_C_LANGUAGE_STANDARD = gnu99;
                INFOPLIST_FILE = AppServer/Info.plist;
                INFOPLIST_KEY_LSBackgroundOnly = YES;
                LD_RUNPATH_SEARCH_PATHS = (
                    "$(inherited)",
                    "@executable_path/../Frameworks",
                );
                MACOSX_DEPLOYMENT_TARGET = 11.0;
                PRODUCT_BUNDLE_IDENTIFIER = com.appserver.app;
                PROVISIONING_PROFILE_SPECIFIER = "";
                STRIP_INSTALLED_PRODUCT = NO;
            };
            name = Debug;
        };

If I could somehow specify that in the briefcase config so that briefcase create didn't overwrite it that would be awesome.

freakboy3742 commented 4 months ago

nevermind - i sorted this out with briefcase create macOS Xcode. might want to update the PR description to add that step if other people might be testing this.

The PR description does say this:

Screenshot 2024-05-22 at 9 16 00 AM

is there any mechanism for injecting custom build options into project.pbxproj? for instance i have customized the template with INFOPLIST_KEY_LSBackgroundOnly = YES; in the buildSettings:

There's no mechanism to directly manipulate the project.pbxproj file; however, you can inject values into the Info.plist using the info key in your configuration. In your case, it sounds like adding info."LSBackgroundOnly" = true would do what you want.

mhsmith commented 4 months ago
  • Do a preliminary review of this Briefcase branch
  • Test and land the non-app templates (except for Windows VisualStudio) using this branch
  • Use this branch to manually generate stubs for the two app templates

Why not VisualStudio in step 2?

freakboy3742 commented 4 months ago

Why not VisualStudio in step 2?

It's a chicken and egg thing. The VisualStudio template requires the UUID change in this PR, so we can't land the template change without also landing this Briefcase branch. The same problem doesn't exist for the Xcode template because that template doesn't depend on any new Briefcase features.

If it would help, I could extract the UUID piece into a standalone PR; it's nicely self contained, and would then allow running the Visual Studio template against a mainline Briefcase.

freakboy3742 commented 4 months ago

If it would help, I could extract the UUID piece into a standalone PR; it's nicely self contained, and would then allow running the Visual Studio template against a mainline Briefcase.

I've opened #1850 to include just the UUID piece; I've modified the landing instructions to reflect the extra step of landing that PR.

mhsmith commented 4 months ago

With the macOS app template, briefcase run worked fine, but briefcase package gave this error:

[consoletest] Building PKG...
Installing license... errored

Log saved to /Users/msmith/git/beeware/apps/consoletest/logs/briefcase.2024_06_04-21_50_50.package.log

Traceback (most recent call last):
  File "/Users/msmith/.venv/bw-38/bin/briefcase", line 8, in <module>
    sys.exit(main())
  File "/Users/msmith/git/beeware/briefcase/src/briefcase/__main__.py", line 29, in main
    command(**options)
  File "/Users/msmith/git/beeware/briefcase/src/briefcase/commands/package.py", line 151, in __call__
    state = self._package_app(
  File "/Users/msmith/git/beeware/briefcase/src/briefcase/commands/package.py", line 99, in _package_app
    state = self.package_app(app, **full_options(state, options))
  File "/Users/msmith/git/beeware/briefcase/src/briefcase/platforms/macOS/__init__.py", line 878, in package_app
    self.package_pkg(
  File "/Users/msmith/git/beeware/briefcase/src/briefcase/platforms/macOS/__init__.py", line 931, in package_pkg
    (installer_path / "resources").mkdir(exist_ok=True)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/pathlib.py", line 1288, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/msmith/git/beeware/apps/consoletest/build/consoletest/macos/app/installer/resources'
(bw-38) msmith@mbp consoletest % ll /Users/msmith/git/beeware/apps/consoletest/build/consoletest/macos/app/
total 16
drwxr-xr-x  3 msmith  staff   96 Jun  4 21:50 Console test.app
-rw-r--r--  1 msmith  staff  334 Jun  4 21:49 Entitlements.plist
drwxr-xr-x  2 msmith  staff   64 Jun  4 21:50 app_packages.x86_64
-rw-r--r--  1 msmith  staff  563 Jun  4 21:49 briefcase.toml
drwxr-xr-x  6 msmith  staff  192 Jun  4 21:50 support

I added installer_path.mkdir(exist_ok=True), but then I got this:

[consoletest] Building PKG...
Installing license... done
Copying app into products folder... done
Writing component manifest... done

>>> Running Command:
>>>     pkgbuild --root /Users/msmith/git/beeware/apps/consoletest/build/consoletest/macos/app/installer/root --component-plist /Users/msmith/git/beeware/apps/consoletest/build/consoletest/macos/app/installer/components.plist --install-location '/Library/Console test' --scripts /Users/msmith/git/beeware/apps/consoletest/build/consoletest/macos/app/installer/scripts /Users/msmith/git/beeware/apps/consoletest/build/consoletest/macos/app/installer/packages/consoletest.pkg
>>> Working Directory:
>>>     /Users/msmith/git/beeware/apps/consoletest
pkgbuild: error: Cannot write package to "/Users/msmith/git/beeware/apps/consoletest/build/consoletest/macos/app/installer/packages/consoletest.pkg". (The file “scripts” couldn’t be opened.)
pkgbuild: Reading components from /Users/msmith/git/beeware/apps/consoletest/build/consoletest/macos/app/installer/components.plist
pkgbuild: Adding component at Console test.app
>>> Return code: 1
Building app package... done

>>> Running Command:
>>>     productbuild --distribution /Users/msmith/git/beeware/apps/consoletest/build/consoletest/macos/app/installer/Distribution.xml --package-path /Users/msmith/git/beeware/apps/consoletest/build/consoletest/macos/app/installer/packages --resources /Users/msmith/git/beeware/apps/consoletest/build/consoletest/macos/app/installer/resources '/Users/msmith/git/beeware/apps/consoletest/dist/Console test-0.0.1.pkg'
>>> Working Directory:
>>>     /Users/msmith/git/beeware/apps/consoletest
productbuild: error: Specified distribution "/Users/msmith/git/beeware/apps/consoletest/build/consoletest/macos/app/installer/Distribution.xml" not found.
>>> Return code: 1
Building Console test-0.0.1.pkg... done

[consoletest] Packaged dist/Console test-0.0.1.pkg
(bw-38) msmith@mbp consoletest % echo $?
0

There are at least two problems here:

mhsmith commented 4 months ago

With the Windows app template, briefcase run and briefcase package worked fine, but after running the installer:

freakboy3742 commented 4 months ago
  • None of the things in the installer directory of the Xcode template are present in the app template, including resources/welcome.html, scripts/postinstall, and Distribution.xml.

Apologies - I had those modifications locally, but managed to not commit them. They're in the template now.

  • Two consecutive commands failed, but Briefcase continued running and returned a successful exit code.

Looks like I forgot to add check=True to the two pkgbuild and productbuild command invocations. Fixed.

With the Windows app template, briefcase run and briefcase package worked fine, but after running the installer:

  • It created a Start menu shortcut to "C:\Program Files\Jane Developer\Console test\Console test.exe", but the file is actually called consoletest.exe. Unlike on macOS, this shortcut is not useless because it would open a terminal window, so it's probably worth fixing it rather than removing it.

Agreed - I've added this to the app template, and opened beeware/briefcase-windows-VisualStudio-template#32 to add the same change to the VS project.

  • The PATH is not updated, because the .wxs file updates aren't present in the app template.

Fixed.