Build system revamp - Githubissues

cmsj commented 2 years ago

This is likely to be of most interest to @latenitefilms

While working on adding Shortcuts support to Hammerspoon, I have needed to do local builds that are fully signed, but not notarized and without symbols uploaded to Sentry, but use the -dev entitlements. This has been..... challenging, and frustrating.

Right now we have a weird mix of Xcode build phase scripts, librelease.sh, the Makefile and github actions helper scripts.

I think I'm going to rip all of this apart and refactor it down to:

Remove the makefile because it prevents environment variables being preserved into xcodebuild (and thus into the build phase scripts)
Rework the build-phase scripts so they are more heavily influenced by environment variables, but by default will focus mostly on just making a usable build from Xcode
Try to move things out of build-phase scripts and into xcconfig files where possible (e.g. signing could do that)
Rework the release script into something that stops trying to guess what it should be doing and just adds command line arguments for the various steps it does
Separate release.sh into build.sh and release.sh
Rework the github actions scripts to fit with the above

Arguments I'm currently thinking need to be on build.sh are:

--scheme (Hammerspoon or Release) }
--configuration (Debug or Release) }- might want to even revisit if these two are still worth having vs xcconfig files.
--sign (which requires the presence of a signing token file)
--entitlements (Hammerspoon or Hammerspoon-dev)
--docs (build only the docs)

latenitefilms commented 2 years ago

Sounds great!

You can find CommandPost's current release script here. We've added a few extra things in librelease.sh, as we release in a DMG. If you could consider making it easy to add additions like this, that would be amazing! Better yet if there's a way we can just use Hammerspoon's release scripts, but trigger the CommandPost changes via arguments?

Another thing that could be handy is a --installDependancies option, as there's quite a few now with Hammerspoon.

Homebrew: /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

greadlink: brew install coreutils

jq: brew install jq

pip: python -m ensurepip --upgrade pip3 install -r requirements.txt

xcbeautify: brew install xcbeautify

gawk: brew install gawk

sentry-cli: brew install getsentry/tools/sentry-cli

cmsj commented 2 years ago

@latenitefilms yeah I like the idea of a dependency installer.

Progress Report: I've gotten pretty close to finished with moving all of our build settings out of Xcode's extensive UI and into xcconfig files. This isn't strictly necessary for this process, but it makes them much easier to manage, and means that all the extensions can share one or two common build settings files instead of each having their own mostly-identical build settings in Xcode's UI. With a bit more refactoring and use of #include statements we should be able to get to a situation where anyone can build in Xcode, or with xcodebuild, I can generate signed release builds and CommandPost can do the same, with virtually no changes between repos.

Edit: I'm working in the convert-to-xcconfig branch in cmsj/hammerspoon

cmsj commented 2 years ago

Another progress report:

Code signing is now fully native Xcode signing, no more script required
Makefile is fully replaced by build.sh
Docs can be fully-yet-granularly produced by build.sh
Brew/pip dependencies can be installed by build.sh (although I'm not going to install Brew for people)

cmsj commented 2 years ago

Remaining tasks:

[x] Implement Notarization in build.sh
[x] Implement dSYM uploading with Sentry in build.sh
[x] Implement Releasing in build.sh
[x] Check if we still really need the Copy LuaSkin.framework to build/bundle build phase
[x] Convert hs build settings to an xcconfig
[x] Add support for running tests, for CI
[x] Clean out unused remnants of librelease.sh
[x] Update CI scripts to use build.sh
[x] Audit which token files are no longer required (certainly the signing ones aren't)
[ ] Port over dpkg building from CommandPost release.sh
[ ] Help CommandPost folk move to build.sh (e.g. check that they can correctly override signing config without changing any of our files)
[x] Figure out where ld: directory not found for option '-F/Users/cmsj/hacking/hammerspoon/hammerspoon/build/Hammerspoon/Build/Products/Debug' is coming from
[x] Rework how extensions are named (ie no more faceless init.lua and internal.so files)
[x] Ensure all instances of "FIXME: 2972" are taken care of
[x] Figure out a replacement for rebuild.sh

cmsj commented 2 years ago

@latenitefilms so, the PR for this is now in a state where it's passing tests about as well as they were passing before (ie websocket and window are... unreliable). I honestly don't know yet what else, if anything, is broken. Realistically the only way to find out is to move it towards release.

However, if I merge it, I am signing you up for a pretty substantial amount of work - I have no idea how you would be able to merge it, but I'm willing to help and to guide y'all through the changes it brings, and we can get the dmg building integrated into our build.sh so you don't need to carry that as a separate delta.

latenitefilms commented 2 years ago

The way CommandPost currently works is that we have CommandPost-App repo, which is our fork of Hammerspoon - which we try and keep mostly in sync. Then we have our main CommandPost repo, which contains all our Lua code and any pre-built third party Objective-C extensions, as well as our build scripts. During the build process, we basically copy all the CommandPost files into the CommandPost-App application bundle.

So it's possible, we could just ignore your new build scripts and just re-work our existing build scripts to take into account any changes in the Xcode project itself?

Although, I'm really hoping we can just make use of 99% of your build process, and just disable some things (such as IRC posting, etc), and add some additional steps, such as copying files from the CommandPost repo, signing our uninstall too, and bundling up everything into a DMG at the very end.

One thing to note, is that I actually use our release script for all builds - even when just testing things.

Because our release script is basically just your original one, with some things commented out and a couple of additions, it should be pretty easy for you to understand and follow.

I think really, apart from changing some things from "Hammerspoon" to "CommandPost" (maybe we could put the application name as a variable?), these are really only the two major additions:

function build_uninstall() {
    echo "Building Uninstall Script..."
    rm -rf ../CommandPost/scripts/inc/uninstall/Uninstall\ CommandPost.app
    osacompile -x -o ../CommandPost/scripts/inc/uninstall/Uninstall\ CommandPost.app ../CommandPost/scripts/inc/uninstall/Uninstall\ CommandPost.scpt
    cp ../CommandPost/scripts/inc/uninstall/applet.icns ../CommandPost/scripts/inc/uninstall/Uninstall\ CommandPost.app/Contents/Resources/applet.icns
    xattr -cr ../CommandPost/scripts/inc/uninstall/Uninstall\ CommandPost.app
    codesign --verbose --force --deep --options=runtime --timestamp --entitlements ../CommandPost/scripts/inc/uninstall/Uninstall\ CommandPost.entitlements --sign "Developer ID Application: LateNite Films Pty Ltd" ../CommandPost/scripts/inc/uninstall/Uninstall\ CommandPost.app
    codesign -dv --verbose=4 ../CommandPost/scripts/inc/uninstall/Uninstall\ CommandPost.app
}

function build_dmgcanvas() {

  echo "Remove Old DMG..."
  rm -f "../CommandPost-Releases/${VERSION}/CommandPost_${VERSION}.dmg"

  echo "Building DMG..."
  mkdir -p "../CommandPost-Releases/${VERSION}"
  /Applications/DMG\ Canvas.app/Contents/Resources/dmgcanvas "../CommandPost/scripts/inc/dmgcanvas/CommandPost.dmgCanvas" "../CommandPost-Releases/${VERSION}/CommandPost_${VERSION}.dmg" -setFilePath CommandPost.app "${HAMMERSPOON_HOME}/build/CommandPost.app" -setFilePath "Uninstall CommandPost.app" "../CommandPost/scripts/inc/uninstall/Uninstall CommandPost.app" -setFilePath "Applications" "/Applications/"

  if [ ! -f "../CommandPost-Releases/${VERSION}/CommandPost_${VERSION}.dmg" ]; then
    fail "DMG Creation Failed"
  else
    echo "DMG Creation Successful"
  fi
}

I'm happy to take your guidance on best way to tackle, but if you're wanting to just merge this and see what happens, I'm totally ok with that.

latenitefilms commented 2 years ago

Oh, I guess it's worth noting, that some of the tweaks we make are in Xcode itself - not just the build scripts - that's where we're copying across files (search for "Copy CommandPost Files" in our Xcode project).

cmsj commented 2 years ago

So, the Xcode build settings stuff is going to be tedious, but it has a really nice view that lets you figure out exactly what is going on - click into the CommandPost target, go to Build Settings at the top, then make sure "Customised" and "Levels" are selected below. That shows you exactly where each build setting is coming from.

What I did was go through every single target and figure out where a non-default setting was coming from, and then put it into the equivalent xcconfig file and delete it from Xcode's view. Takes a while to do, but works really well. What's particularly cool about xcconfig files is that they can do conditional #include's, so we can have, in any of the xcconfig files that we need to, something like #include? "Foo-Downstream.xcconfig" and it will ignore that missing file in Hammerspoon, but you can place your build settings tweaks in a Foo-Downstream.xcconfig file and they'll get picked up 😀

I would be very happy to implement something similar in build.sh - it would be fantastic if we can get to a point where you can put all of your build/release customisations in separate files and not need to modify ours.

latenitefilms commented 2 years ago

Actually, I think the only addition we made in Xcode was copying the CommandPost files, which could/should probably only happy in the build scripts anyway.

latenitefilms commented 2 years ago

FYI - I attempted to merge your branch into a new brand in CommandPost-App, so you can see the changes:

https://github.com/CommandPost/CommandPost-App/tree/hs-build-revamp

latenitefilms commented 2 years ago

@cmsj - It looks like window_layout.lua has made it's way to the root folder in your branch?

cmsj commented 2 years ago

@latenitefilms I checked out your hs-build-revamp branch and it seems like it builds correctly, from a first glance :)

cmsj commented 2 years ago

@cmsj - It looks like window_layout.lua has made it's way to the root folder in your branch?

Thanks, fixed

cmsj commented 2 years ago

I think really, apart from changing some things from "Hammerspoon" to "CommandPost" (maybe we could put the application name as a variable?), these are really only the two major additions:

Done (436b5908). You'll want a file one level above the hammerspoon repo checkout called Hammerspoon-Downstream.xcconfig:

❯ cat ../Hammerspoon-Downstream.xcconfig
PRODUCT_NAME=CommandPost

and an environment variable for the build process:

APP_NAME=CommandPost ./scripts/build.sh build

and that should do it. I imagine there will be some bits I missed that we'll need to shake out, but I'm definitely committed to making this work if at all possible.

FWIW, that downstream xcconfig is also where you'd set the signing info for release builds, if we can get everything of yours integrated into the main build scripts:

CODE_SIGN_IDENTITY = Developer ID Application
DEVELOPMENT_TEAM = VQCYSNZB89

cmsj commented 2 years ago

So I'm pretty much ready to merge the PR now, but I'll hold for a day or two in case anyone has further feedback

cmsj commented 2 years ago

Apologies all, there will probably be a bit of noise on master for a little while until I get the per-commit signed builds working again.

cmsj commented 2 years ago

Ok, the noise on master should be done now, the build system appears to be stable and per-commit signed builds are being generated correctly. Phew!

latenitefilms commented 2 years ago

@cmsj - Woohoo! Nice work!

Is this now at a stage that I should attempt to merge into CommandPost-App and modify to make work?

Hammerspoon / hammerspoon

Build system revamp #2972