search

CoatiSoftware / Sourcetrail

Sourcetrail - free and open-source interactive source explorer
https://www.sourcetrail.com/
GNU General Public License v3.0
14.86k stars 1.4k forks source link

App does not launch on macOS 10.11 (El Capitan) - have Travis CI build it for you! #981

Open phil-blain opened 4 years ago

phil-blain commented 4 years ago

As I discovered in https://github.com/CoatiSoftware/Sourcetrail/issues/619#issuecomment-609498312, the currently available binary releases of Sourcetrail (2020.1.117, 2019.4.102 and 2019.4.61) all do not work on macOS 10.11. When clicked, the app appears in the Dock for a few milliseconds then disappears.

As @egraether made me realize in https://github.com/CoatiSoftware/Sourcetrail/issues/619#issuecomment-609501931, this is due to the fact that the app is built against a version of Qt > 5.11 (5.12 according to the README), and Qt 5.11 is the last release supporting El Capitan.

I was able to compile the app against Qt 5.11 using Travis CI, it can be downloaded from my fork: https://github.com/phil-blain/Sourcetrail/releases/tag/el_capitan

I will describe the steps I took if other people want to use Sourcetrail on El Capitan and don't want to just download the binary from my fork.

The plan

  1. Fork the repo
  2. If you didn't already, sign up for Travis CI and activate it for you Sourcetrail fork
  3. Replace the existing Linux build script in .travis.yml by the macOS build script below, commit, push and let Travis build the app for you!

Justification for using Travis

I first tried to download and install Qt 5.11 locally, but for an unknown reason neither Qt 5.10 nor Qt 5.11 are available on the Qt downloads archive. I also tried extracting the Qt@5.11 formula from Homebrew and using it to build Qt 5.11 and my Mac but there was an obscure compilation error that I did not want to dwelve into.

Detailed steps

1. Extracting Qt@5.11

We will use Homebrew to install Qt 5.11 in the Travis environment. The recommended way to install old versions of packages with Hombrew is brew extract.

Note: this step can be skipped if you want to use my extracted formula: https://github.com/phil-blain/homebrew-versions/blob/master/Formula/qt%405.11.rb

  1. If you don't have it installed, install Hombrew
  2. Create a tap for the formula you are about to extract: 
    gh_username=<GitHub username>
brew tap-new $gh_username/homebrew-versions
  3. Extract Qt@5.11: 
    git -C $(brew --repository homebrew/core) fetch --unshallow
brew extract qt $gh_username/homebrew-versions --version=5.11
cd $(brew --repository $gh_username/versions)

  4. Fix the extracted formula to be able to use existing bottles, i.e. apply this patch to the formula:

    diff --git a/Formula/qt@5.11.rb b/Formula/qt@5.11.rb
index 0821c5b..5fb9675 100644
--- a/Formula/qt@5.11.rb
+++ b/Formula/qt@5.11.rb
@@ -32,7 +32,17 @@ class QtAT511 < Formula
    end
  end

+
+  # force "qt" in URL for bottle download
+  # https://github.com/Homebrew/brew/issues/6059
+  def name
+    "qt"
+  end
+
  bottle do
+    # force "bottles" subdir for download
+    # https://github.com/Homebrew/brew/issues/6059
+    root_url "https://homebrew.bintray.com/bottles"
    sha256 "84a1a758d7881f9a446fd76fe4360ce7c5508c48761a74032a361996df1f79ec" => :mojave
    sha256 "ad85f42e73648dc47c8c903e0045e6b8cf62c99888144c73893a0226f2a5b51e" => :high_sierra
    sha256 "cd58fa592235c48ae050861eb3fca82ecfe54ddf103e5962b2bf9621f815f640" => :sierra
  5. commit this change: 
    git add .
git commit -m "extract qt@5.11"
  6. Create a new repository on GitHub with name homebrew-versions, add it as your origin remote and push: 
    git remote add origin git@github.com:$gh_username/homebrew-versions.git
git push -u origin HEAD

2. Transforming .travis.yml for building on macOS.

The current .travis.yml file in the repo is used to build on Linux. We will remove its content and replace it with instructions to build the app on macOS, and use Travis' integration with GitHub releases to get a downloadable disk image and zip archive.

  1. Clone your fork of Sourcetrail, then replace the content of the .travis.yml with the following:

    os: osx

osx_image: xcode9.4

language: cpp

install:
  - export HOMEBREW_VERBOSE_USING_DOTS=1
  - export HOMEBREW_VERBOSE=1
  - export HOMEBREW_NO_AUTO_UPDATE=1
  # Do not cleanup automatically
  - export HOMEBREW_NO_INSTALL_CLEANUP=1
  # Useful for Travis debug build
  - export HOMEBREW_NO_ANALYTICS=1
  - brew tap <gh_username>/homebrew-versions
  - brew install -f qt@5.11
  # Needed for bundle_install script (`convert`)
  - brew install imagemagick
  # For C/C++ support
  - brew install llvm@9
  # Make times out, let's try ninja
  - brew install ninja

script:
  - export MACOSX_DEPLOYMENT_TARGET=10.11
  - mkdir -p build/Release
  - cd build/Release
  - cmake -G "Ninja" -DQt5_DIR=$(brew --prefix qt)/lib/cmake/Qt5 -DBUILD_CXX_LANGUAGE_PACKAGE=ON -DClang_DIR=$(brew --prefix llvm@9)/lib/cmake/clang ../..
  - ninja -v
  - ./app/bundle_install.sh
  - mkdir deploy
  - mv app/Sourcetrail*.zip deploy/Sourcetrail.zip
  - mv app/Sourcetrail*.dmg deploy/Sourcetrail.dmg
  - cd ../..

    Don't forget to replace <gh_username> with your GitHub username, or mine (phil-blain) if you want to use my extracted formula.

    Notes:

    • I use osx_image: xcode9.4 (one of Travis' High Sierra images) because all Sourcetrail dependencies exists in Hombrew as binary packages for High Sierra, which speeds up the build a lot.
    • I use ninja to build since I it's faster than make.
    • We need to export MACOSX_DEPLOYMENT_TARGET=10.11 for the app to be able to run on El Capitan.

    Then, create a new branch and commit:

    git checkout -b setup-releases origin/master
git commit -am ".travis.yml: build on macOS"

  2. Then, install the Travis CI command line client to setup deployment to GitHub releases.

  3. Setup the integration.

    cd path/to/Sourcetrail 
travis setup releases --com
    • Input you GitHub username and password,
    • 'File to Upload' : enter whatever string, we will change it
    • 'Deploy only from /Sourcetrail' : 'yes'
    • 'Deploy from public-release branch?' : 'no'
    • Encrypt API key? : 'yes' (Important to encrypt your key)

    This should add a deploy section to the .travis.yml.

  4. Modify the file entry and add on.tags and skip_cleanup sections, so that the deploy section looks like this:

    deploy:
  provider: releases
  file: 
    - build/Release/deploy/Sourcetrail.zip
    - build/Release/deploy/Sourcetrail.dmg
  skip_cleanup: true
  overwrite: true
  api_key:
    secure: <encrypted key>
  on:
    tags: true
    repo: <gh_username>/Sourcetrail

    Don't forget to replace with your GitHub username.

    Commit this change:

    git commit -am ".travis.yml: setup GitHub releases"

  5. I also needed to modify the bundle_install.sh.in CMake script template so that the needed library paths for the Sourcetrail_indexer companion app are relocatable. Also, I removed interaction (read), skipped app notarization since this is not needed on El Capitan, and added a zip archive. Apply this patch:

    diff --git a/setup/macOS/bundle_install.sh.in b/setup/macOS/bundle_install.sh.in
index 15a92499..56323b2d 100755
--- a/setup/macOS/bundle_install.sh.in
+++ b/setup/macOS/bundle_install.sh.in
@@ -81,12 +81,21 @@ cp -R ../../../bin/app/data/license $RES_DIR/data/license
 mkdir -p $RES_DIR/data/cxx/include
 cp -R ../../../bin/app/data/cxx/include/* $RES_DIR/data/cxx/include

-echo -e $INFO "run macdeployqt to copy Qt Frameworks and Plugins"
+echo -e $INFO "running macdeployqt to copy Qt Frameworks and Plugins"
 $QT_DIR/bin/macdeployqt $BUNDLE_PATH

-echo -e $INFO "edit qt.conf"
+echo -e $INFO "editing qt.conf"
 sed -i '' -e 's/PlugIns/Plugins/g' $RES_DIR/qt.conf

+echo -e $INFO "App dependencies"
+otool -l $APP_PATH | grep RPATH -A2
+otool -L $APP_PATH
+
+echo -e $INFO "Indexer dependencies"
+otool -l $INDEXER_PATH | grep RPATH -A2
+otool -L $INDEXER_PATH
+
+echo -e $INFO "Making app and indexer relocatable"
 install_name_tool -delete_rpath "$CLANG_DIR/./lib" $APP_PATH
 install_name_tool -delete_rpath "$BOOST_DIR/stage/lib" $APP_PATH
 install_name_tool -delete_rpath "@executable_path/../Frameworks" $APP_PATH
@@ -97,12 +106,19 @@ install_name_tool -delete_rpath "$BOOST_DIR/stage/lib" $INDEXER_PATH
 install_name_tool -delete_rpath "$QT_DIR/lib" $INDEXER_PATH
 install_name_tool -add_rpath "@loader_path/../Frameworks" $INDEXER_PATH

+install_name_tool -change /usr/local/opt/qt/lib/QtSvg.framework/Versions/5/QtSvg @rpath/QtSvg.framework/Versions/5/QtSvg $INDEXER_PATH
+install_name_tool -change /usr/local/opt/qt/lib/QtWidgets.framework/Versions/5/QtWidgets @rpath/QtWidgets.framework/Versions/5/QtWidgets $INDEXER_PATH
+install_name_tool -change /usr/local/opt/qt/lib/QtGui.framework/Versions/5/QtGui  @rpath/QtGui.framework/Versions/5/QtGui $INDEXER_PATH
+install_name_tool -change /usr/local/opt/qt/lib/QtNetwork.framework/Versions/5/QtNetwork  @rpath/QtNetwork.framework/Versions/5/QtNetwork $INDEXER_PATH
+install_name_tool -change /usr/local/opt/qt/lib/QtCore.framework/Versions/5/QtCore  @rpath/QtCore.framework/Versions/5/QtCore $INDEXER_PATH
+install_name_tool -change /usr/local/opt/llvm/lib/libc++.1.dylib  @rpath/libc++.1.dylib $INDEXER_PATH
+
 echo -e $INFO "App dependencies"
-# otool -l $APP_PATH
+otool -l $APP_PATH | grep RPATH -A2
 otool -L $APP_PATH

 echo -e $INFO "Indexer dependencies"
-# otool -l $INDEXER_PATH
+otool -l $INDEXER_PATH | grep RPATH -A2
 otool -L $INDEXER_PATH

@@ -218,49 +234,15 @@ cp -R ../../../ide_plugins/qt_creator $PACKAGE_DIR/plugins/qt_creator/

 echo -e $SUCCESS "Created bundle successfully!";

+echo -e $INFO "Creating DMG package";

+ln -s /Applications $PACKAGE_DIR/Applications;
+hdiutil create ${PACKAGE_DIR}_macOS_64bit.dmg -volname ${PACKAGE_DIR} -srcfolder ${PACKAGE_DIR};
+rm $PACKAGE_DIR/Applications;

-while true; do
-    read -p "Do you want to create a DMG package? [y/n] " yn
-    case $yn in
-        [Yy]* )
-           echo -e $INFO "create DMG package";
-
-           ln -s /Applications $PACKAGE_DIR/Applications;
-           hdiutil create ${PACKAGE_DIR}_macOS_64bit.dmg -volname ${PACKAGE_DIR} -srcfolder ${PACKAGE_DIR};
-           rm $PACKAGE_DIR/Applications;
+echo -e $SUCCESS "Created DMG package successfully!";

-           echo -e $SUCCESS "Created DMG package successfully!";
+echo -e $INFO "Creating zip archive";
+zip -q -r -X ${PACKAGE_DIR}.zip $PACKAGE_DIR
+echo -e $SUCCESS "Created zip archive successfully!";

-           break;;
-        [Nn]* )
-           echo -e $INFO "Skip DMG package.";
-           break;;
-        * ) echo "Please answer yes or no.";;
-    esac
-done
-
-
-
-while true; do
-    read -p "Do you want to notarize the app? [y/n] " yn
-    case $yn in
-        [Yy]* )
-           echo -e $INFO "Notarize app.";
-
-           ditto -c -k --keepParent ${BUNDLE_PATH} ${BUNDLE_PATH}.zip;
-           xcrun altool --notarize-app --primary-bundle-id "com.sourcetrail" --username "egraether@coati.io"
-               --password "@keychain:sourcetrail_notarization" --file ${BUNDLE_PATH}.zip;
-           # xcrun altool --notarization-history 0 -u "egraether@coati.io" -p "@keychain:sourcetrail_notarization";
-           # xcrun altool --notarization-info <identifier> -u "egraether@coati.io" -p "@keychain:sourcetrail_notarization";
-           rm ${BUNDLE_PATH}.zip;
-
-           echo -e $SUCCESS "Notarized app successfully!";
-
-           break;;
-        [Nn]* )
-           echo -e $INFO "Skip app notarization.";
-           break;;
-        * ) echo "Please answer yes or no.";;
-    esac
-done

    Note: this patch can be applied by copying the above content, pasting it a file that you name bundle.diff, then running git apply bundle.diff.

    Then, commit it :

    git commit -am "bundle_install.sh.in: fix indexer library paths"

  6. Tag this commit and push it to your fork:

    git tag -am "Sourcetrail for El Capitan" el_capitan
git push --tags

  7. Travis CI will build the app and deploy it your fork. You can monitor the process by navigating to https://travis-ci.com/github/<gh_username>/Sourcetrail

  8. Once the build is green, navigate to https://github.com/<gh_username>/Sourcetrail/releases/tag/el_capitan, download the disk image or the zip file, install the app and enjoy !

Caveats

LouisStAmour commented 4 years ago

Due to Qt’s long-term servicing approach, Qt 5.10 and 5.11 are only available in source releases. LTS branches are 5.9, 5.12 and 5.15.

So https://download.qt.io/archive/qt/5.9/5.9.9/qt-opensource-mac-x64-5.9.9.dmg.mirrorlist is a supported copy of Qt for older MacOS.

Starting in 5.15, LTS releases will not occur after the next version increment (Qt is switching to a model similar to Java’s), which means compiling from open source or applying patches to source would be the only way to stay current on older versions of Qt. I’d be okay with the new policy if backwards compatibility were maintained, but to see that get disregarded so quickly means if a third-party developer (us) want to support older releases, we’d have to compile our own distribution of Qt.

That said, it’s possible the only incompatibility Qt has is that by default the compatible SDK define is set as 10.13 at https://doc-snapshots.qt.io/qt5-5.15/macos.html#target-platforms ... it’s thus possible to set that value to an earlier value, 10.11 for example, and then run UI test suites in 10.11 to validate the app’s functionality. To that end, older macOS support is very likely limited by what VMs we have available for automated CI testing in future, and we’d probably have to compile those older Mac VMs ourselves or simply say the builds are experimental.

egraether commented 4 years ago

Thanks for looking into this and providing this extensive description! I think it would be a good idea to move to Travis for our macOS builds, if it also works for all languages. Currently I'm building macOS builds locally on my machine, which sometimes causes some issues.

phil-blain commented 4 years ago

@egraether on your machine do you build with make, ninja or Xcode (or xcode-build) ? I was surprised that I needed to tweak bundle_install.sh.in to fix the library paths for Sourcetrail_indexer ...

egraether commented 4 years ago

Xcode

MikeMcQuaid commented 4 years ago

Thanks for looking into this and providing this extensive description! I think it would be a good idea to move to Travis for our macOS builds, if it also works for all languages. Currently I'm building macOS builds locally on my machine, which sometimes causes some issues.

Apologies for the driveby but I noticed this linked from Homebrew's issue tracker. I'd strongly recommend using GitHub Actions rather than Travis CI if you're building on macOS; the workers are significantly more powerful.

You may notice I am a GitHub employee: this is unrelated to the above; if I thought GitHub Actions was trash I wouldn't be using it for Homebrew (as I'm not paid for my Homebrew work) 😁