Config GitHub Action to enable DevOps for building/publishing CryptoShark automatically

huan commented 4 years ago

Thank @oleavr for fixing the macOS version of CryptoShard!

As I promised at https://github.com/frida/cryptoshark/issues/6#issuecomment-646946998 , I'll try to setup GitHub Action to build & publish the CryptoShark automatically with DevOps in the following weeks.

Please stay tuned!

oleavr commented 4 years ago

Yay, thanks a lot for doing that!

I just added some missing CI bits for frida-qml so future Frida releases will provide frida-qml prebuilds for:

Windows/x86
Windows/x86_64
macOS/x86_64
Linux/x86_64

(I also backfilled Windows- and Linux-binaries for the current stable release (12.11.4) so there's no need to publish a new release just to add these three new assets.)

I anticipate that using a prebuilt version won't be possible for all Linux distro builds due to different libstdc++ ABIs – the one produced by our CI is built on Ubuntu 20.04, so it requires a fairly up-to-date distro. So next I will look into adding frida-qml.pri to frida-qml, so it can be easily used as a git submodule. I will also look into making frida-qml able to use a devkit (from Frida's releases section mentioned above) so it can be built independently from Frida. (This is the direction I'd like to take the language bindings in, so we don't have to release a new version of Frida just to fix a bug in the Qml-bindings.)

oleavr commented 4 years ago

@huan I've now finished the build system refactoring. In short:

The bootstrap script does the bare minimum so Qt Creator can be used to open Cryptoshark.pro and build it. This is what developers who prefer using an IDE are meant to use.
The build script both bootstraps and builds the app. This is what a CI worker should use.
The frida-qml plugin is now a submodule, so it can be improved/updated independently of Frida.
We configure frida-qml to use a specific frida-core devkit, which bootstrap downloads and extracts.
On Windows and macOS we should be statically linking Qt. The build instructions in the README are now up-to-date and explains how to do this.

The last point will probably be tricky with existing cloud solutions as it will make the build process really long, so we will either have to figure out a caching solution for that, or use a CI we control. We could use Frida's CI for this. (We're using Buildbot.) Let me know what you think is the way to go.

huan commented 4 years ago

I'm trying to set up GitHub Actions by following the README & Issue Comment in my forked repo at https://github.com/huan/cryptoshark today.

The Good News

Our Linux, macOS, and Windows GitHub Actions all start working now, see: https://github.com/huan/cryptoshark/actions
Actions from the macOS seem finished building after 35 minutes of hard work, I'll try to find the local files to publish them to artifacts.

The Bad News

The Windows can not find build command, I'm trying to fix it. See: https://github.com/huan/cryptoshark/runs/950587359?check_suite_focus=true#step:4:7
I can not build CryptoShark on my local Ubuntu 20, the same error occurred in CI at https://github.com/huan/cryptoshark/runs/950587284?check_suite_focus=true#step:3:532 which I believe I need your advise for how to fix it:

Info: creating stash file /home/runner/work/cryptoshark/build-cryptoshark-x86_64/.qmake.stash
cd ext/frida-qml/ && ( test -e Makefile || /usr/lib/qt5/bin/qmake -o Makefile /home/runner/work/cryptoshark/cryptoshark/ext/frida-qml/frida-qml.pro CONFIG+=silent ) && make -f Makefile 
WARNING: Failure to find: plugins.qmltypes
make[1]: Entering directory '/home/runner/work/cryptoshark/build-cryptoshark-x86_64/ext/frida-qml'
/usr/lib/qt5/bin/qmake -install qinstall /home/runner/work/cryptoshark/cryptoshark/ext/frida-qml/qmldir ../../qml/Frida/qmldir
make[1]: *** No rule to make target 'plugins.qmltypes', needed by '../../qml/Frida/plugins.qmltypes'.  Stop.
make[1]: *** Waiting for unfinished jobs....
make[1]: Leaving directory '/home/runner/work/cryptoshark/build-cryptoshark-x86_64/ext/frida-qml'
make: *** [sub-ext-frida-qml-make_first] Error 2
Makefile:70: recipe for target 'sub-ext-frida-qml-make_first' failed
##[error]Process completed with exit code 2.

huan commented 4 years ago

Update: macOS worked

Good news!

The macOS version of our beautiful CryptoShark was built and published fully automatically by our DevOps scripts successfully today!

See:

The Script: https://github.com/huan/cryptoshark/blob/latest_macos/.github/workflows/macos.yml
The artifact: https://github.com/huan/cryptoshark/releases/tag/latest_macos

huan commented 4 years ago

Hi @oleavr,

Good weekend!

Now we have made the macOS working with GitHub Actions, please see my Pull Request and let me know if there's anything that needs to be tuned.

It will work for every push and release the binary in a release named latest_macos.

oleavr commented 4 years ago

Hey @huan and good weekend!

Yay, this is awesome; thank you! 🙌

Down the road for macOS it would be great to publish the .app directory (ideally inside a .dmg) by extending our build-qt script to also build macdeployqt so we can use that. Once that works we should also be able to cache the dist directory that's produced by build-qt, to speed things up sufficiently that we can run the action on every push.

oleavr commented 4 years ago

@huan We now publish a .dmg for macOS. Only two platforms to go :-)

huan commented 4 years ago

Awesome! Great to see that we can public .dmg automatically now! :+1:

I'll try to make it work under Linux in the following weeks.

oleavr commented 4 years ago

@huan I need Linux binaries myself so I'll try to add some baseline Linux support tonight – but only one distro (Ubuntu) – then I'll leave it at that and have a look at Windows.

oleavr commented 4 years ago

Finished Ubuntu 20.04 the other day, and Windows just now. So we can now finally close this 🎉

huan commented 4 years ago

Wow, what a wonderful day!

frida / cryptoshark