search

NHDaly / ApplicationBuilder.jl

[deprecated] Compile, bundle, and release julia software
MIT License
168 stars 15 forks source link

tracking: Building an application that uses `Blink.jl` #5

Open NHDaly opened 6 years ago

NHDaly commented 6 years ago

I'm pulling out this issue to track our attempt to build and distribute an App that uses Blink.jl.

Things we still need to solve:

NHDaly commented 6 years ago

@ranjanan: Regarding your comment about Blink.resources, I'm also seeing an error that's probably the same thing. It looks like the problem I'm running into now is this call to @init: https://github.com/JunoLab/Blink.jl/blob/v0.6.2/src/content/content.jl#L54-L56

Is this what you're seeing, too?:

fatal: error thrown and no exception handler available.
Base.InitError(mod=:Blink, error=Base.MethodError(f=typeof(Base.Filesystem.joinpath)(), args=(nothing, "..", "..", "res", "blink.js"), world=0x0000000000005
f00))
rec_backtrace at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
jl_throw at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
jl_method_error_bare at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
jl_method_error at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
jl_lookup_generic_ at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
jl_apply_generic at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
jl_f__apply at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
resolve at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/blink.dylib (unknown line)
macro expansion at /Users/daly/.julia/v0.6/Blink/src/content/content.jl:55 [inlined]
#23 at /Users/daly/.julia/v0.6/Lazy/src/macros.jl:338
unknown function (ip: 0x11d276a6f)
__init__ at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/blink.dylib (unknown line)
jlcall___init___6545 at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/blink.dylib (unknown line)
jl_module_run_initializer at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
_julia_init at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
julia_init at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
main at /Users/daly/src/build-jl-app-bundle.bak/./builddir/HelloBlink.app/Contents/MacOS/blink (unknown line)
NHDaly commented 6 years ago

Okay!! I think i've fixed most of the problems! I've just uploaded examples/Blink.jl, which builds into a working app!

Here is the whole command I used to build the app (including signing it so I could share it here):

$ julia ~/.julia/v0.6/ApplicationBuilder/build_app.jl \
   -R ~/.julia/v0.6/Blink/deps/Julia.app \
   -R ~/.julia/v0.6/Blink/src/AtomShell/main.js \
   -R ~/.julia/v0.6/Blink/src/content/main.html \
   -R ~/.julia/v0.6/Blink/res \
   -L ~/.julia/v0.6/HttpParser/deps/usr/lib/libhttp_parser.dylib \
   -L ~/.julia/v0.6/MbedTLS/deps/usr/lib/libmbedcrypto.2.7.1.dylib \
   --bundle_identifier "com.nhdalyMadeThis.HelloBlink" --app_version=0.1 --certificate "Developer ID Application: nhdalyMadeThis, LLC" \
   examples/blink.jl "HelloBlink"  &&  ./builddir/HelloBlink.app/Contents/MacOS/blink
NHDaly commented 6 years ago

So that's good news! The one remaining problem I'm having is that the OS seems to forget the application is open, so if you double-click it again while it's already open, a second version of the app will open and then hang forever. It even refused to Force Quit (I had to restart to get it to quit). :(

I'm assuming it's something to do with the fact that the application opens the embedded Julia.app after starting, and it's then running two Mac Applications, and this somehow bothers the OS.

Interestingly, it shows up in Activity Monitor as two different apps: blink and Julia, neither of which are its actual name, HelloBlink.app. The other test apps all show up correctly with their app bundle name. blink is the name of the executable. So that's something to look into...

NHDaly commented 6 years ago

So actually, reading a bit more, I wonder if the right way to do this for apps using Blink (ie, apps using Electron) is to utilize the already existing electron app building tools? For example: https://github.com/electron-userland/electron-builder https://medium.com/@flaqueEau/releasing-an-electron-app-on-the-mac-app-store-c32dfcd9c2bd

That is, perhaps instead of embedding a prebuilt Electron app inside our application, maybe we should be building a new Electron app which contains our compiled code?

Figuring that out seems like a fair amount of work, though....

ranjanan commented 6 years ago

@NHDaly thank you for all your work and progress. Yes, that is indeed what I was referring to when I said I ran into another problem with Blink: @init was initialising the paths at run time, so I kind of just removed @init so it basically initialised the resources dict at compile time. One other option I am considering playing with is changing the paths at run time instead of compile time like we are currently doing. Anyhoo, I managed to build my Blink app by just removing the @init.

I have made some progress on Windows, and I have a Blink app that works whenever you double click on the blink.exe. I have some code as well for bundling it all together. Would you mind if I sent a PR? The PR would also involve reorganising build_app.jl as a Julia package instead of a command line utility. Other than that, the windows version will also have the option to create an installer, which I think most people will like.

As for the electron builder stuff, that's a complete tangent. I think people only care about having their Julia code be packaged and shipped in as easy a manner as possible, and I think what we have now for Blink is good enough.

More soon, along with my PR!

NHDaly commented 6 years ago

:D That all sounds excellent! Thanks for your help! I'm excited to see a windows build!!

Anyhoo, I managed to build my Blink app by just removing the @init.

Great! I'm glad we got that figured out.

Would you mind if I sent a PR?

!! Yes please! :D

The PR would also involve reorganising build_app.jl as a Julia package instead of a command line utility.

Yeah i think that's a good idea. I've been thinking it would make sense to follow the format of https://github.com/JuliaLang/PackageCompiler.jl, where all the functionality is in src/static_julia.jl, but they still provide a command line utility which simply does arg-parsing and then calls the library: juliac.jl. But we can talk more about specific details if you send a PR over! :)

NHDaly commented 6 years ago

As for the electron builder stuff, that's a complete tangent. I think people only care about having their Julia code be packaged and shipped in as easy a manner as possible, and I think what we have now for Blink is good enough.

sounds good!

NHDaly commented 6 years ago

Hey, @ranjanan.

Are you still able to statically compile a Blink application using this package? I've found that when I try now, I can't get it to work. See this branch for the failing test: https://github.com/NHDaly/ApplicationBuilder.jl/tree/blink_bundled Specifically, here: https://github.com/NHDaly/ApplicationBuilder.jl/blob/c1e5a548949533aafd276cd8b636b1748a85311f/test/BuildApp.jl#L65

I'm testing that it is in fact working without julia installed -- and sadly, it isn't. I am currently planning on using a Blink example in my talk, but I'm sad that I can't show it fully self-contained. :( Are you able to get this to work?

The problem I'm currently stuck on is that MacroTools uses this animals_file in its __init__() method, so we cannot override it. :/

lucatrv commented 6 years ago

@NHDaly you could try to go back in history with git to find out which commit introduced the issue.

ranjanan commented 6 years ago

Hi @NHDaly, do you happen to have the error trace in a gist somewhere?

NHDaly commented 6 years ago

@lucatrv thanks that's a good idea. Sadly, the problem isn't with this package, it's that Blink and all of its dependent packages have changed. We might still be able to get it to work, but my suspicion is that it's joined GTK and TK and the other packages whose __init__ methods prevent them from being able to make a standalone app right now (because the __init__ methods reference hard-coded paths and as far as I can tell, you can't override the __init__ method in a module because it's defined and ran before you have a chance to override it. :/

So I think this class of problems can only be solved by modifying the packages themselves, and/or BinDeps, BinaryProvider, etc, which actually set those hard-coded paths.

NHDaly commented 6 years ago

@ranjanan No, i don't yet.. I'll put something together to share! :)

ranjanan commented 6 years ago

@NHDaly, I think we should be able to send PRs to make generic changes to every package we need to make them statically compilable (and therefore shippable). Packages will definitely have an interest in being shippable.

NHDaly commented 6 years ago

@ranjanan Okay, it's not a gist, but will this do? I've pushed up a commit with a new @test_broken, which shows what i'm talking about:

https://github.com/NHDaly/ApplicationBuilder.jl/commit/74d8667a63eb60b51e4b35d0cd8ef61884f95e18 Here's the Travis run: https://travis-ci.org/NHDaly/ApplicationBuilder.jl/jobs/412419989

But tl;dr, it builds Blink using the current example build file here, and then renames .julia and tries to run the resultant binary. That tests whether or not it can run without julia installed (or whether it's still depending on hard-coded paths inside the packages). This is the output when I run that manually:

17:48:32 $ mv ~/.julia ~/.julia.bak
 17:48:33 $ /var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmp9mz15V/HelloBlink.app/Contents/MacOS/blink
WARNING: redefining constant JULIA_HOME
fatal: error thrown and no exception handler available.
Base.InitError(mod=:MacroTools, error=Base.SystemError(prefix="opening file /Users/daly/.julia/v0.6/MacroTools/src/../animals.txt", errnum=2, extrainfo=nothing))
rec_backtrace at /private/var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmp9mz15V/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
jl_throw at /private/var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmp9mz15V/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
systemerror at /private/var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmp9mz15V/HelloBlink.app/Contents/MacOS/blink.dylib (unknown line)
open at /private/var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmp9mz15V/HelloBlink.app/Contents/MacOS/blink.dylib (unknown line)
open at /private/var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmp9mz15V/HelloBlink.app/Contents/MacOS/blink.dylib (unknown line)
__init__ at /private/var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmp9mz15V/HelloBlink.app/Contents/MacOS/blink.dylib (unknown line)
jlcall___init___4507 at /private/var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmp9mz15V/HelloBlink.app/Contents/MacOS/blink.dylib (unknown line)
jl_module_run_initializer at /private/var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmp9mz15V/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
_julia_init at /private/var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmp9mz15V/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
julia_init at /private/var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmp9mz15V/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
main at /var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmp9mz15V/HelloBlink.app/Contents/MacOS/blink (unknown line)
 17:48:43 $ mv ~/.julia.bak ~/.julia

You can see it's trying to open a missing file from inside MacroTools's __init__ from inside jl_module_run_initializer.

So then, I tried adding animals.txt to the things I'm redefining (in this commit) -- (and also pulled MacroTools to master to make sure to get the change you made there: MacroTools.jl/pull/77).

But the problem now is that it still can't find animals.txt _because it hasn't executed change_dir_if_bundle yet!_ The __init__ method is run when the module is loaded, which is way before julia_main is run. 

Base.InitError(mod=:MacroTools, error=Base.SystemError(prefix="opening file animals.txt", errnum=2, extrainfo=nothing))

So I think the takeaway here is that -- in order to be shippable -- a Package cannot do any initialization that references the filesystem in their __init__ function. If something like that is necessary, they would have to provide a custom init that the user could call after cding to the right place or something.

NHDaly commented 6 years ago

@NHDaly, I think we should be able to send PRs to make generic changes to every package we need to make them statically compilable (and therefore shippable). Packages will definitely have an interest in being shippable.

Yeah, I agree. I think you're right. Figuring out the best way to do that, though, is maybe something we can talk to people about this week! :)

ranjanan commented 6 years ago

because it hasn't executed change_dir_if_bundle yet! The init method is run when the module is loaded, which is way before julia_main is run.

Yes, the right fix is to put animals.txt in a separate function that runs way later. That's the true fix. I'll discuss this week with you about the list of changes. In fact, maybe we should sit and discuss about our talks as well. :-)

NHDaly commented 6 years ago

Or, maybe we could somehow disable running the module initializers when compiling, and then just manually call them all after doing the cd.

In fact, we could even do all that in C... which could be easier.

NHDaly commented 6 years ago

In fact, maybe we should sit and discuss about our talks as well. :-)

Yeah that would be excellent!! :)

I'll be in london tomorrow (monday) morning!!!!! :D

NHDaly commented 6 years ago

Oh! Okay cool! With ranjan's help offline, i've fixed it! :) https://github.com/NHDaly/ApplicationBuilder.jl/commit/c89a401cb823193c07b22c13f21e8da084b2bb5d

It required making this change to MacroTools, which i'll send a PR for later: https://github.com/NHDaly/MacroTools.jl/commit/2b4034b6f29d74eb409ff1a6b64b7a8908c9e2ca

yay

gabrielfreire commented 5 years ago

Hi Guys,

Nice discussion. @NHDaly thank you for putting so much effort on this.

I had a lot of problems to run my executable in another computer with the HTTP package because of MbedTLS on Windows 10 and Julia 1.1.0.

i solved by injecting this code inside my julia_main

MbedTLS = HTTP.Servers.MbedTLS
if get(ENV, "COMPILING_APPLE_BUNDLE", "false") == "true"
        # i had to clear out this const depsjl_path because 
        # it points to deps.jl which sets the lib paths in runtime
        Core.eval(MbedTLS, :(const depsjl_path = ""))

        # set lib path relative to PROGRAM_FILE path
        # p.s. single \ or double \\ won't work. 
        # joinpath, abspath, dirname, etc will resolve to wrong paths and will break runtime
        # thankfully julia works well with \\\\ and its possible to have 
        # relative paths to the executable installation
        Core.eval(MbedTLS, :(const libmbedcrypto = "..\\\\lib\\\\$(basename(libmbedcrypto))"))
        Core.eval(MbedTLS, :(const libmbedtls = "..\\\\lib\\\\$(basename(libmbedtls))"))
        Core.eval(MbedTLS, :(const libmbedx509 = "..\\\\lib\\\\$(basename(libmbedx509))"))
else
        # Check if PROGRAM_FILE is my program and not PackageCompiler
        if occursin("MyOwnProgramName", PROGRAM_FILE)
            this_path = abspath(dirname(PROGRAM_FILE))
            cd(this_path) # cd there otherwise it won't find the libraries
        end
end

and my libraries array looks like this

build_app_bundle(app_path, 
                appname="MyOwnProgramName", 
                snoopfile="call_functions.jl", 
                verbose=true, 
                create_installer=true,
                libraries=[HTTP.Servers.MbedTLS.libmbedcrypto, 
                               HTTP.Servers.MbedTLS.libmbedtls, 
                               HTTP.Servers.MbedTLS.libmbedx509])

BTW, i think COMPILING_APPLE_BUNDLE should be COMPILING_BUNDLE hehe

Cheers.