search

MakieOrg / Makie.jl

Interactive data visualizations and plotting in Julia
https://docs.makie.org/stable
MIT License
2.39k stars 306 forks source link

Precompilation of Makie does not work #312

Closed asinghvi17 closed 5 years ago

asinghvi17 commented 5 years ago

While snooping runtests, the following error occurred (while Makie was being tested): 

[ Info: activating new environment at ~/.julia/packages/PackageCompiler/oT98U/packages/Makie/Project.toml.
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
[ Info: using reference images for version 0.0.9 (already downloaded)
[ Info: starting from index 1
[ Info: Caching fonts, this may take a while. Needed only on first run!
running tutorial_simple_scatter
/Users/anshul/.julia/packages/Makie/TquKG/test/test_recordings/tutorial_simple_scatter/media/image.jpg
┌ Warning: Snoop file errored. Precompile statements were recorded untill error!
│   exception =
│    LoadError: MethodError: no method matching backend_show(::GLMakie.GLBackend, ::IOContext{IOStream}, ::MIME{Symbol("image/jpeg")}, ::Scene)
│    Closest candidates are:
│      backend_show(::Any, ::IO, !Matched::MIME{Symbol("text/plain")}, ::Scene) at /Users/anshul/.julia/packages/AbstractPlotting/NOT5R/src/display.jl:72
│      backend_show(::GLMakie.GLBackend, ::IO, !Matched::MIME{Symbol("image/png")}, ::Scene) at /Users/anshul/.julia/packages/GLMakie/rAuGK/src/gl_backend.jl:69
│      backend_show(::Any, ::IO, !Matched::MIME{Symbol("text/plain")}, !Matched::Combined) at /Users/anshul/.julia/packages/AbstractPlotting/NOT5R/src/display.jl:89
│    in expression starting at /Users/anshul/.julia/packages/Makie/TquKG/test/runtests.jl:20
└ @ Main ~/.julia/packages/PackageCompiler/oT98U/sysimg/run_julia_code.jl:12

It seems that a JPEG is being loaded while a PNG is expected in MakieGallery.

SimonDanisch commented 5 years ago

You need GLMakie#master ... just about to tag a new version of it ...

SimonDanisch commented 5 years ago

Jeez, these version mismatches really get annoying... Maybe we need a fake Makie mono repository at some point

asinghvi17 commented 5 years ago

Fakie.jl? :P

SimonDanisch commented 5 years ago

:D Could just be the current Makie repository, which is right now almost empty anyways ;)

asinghvi17 commented 5 years ago

The issue is still occurring on the latest versions of GLMakie and AbstractPlotting.

[ Info: activating new environment at ~/.julia/packages/PackageCompiler/oT98U/packages/Makie/Project.toml.
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
[ Info: using reference images for version 0.0.9 (already downloaded)
[ Info: starting from index 1
running tutorial_simple_scatter
/Users/anshul/.julia/packages/Makie/TquKG/test/test_recordings/tutorial_simple_scatter/media/image.jpg
┌ Warning: Snoop file errored. Precompile statements were recorded untill error!
│   exception =
│    LoadError: MethodError: no method matching backend_show(::GLMakie.GLBackend, ::IOContext{IOStream}, ::MIME{Symbol("image/jpeg")}, ::Scene)
│    Closest candidates are:
│      backend_show(::Any, ::IO, !Matched::MIME{Symbol("text/plain")}, ::Scene) at /Users/anshul/.julia/packages/AbstractPlotting/NOT5R/src/display.jl:72
│      backend_show(::GLMakie.GLBackend, ::IO, !Matched::MIME{Symbol("image/png")}, ::Scene) at /Users/anshul/.julia/packages/GLMakie/rAuGK/src/gl_backend.jl:69
│      backend_show(::Any, ::IO, !Matched::MIME{Symbol("text/plain")}, !Matched::Combined) at /Users/anshul/.julia/packages/AbstractPlotting/NOT5R/src/display.jl:89
│    in expression starting at /Users/anshul/.julia/packages/Makie/TquKG/test/runtests.jl:20
└ @ Main ~/.julia/packages/PackageCompiler/oT98U/sysimg/run_julia_code.jl:12
asinghvi17 commented 5 years ago 
function save_media(entry, x::Scene, path::String)
    path = joinpath(path, "image.jpg")
    save(path, x)
    [path]
end

is basically where the error is being thrown from (this is in https://github.com/JuliaPlots/MakieGallery.jl/blob/98d91921fc16626ed121649e5cba3d697a547b33/src/io.jl#L1). It seems GLMakie doesn't want to save as JPG.

asinghvi17 commented 5 years ago 
function AbstractPlotting.backend_show(::GLBackend, io::IO, m::MIME"image/jpeg", scene::Scene)
    img = scene2image(scene)
    FileIO.save(FileIO.Stream(FileIO.format"JPEG", io), img)
end

(from https://github.com/JuliaPlots/GLMakie.jl/blob/9847f3c4c2687fce404f78f06df8c21135cce302/src/gl_backend.jl#L77-L80 )

exists in GLMakie, but it seems it's not being found? This is pretty weird...it's persisted through a version as well.

asinghvi17 commented 5 years ago 

julia> using Makie

julia> AbstractPlotting.current_backend
Base.RefValue{Union{Missing, AbstractBackend}}(GLMakie.GLBackend())

help?> AbstractPlotting.backend_show
  No documentation found.

  AbstractPlotting.backend_show is a Function.

  # 3 methods for generic function "backend_show":
  [1] backend_show(backend, io::IO, ::MIME{Symbol("text/plain")}, scene::Scene) in AbstractPlotting at /Users/anshul/.julia/packages/AbstractPlotting/NOT5R/src/display.jl:72
  [2] backend_show(backend, io::IO, ::MIME{Symbol("text/plain")}, plot::Combined) in AbstractPlotting at /Users/anshul/.julia/packages/AbstractPlotting/NOT5R/src/display.jl:89
  [3] backend_show(::GLMakie.GLBackend, io::IO, m::MIME{Symbol("image/png")}, scene::Scene) in GLMakie at /Users/anshul/.julia/packages/GLMakie/rAuGK/src/gl_backend.jl:69
SimonDanisch commented 5 years ago

Maybe compile_incremental ends up adding the wrong GLMakie version.. can you try to dev GLMakie?

asinghvi17 commented 5 years ago

Yep. [e9467ef8] + GLMakie v0.0.4 is in Project.toml in PackageCompiler. Just caught that.

screen shot 2019-03-06 at 1 49 06 10500pm

is a screenshot from METADATA.jl - the latest PR was for v0.0.4, maybe attobot hasn't opened new pull requests for the new tags?

asinghvi17 commented 5 years ago

Also, dev GLMakie still results in PackageCompiler using GLMakie v0.0.4...

SimonDanisch commented 5 years ago

Weird...guess that's a PackageCompiler bug then...

asinghvi17 commented 5 years ago

As discussed on Slack, the solution, since PackageCompiler uses the latest tagged version on METADATA, is to manually construct a snoopfile from the runtests.jl of the packages you want to compile.

Below is some code that will do this. However, no guarantees are made that you will keep your sanity.


using PackageCompiler

const pkgs = [:Makie, :AbstractPlotting, :StatsMakie] # replace this with whatever packages you want to precompile

const writedir = "~" |> expanduser # replace this with wherever you want the snoopfile and precompile to be written

testfiles = pkgs .|> pkg -> abspath(joinpath(dirname(Base.find_package(string(pkg))), "..")) .|> x -> joinpath(x, "test") .|> x -> joinpath(x, "runtests.jl") .|> realpath .|> x -> "include(\""*x*"\")\n"

mkdir(writedir) # make sure directory exists

touch(joinpath(writedir, "snoopfile.jl")) # make sure file exists

open(joinpath(writedir, "snoopfile.jl"), "w") do f
    write.(f, testfiles)
end

precompile = PackageCompiler.snoop(nothing, nothing, joinpath(writedir, "snoopfile.jl"), joinpath(writedir, "precompile.jl"), reuse = false, blacklist = [])

cso, nso = compile_incremental(nothing, precompile) # compile incrementally the calls from the snoopfile to the sysimage.
asinghvi17 commented 5 years ago

Error during test as well:

running comparing_contours,_image,_surfaces_and_heatmaps
/Users/anshul/.julia/packages/Makie/TquKG/test/test_recordings/comparing_contours,_image,_surfaces_and_heatmaps/media/image.jpg
running statsmakie
/Users/anshul/.julia/packages/Makie/TquKG/test/test_recordings/statsmakie/media/image2.jpg
/Users/anshul/.julia/packages/Makie/TquKG/test/test_recordings/statsmakie/media/image3.jpg
/Users/anshul/.julia/packages/Makie/TquKG/test/test_recordings/statsmakie/media/image4.jpg
/Users/anshul/.julia/packages/Makie/TquKG/test/test_recordings/statsmakie/media/image5.jpg
/Users/anshul/.julia/packages/Makie/TquKG/test/test_recordings/statsmakie/media/image6.jpg
/Users/anshul/.julia/packages/Makie/TquKG/test/test_recordings/statsmakie/media/image7.jpg
/Users/anshul/.julia/packages/Makie/TquKG/test/test_recordings/statsmakie/media/image8.jpg
/Users/anshul/.julia/packages/Makie/TquKG/test/test_recordings/statsmakie/media/image9.jpg
/Users/anshul/.julia/packages/Makie/TquKG/test/test_recordings/statsmakie/media/image10.jpg
/Users/anshul/.julia/packages/Makie/TquKG/test/test_recordings/statsmakie/media/image11.jpg
/Users/anshul/.julia/packages/Makie/TquKG/test/test_recordings/statsmakie/media/image12.jpg
/Users/anshul/.julia/packages/Makie/TquKG/test/test_recordings/statsmakie/media/image14.jpg
/Users/anshul/.julia/packages/Makie/TquKG/test/test_recordings/statsmakie/media/image15.jpg
/Users/anshul/.julia/packages/Makie/TquKG/test/test_recordings/statsmakie/media/image16.jpg
/Users/anshul/.julia/packages/Makie/TquKG/test/test_recordings/statsmakie/media/image17.jpg
/Users/anshul/.julia/packages/Makie/TquKG/test/test_recordings/statsmakie/media/image18.jpg
/Users/anshul/.julia/packages/Makie/TquKG/test/test_recordings/statsmakie/media/image19.jpg
/Users/anshul/.julia/packages/Makie/TquKG/test/test_recordings/statsmakie/media/image20.jpg
/Users/anshul/.julia/packages/Makie/TquKG/test/test_recordings/statsmakie/media/image21.jpg
/Users/anshul/.julia/packages/Makie/TquKG/test/test_recordings/statsmakie/media/image22.jpg
/Users/anshul/.julia/packages/Makie/TquKG/test/test_recordings/statsmakie/media/image23.jpg
Reference Image Tests: Error During Test at /Users/anshul/.julia/packages/MakieGallery/Zh28t/src/testruns.jl:84
  Got exception outside of a @test
  SystemError: unable to read directory /Users/anshul/.julia/packages/Makie/TquKG/test/tested_different/3d_contour_with_2d_contour_slices/media: No such file or directory
  Stacktrace:
   [1] readdir(::String) at ./file.jl:625
   [2] #run_comparison#126(::Float64, ::Function, ::String, ::String, ::String) at /Users/anshul/.julia/packages/MakieGallery/Zh28t/src/testruns.jl:91
   [3] run_comparison(::String, ::String, ::String) at /Users/anshul/.julia/packages/MakieGallery/Zh28t/src/testruns.jl:84
   [4] top-level scope at none:0
   [5] include at ./boot.jl:326 [inlined]
   [6] include_relative(::Module, ::String) at ./loading.jl:1038
   [7] include(::Module, ::String) at ./sysimg.jl:29
   [8] include(::String) at ./client.jl:403
   [9] top-level scope at none:0
   [10] include at ./boot.jl:326 [inlined]
   [11] include_relative(::Module, ::String) at ./loading.jl:1038
   [12] include(::Module, ::String) at ./sysimg.jl:29
   [13] include(::String) at ./client.jl:403
   [14] top-level scope at /Users/anshul/.julia/packages/PackageCompiler/oT98U/sysimg/run_julia_code.jl:5
   [15] include at ./boot.jl:326 [inlined]
   [16] include_relative(::Module, ::String) at ./loading.jl:1038
   [17] include(::Module, ::String) at ./sysimg.jl:29
   [18] exec_options(::Base.JLOptions) at ./client.jl:267
   [19] _start() at ./client.jl:436
Test Summary:         | Error  Total
Reference Image Tests |     1      1
┌ Warning: Snoop file errored. Precompile statements were recorded untill error!
│   exception =
│    LoadError: LoadError: Some tests did not pass: 0 passed, 0 failed, 1 errored, 0 broken.
│    in expression starting at /Users/anshul/.julia/packages/Makie/TquKG/test/runtests.jl:21
│    in expression starting at /Users/anshul/snoopfile.jl:1
└ @ Main ~/.julia/packages/PackageCompiler/oT98U/sysimg/run_julia_code.jl:7
[ Info: used 16152 out of 16749 precompile statement

On further investigation, the directory /Users/anshul/.julia/packages/Makie/TquKG/test/tested_different exists but has no subdirectories.

I'm running it again to see if there are any errors.

asinghvi17 commented 5 years ago

After GLMakie v0.0.5 merge into METADATA, PackageCompiler now uses v0.0.5. compile_incremental now does not error on tutorial_simple_scatter.

asinghvi17 commented 5 years ago

@SimonDanisch I think we might need to fix the WorldClim visualization: 

running type_recipe_for_molecule_simulation
┌ Warning: Snoop file errored. Precompile statements were recorded untill error!
│   exception =
│    LoadError: LoadError: KeyError: key "LD_LIBRARY_PATH" not found
│    in expression starting at worldclim_visualization:4
│    in expression starting at /Users/anshul/.julia/packages/Makie/ogxgz/test/runtests.jl:12
└ @ Main ~/.julia/packages/PackageCompiler/oT98U/sysimg/run_julia_code.jl:12

This issue is fairly simple, I think we just have to change a couple of lines in the example to get it to work.

Also it seems like MakieGallery is printing running <test> after it actually runs it.

SimonDanisch commented 5 years ago

fixed it in MakieGallery

cserteGT3 commented 5 years ago

I have the same issue as @asinghvi17 using the latest versions of Makie related packages (not on #master):

LoadError: LoadError: KeyError: key "LD_LIBRARY_PATH" not found

I thought it's fixed in the latest version of MakieGallery, but now I see it's only on master yet.

Edit: versions:

(v1.1) pkg> st
    Status `C:\Users\cstamas\.julia\environments\v1.1\Project.toml`
  [537997a7] AbstractPlotting v0.9.6
  [c52e3926] Atom v0.7.14
  [6e4b80f9] BenchmarkTools v0.4.2
  [5ae59095] Colors v0.9.5
  [5789e2e9] FileIO v1.0.5
  [f7f18e0c] GLFW v2.3.0
  [e9467ef8] GLMakie v0.0.5
  [4d00f742] GeometryTypes v0.7.3
  [7073ff75] IJulia v1.17.0
  [6218d12a] ImageMagick v0.7.1
  [c601a237] Interact v0.9.1
  [033835bb] JLD2 v0.1.2
  [e5e0dc1b] Juno v0.5.4
  [fc18253b] LazyJSON v0.1.0
  [ee78f7c6] Makie v0.9.3
  [dbd62bd0] MakieGallery v0.0.6
  [283c5d60] MeshCat v0.6.0
  [6ad125db] MeshCatMechanisms v0.3.0
  [66fc600b] ModernGL v1.0.0
  [510215fc] Observables v0.2.3
  [9b87118b] PackageCompiler v0.6.3+ #master (https://github.com/JuliaLang/PackageCompiler.jl.git)
  [df47a6cb] RData v0.6.0
  [ce6b1742] RDatasets v0.6.1
  [dee08c22] RegionTrees v0.2.0
  [295af30f] Revise v1.1.0
  [366cf18f] RigidBodyDynamics v1.4.0
  [6038ab10] Rotations v0.11.1
  [90137ffa] StaticArrays v0.10.3
  [65254759] StatsMakie v0.0.3
  [8603256b] ZChop v0.3.3
sdewaele commented 5 years ago

I was able to precompile Makie using the manually created snoopfile, after adding the packages that are required for testing. (I did also have to wrap the different runtests into their own temporary modules to prevent a naming conflict.) However, I don't get any speedup when I run Julia with the new system image with:

julia -J ~/julia-1.0.0/lib/julia/sys.so

Either with or without the new image, the following code takes 40 seconds to run:

@time begin
    using Makie
    scatter(1:3,1:3)
end

What can be the reason for this?

asinghvi17 commented 5 years ago

That's odd, especially since I'm getting the same problem now. I will attempt to figure out why, but it may take a while, and any assistance would be appreciated.

SimonDanisch commented 5 years ago

manually created snoopfile

Since everything is usually wrapped in a try catch, maybe you edited in a bad way? You can change the @debug to @info, to see if the precompile statemetns go as planned