JuliaLang / PackageCompiler.jl

Compile your Julia Package
https://julialang.github.io/PackageCompiler.jl/dev/
MIT License
1.42k stars 190 forks source link

IOError when compiling Plots #181

Closed karajan9 closed 5 years ago

karajan9 commented 5 years ago

I tried to compile Plots following the procedure posted here. I run Julia 1.1.0 on Windows as Administrator. I get the following error:

julia> using Pkg

julia> pkg"up; add PackageCompiler Plots"
  Updating registry at `C:\Users\Administrator\.julia\registries\General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
  Updating `C:\Users\Administrator\.julia\environments\v1.1\Project.toml`
 [no changes]
  Updating `C:\Users\Administrator\.julia\environments\v1.1\Manifest.toml`
 [no changes]
 Resolving package versions...
  Updating `C:\Users\Administrator\.julia\environments\v1.1\Project.toml`
 [no changes]
  Updating `C:\Users\Administrator\.julia\environments\v1.1\Manifest.toml`
 [no changes]

julia> using PackageCompiler

julia> cp(joinpath(dirname(Pkg.Types.find_project_file()), "Manifest.toml"),   PackageCompiler.sysim
g_folder("Manifest.toml"), force = true )
"C:\\Users\\Administrator\\.julia\\packages\\PackageCompiler\\oT98U\\sysimg\\Manifest.toml"

julia> PackageCompiler.compile_incremental(:Plots, force = true)
[ Info: Recompiling stale cache file C:\Users\Administrator\.julia\compiled\v1.1\Plots\ld3vC.ji for
Plots [91a5bcdd-55d7-5caf-9e0b-520d859cae80]
[ Info: Registered package Plots, using already given UUID: 91a5bcdd-55d7-5caf-9e0b-520d859cae80
  Updating registry at `C:\Users\Administrator\.julia\registries\General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
  Updating `C:\Users\Administrator\.julia\packages\PackageCompiler\oT98U\packages\Plots\Project.toml
`
  [d38c429a] + Contour v0.5.1
  [53c48c17] + FixedPointNumbers v0.5.3
  [28b8d3ca] + GR v0.37.0
  [682c06a0] + JSON v0.20.0
  [442fdcdd] + Measures v0.3.0
  [77ba4419] + NaNMath v0.3.2
  [ccf2f8ad] + PlotThemes v0.3.0
  [995b91a9] + PlotUtils v0.5.5
  [3cdcf5f2] + RecipesBase v0.6.0
  [189a3867] + Reexport v0.2.0
  [ae029012] + Requires v0.5.2
  [992d4aef] + Showoff v0.2.1
  [90137ffa] + StaticArrays v0.10.2
  [2913bbd2] + StatsBase v0.27.0
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [37e2e46d] + LinearAlgebra
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [cf7118a7] + UUIDs
  Updating `C:\Users\Administrator\.julia\packages\PackageCompiler\oT98U\packages\Plots\Manifest.tom
l`
  [3da002f7] + ColorTypes v0.7.5
  [5ae59095] + Colors v0.9.5
  [34da2185] + Compat v1.5.1
  [d38c429a] + Contour v0.5.1
  [864edb3b] + DataStructures v0.15.0
  [53c48c17] + FixedPointNumbers v0.5.3
  [28b8d3ca] + GR v0.37.0
  [682c06a0] + JSON v0.20.0
  [442fdcdd] + Measures v0.3.0
  [e1d29d7a] + Missings v0.4.0
  [77ba4419] + NaNMath v0.3.2
  [bac558e1] + OrderedCollections v1.0.2
  [ccf2f8ad] + PlotThemes v0.3.0
  [995b91a9] + PlotUtils v0.5.5
  [3cdcf5f2] + RecipesBase v0.6.0
  [189a3867] + Reexport v0.2.0
  [ae029012] + Requires v0.5.2
  [992d4aef] + Showoff v0.2.1
  [a2af1166] + SortingAlgorithms v0.3.1
  [90137ffa] + StaticArrays v0.10.2
  [2913bbd2] + StatsBase v0.27.0
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8bb1440f] + DelimitedFiles
  [8ba89e20] + Distributed
  [b77e0a4c] + InteractiveUtils
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [1a1011a3] + SharedArrays
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
 Resolving package versions...
  Updating `C:\Users\Administrator\.julia\packages\PackageCompiler\oT98U\packages\Plots\Project.toml
`
  [9b87118b] + PackageCompiler v0.6.3
  Updating `C:\Users\Administrator\.julia\packages\PackageCompiler\oT98U\packages\Plots\Manifest.tom
l`
  [c7e460c6] + ArgParse v0.6.1
  [9e28174c] + BinDeps v0.8.10
  [b99e7846] + BinaryProvider v0.5.3
  [e1450e63] + BufferedStreams v1.0.0
  [0862f596] + HTTPClient v0.2.1
  [b27032c2] + LibCURL v0.4.1
  [522f3ed2] + LibExpat v0.5.0
  [2ec943e9] + Libz v1.0.0
  [9b87118b] + PackageCompiler v0.6.3
  [b718987f] + TextWrap v0.3.0
  [30578b45] + URIParser v0.4.0
  [c17dfb99] + WinRPM v0.4.2
[ Info: activating new environment at C:\Users\Administrator\.julia\packages\PackageCompiler\oT98U\p
ackages\Plots\Project.toml.
  Updating registry at `C:\Users\Administrator\.julia\registries\General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
  Updating git-repo `https://github.com/JuliaPlots/PlotUtils.jl.git`
  Updating git-repo `https://github.com/JuliaPlots/RecipesBase.jl.git`
  Updating git-repo `https://github.com/JuliaPlots/PlotReferenceImages.jl.git`
 Resolving package versions...
  Updating `C:\Users\Administrator\.julia\packages\PackageCompiler\oT98U\packages\Plots\Project.toml
`
  [5789e2e9] + FileIO v1.0.5
  [6218d12a] + ImageMagick v0.7.1
  [916415d5] + Images v0.17.2
  [b964fa9f] + LaTeXStrings v1.0.3
  [eb903711] + PlotReferenceImages v0.0.0 #master (https://github.com/JuliaPlots/PlotReferenceImages
.jl.git)
  [995b91a9] ↑ PlotUtils v0.5.5 ⇒ v0.5.5+ #master (https://github.com/JuliaPlots/PlotUtils.jl.git)
  [91a5bcdd] + Plots v0.23.0
  [dca85d43] + QuartzImageIO v0.5.2
  [ce6b1742] + RDatasets v0.6.1
  [3cdcf5f2] ↑ RecipesBase v0.6.0 ⇒ v0.6.0+ #master (https://github.com/JuliaPlots/RecipesBase.jl.gi
t)
  [f3b207a7] + StatsPlots v0.10.1
  [b8865327] + UnicodePlots v1.1.0
  [34922c18] + VisualRegressionTests v0.3.0
  Updating `C:\Users\Administrator\.julia\packages\PackageCompiler\oT98U\packages\Plots\Manifest.tom
l`
  [621f4979] + AbstractFFTs v0.3.2
  [7d9fca2a] + Arpack v0.3.0
  [13072b0f] + AxisAlgorithms v0.3.0
  [39de3d68] + AxisArrays v0.3.0
  [336ed68f] + CSV v0.4.3
  [49dc2e85] + Calculus v0.4.1
  [aafaddc9] + CatIndices v0.2.0
  [324d7699] + CategoricalArrays v0.5.2
  [aaaa29a8] + Clustering v0.12.2
  [944b1d66] + CodecZlib v0.5.1
  [c3611d14] + ColorVectorSpace v0.6.2
  [bbf7d656] + CommonSubexpressions v0.2.0
  [ed09eef8] + ComputationalResources v0.3.0
  [8f4d0f93] + Conda v1.2.0
  [150eb455] + CoordinateTransformations v0.5.0
  [dc8bdbbb] + CustomUnitRanges v0.2.0
  [a93c6f00] + DataFrames v0.17.1
  [9a8bc11e] + DataStreams v0.4.1
  [e7dc6d0d] + DataValues v0.4.7
  [01453d9d] + DiffEqDiffTools v0.7.1
  [163ba53b] + DiffResults v0.0.3
  [b552c78f] + DiffRules v0.0.8
  [b4f34e82] + Distances v0.7.4
  [31c24e10] + Distributions v0.16.4
  [8f5d6c58] + EzXML v0.9.0
  [4f61f5a4] + FFTViews v0.2.0
  [7a1cc6ca] + FFTW v0.2.4
  [5789e2e9] + FileIO v1.0.5
  [f6369f11] + ForwardDiff v0.10.2
  [a2bd30eb] + Graphics v0.4.0
  [bbac6d45] + IdentityRanges v0.3.0
  [2803e5a7] + ImageAxes v0.5.0
  [a09fc81d] + ImageCore v0.7.4
  [51556ac3] + ImageDistances v0.1.1
  [6a3955dd] + ImageFiltering v0.5.3
  [6218d12a] + ImageMagick v0.7.1
  [bc367c6b] + ImageMetadata v0.6.1
  [787d08f9] + ImageMorphology v0.1.1
  [4e3cecfd] + ImageShow v0.1.2
  [02fcd773] + ImageTransformations v0.7.2
  [916415d5] + Images v0.17.2
  [9b13fd28] + IndirectArrays v0.5.0
  [a98d9a8b] + Interpolations v0.11.2
  [8197267c] + IntervalSets v0.3.1
  [c8e1da08] + IterTools v1.1.1
  [1c8ee90f] + IterableTables v0.10.0
  [82899510] + IteratorInterfaceExtensions v0.1.1
  [5ab0869b] + KernelDensity v0.5.1
  [b964fa9f] + LaTeXStrings v1.0.3
  [d3d80556] + LineSearches v7.0.1
  [1914dd2f] + MacroTools v0.4.4
  [dbb5928d] + MappedArrays v0.2.1
  [78c3b35d] + Mocking v0.5.7
  [d41bc354] + NLSolversBase v7.1.2
  [b8a86587] + NearestNeighbors v0.4.3
  [4d1e1d77] + Nullables v0.0.8
  [510215fc] + Observables v0.2.3
  [6fe1bfb0] + OffsetArrays v0.10.0
  [429524aa] + Optim v0.17.2
  [90014a1f] + PDMats v0.9.6
  [5432bcbf] + PaddedViews v0.4.2
  [d96e819e] + Parameters v0.10.3
  [69de0a69] + Parsers v0.2.16
  [eb903711] + PlotReferenceImages v0.0.0 #master (https://github.com/JuliaPlots/PlotReferenceImages
.jl.git)
  [995b91a9] ↑ PlotUtils v0.5.5 ⇒ v0.5.5+ #master (https://github.com/JuliaPlots/PlotUtils.jl.git)
  [91a5bcdd] + Plots v0.23.0
  [85a6dd25] + PositiveFactorizations v0.2.1
  [92933f4c] + ProgressMeter v0.9.0
  [1fd47b50] + QuadGK v2.0.3
  [dca85d43] + QuartzImageIO v0.5.2
  [df47a6cb] + RData v0.6.0
  [ce6b1742] + RDatasets v0.6.1
  [b3c3ace0] + RangeArrays v0.3.1
  [c84ed2f1] + Ratios v0.3.0
  [3cdcf5f2] ↑ RecipesBase v0.6.0 ⇒ v0.6.0+ #master (https://github.com/JuliaPlots/RecipesBase.jl.gi
t)
  [79098fc4] + Rmath v0.5.0
  [6038ab10] + Rotations v0.10.0
  [b9d75638] + SIUnits v0.1.0
  [699a6c99] + SimpleTraits v0.8.0
  [276daf66] + SpecialFunctions v0.7.2
  [4c63d2b9] + StatsFuns v0.8.0
  [f3b207a7] + StatsPlots v0.10.1
  [3783bdb8] + TableTraits v0.4.1
  [382cd787] + TableTraitsUtils v0.3.1
  [bd369af6] + Tables v0.1.15
  [9b435220] + TexExtensions v0.1.0
  [06e1c1a7] + TiledIteration v0.2.3
  [f269a46b] + TimeZones v0.8.5
  [3bb67fe8] + TranscodingStreams v0.8.1
  [b8865327] + UnicodePlots v1.1.0
  [81def892] + VersionParsing v1.1.3
  [34922c18] + VisualRegressionTests v0.3.0
  [ea10d353] + WeakRefStrings v0.5.6
  [cc8bc4a8] + Widgets v0.4.4
  [efce3f68] + WoodburyMatrices v0.4.1
  [9fa8497b] + Future
  [9abbd945] + Profile
  [4607b0f0] + SuiteSparse
  Building Conda ──────→ `C:\Users\Administrator\.julia\packages\Conda\CpuvI\deps\build.log`
  Building FFTW ───────→ `C:\Users\Administrator\.julia\packages\FFTW\p7sLQ\deps\build.log`
  Building ImageMagick → `C:\Users\Administrator\.julia\packages\ImageMagick\d5KBL\deps\build.log`
Test Summary: | Pass  Total
GR            |    2      2
Test Summary: | Pass  Total
UnicodePlots  |    3      3
Test Summary: | Pass  Total
Axes          |    6      6
Test Summary: |
NoFail        | No tests
[ Info: used 787 out of 787 precompile statements
┌ Info: activating new environment at C:\Users\Administrator\.julia\packages\PackageCompiler\oT98U\p
ackages\Project.toml.
└ @ Pkg.API C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\
v1.1\Pkg\src\API.jl:519
  Updating registry at `C:\Users\Administrator\.julia\registries\General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
┌ Warning: QuartzImageIO.jl can only be used on Apple macOS. Suggested usage is
│     @static if Sys.isapple()
│         using QuartzImageIO
│         # QuartzImageIO specific code goes here
│     end
└ @ QuartzImageIO C:\Users\Administrator\.julia\packages\QuartzImageIO\9DhKg\src\QuartzImageIO.jl:72
3
Build shared library "C:\\Users\\Administrator\\.julia\\packages\\PackageCompiler\\oT98U\\sysimg\\sy
s.dll":
  `'C:\Users\Administrator\.julia\packages\WinRPM\Y9QdZ\deps\usr\x86_64-w64-mingw32\sys-root\mingw\b
in\gcc.exe' --sysroot 'C:\Users\Administrator\.julia\packages\WinRPM\Y9QdZ\deps\usr\x86_64-w64-mingw
32\sys-root' -shared '-DJULIAC_PROGRAM_LIBNAME="C:\Users\Administrator\.julia\packages\PackageCompil
er\oT98U\sysimg\sys.dll"' -o 'C:\Users\Administrator\.julia\packages\PackageCompiler\oT98U\sysimg\sy
s.dll' -Wl,--whole-archive 'C:\Users\Administrator\.julia\packages\PackageCompiler\oT98U\sysimg\sys.
a' -Wl,--no-whole-archive -std=gnu99 '-IC:\Users\Karajan\AppData\Local\Julia-1.1.0\include\julia'
-DJULIA_ENABLE_THREADING=1 '-LC:\Users\Karajan\AppData\Local\Julia-1.1.0\bin' -Wl,--stack,8388608
-ljulia -lopenlibm -m64 -O3 -Wl,--export-all-symbols`
ERROR: IOError: unlink: permission denied (EACCES)
Stacktrace:
 [1] uv_error at .\libuv.jl:85 [inlined]
 [2] unlink(::String) at .\file.jl:727
 [3] #rm#9(::Bool, ::Bool, ::Function, ::String) at .\file.jl:253
 [4] #rm at .\none:0 [inlined]
 [5] #checkfor_mv_cp_cptree#10(::Bool, ::Function, ::String, ::String, ::String) at .\file.jl:291
 [6] #checkfor_mv_cp_cptree at .\none:0 [inlined]
 [7] #cp#12(::Bool, ::Bool, ::Function, ::String, ::String) at .\file.jl:330
 [8] (::getfield(Base.Filesystem, Symbol("#kw##cp")))(::NamedTuple{(:force,),Tuple{Bool}}, ::typeof(
cp), ::String, ::String) at .\none:0
 [9] #compile_incremental#62(::Bool, ::Bool, ::Bool, ::Nothing, ::Function, ::String, ::String) at C
:\Users\Administrator\.julia\packages\PackageCompiler\oT98U\src\incremental.jl:88
 [10] #compile_incremental#63 at .\none:0 [inlined]
 [11] (::getfield(PackageCompiler, Symbol("#kw##compile_incremental")))(::NamedTuple{(:force,),Tuple
{Bool}}, ::typeof(compile_incremental), ::Symbol) at .\none:0
 [12] top-level scope at none:0

julia>

It seems that there is some problem mixing the Administrator (which I need to make ImageMagick build) and my usual user where Julia is installed?

SimonDanisch commented 5 years ago

force true is relatively buggy on windows! i dont really know what to do about it^^ i think you'll need to close julia and manually copy them. make sure that you have a back-up! https://nextjournal.com/sdanisch/static-cairomakie in the last cell is described which paths you need to copy

karajan9 commented 5 years ago

That worked, thank you. I wasn't aware the building worked and only the copying was problematic. When starting Julia I now get the following message

┌ Warning: QuartzImageIO.jl can only be used on Apple macOS. Suggested usage is
│     @static if Sys.isapple()
│         using QuartzImageIO
│         # QuartzImageIO specific code goes here
│     end
└ @ QuartzImageIO C:\Users\Administrator\.julia\packages\QuartzImageIO\9DhKg\src\QuartzImageIO.jl:72
3
Starting Julia...

but everything seems to work fine. Thanks again for your awesome work!

Edit: every time something recompiles there are lots of these messages.

SimonDanisch commented 5 years ago

Hm, yeah, that's annoying... This might only be solvable by explicitely special casing QuartzImageIO.

goingtosleep commented 5 years ago

That worked, thank you. I wasn't aware the building worked and only the copying was problematic. When starting Julia I now get the following message

┌ Warning: QuartzImageIO.jl can only be used on Apple macOS. Suggested usage is
│     @static if Sys.isapple()
│         using QuartzImageIO
│         # QuartzImageIO specific code goes here
│     end
└ @ QuartzImageIO C:\Users\Administrator\.julia\packages\QuartzImageIO\9DhKg\src\QuartzImageIO.jl:72
3
Starting Julia...

but everything seems to work fine. Thanks again for your awesome work!

Edit: every time something recompiles there are lots of these messages.

Is plot(rand(100)) fast? I tried compiling Plots on Windows, using Plots was fast but then plot(rand(100)) I had to wait as usual.

SimonDanisch commented 5 years ago

did you run with the new system image?

goingtosleep commented 5 years ago

Yes, I replaced sys.dll and got the same warning about QuartzImageIO.jl at startup. using Plots is fast, but plot is slow.

giordano commented 5 years ago

I got the same error as the one reported in the first message with Julia 1.1.0 on Arch Linux, so it's not Windows-only:

ERROR: IOError: unlink: permission denied (EACCES)
Stacktrace:
 [1] uv_error at ./libuv.jl:85 [inlined]
 [2] unlink(::String) at ./file.jl:727
 [3] #rm#9(::Bool, ::Bool, ::Function, ::String) at ./file.jl:253
 [4] #rm at ./none:0 [inlined]
 [5] #checkfor_mv_cp_cptree#10(::Bool, ::Function, ::String, ::String, ::String) at ./file.jl:291
 [6] #checkfor_mv_cp_cptree at ./none:0 [inlined]
 [7] #cp#12(::Bool, ::Bool, ::Function, ::String, ::String) at ./file.jl:330
 [8] (::getfield(Base.Filesystem, Symbol("#kw##cp")))(::NamedTuple{(:force,),Tuple{Bool}}, ::typeof(cp), ::String, ::String) at ./none:0
 [9] #compile_incremental#62(::Bool, ::Bool, ::Bool, ::Nothing, ::Function, ::String, ::String) at /home/mose/.julia/packages/PackageCompiler/oT98U/src/incremental.jl:88
 [10] #compile_incremental#63 at ./none:0 [inlined]
 [11] (::getfield(PackageCompiler, Symbol("#kw##compile_incremental")))(::NamedTuple{(:force,),Tuple{Bool}}, ::typeof(compile_incremental), ::Symbol) at ./none:0
 [12] top-level scope at none:0
SimonDanisch commented 5 years ago

well, coverage of the plots tests isn't great, so lots of functions will stay uncompiled ;) i only tested with scatter and that seemed to be fast!

karajan9 commented 5 years ago

I got the following times: With PackageCompiler:

julia> @time using Plots
  0.001365 seconds (811 allocations: 45.156 KiB)

julia> @time plot(rand(100))
  4.528679 seconds (9.51 M allocations: 472.503 MiB, 5.71% gc time)

julia> @time scatter(rand(100))
  0.041636 seconds (71.78 k allocations: 3.643 MiB)

and after a restart:

julia> @time using Plots
  0.002524 seconds (811 allocations: 45.156 KiB)

julia> @time scatter(rand(100))
  4.950979 seconds (9.56 M allocations: 474.999 MiB, 5.36% gc time)

julia> @time plot(rand(100))
  0.019660 seconds (22.45 k allocations: 1.147 MiB)

Without PackageCompiler:

julia> @time using Plots
  5.786976 seconds (9.94 M allocations: 531.261 MiB, 4.57% gc time)

julia> @time plot(rand(100))
 20.307255 seconds (46.62 M allocations: 2.270 GiB, 6.82% gc time)

julia> @time scatter(rand(100))
  0.044161 seconds (101.00 k allocations: 5.193 MiB)

and after a restart:

julia> @time using Plots
  5.994450 seconds (9.94 M allocations: 531.259 MiB, 4.46% gc time)

julia> @time scatter(rand(100))
 20.743231 seconds (46.66 M allocations: 2.272 GiB, 6.72% gc time)

julia> @time plot(rand(100))
  0.014998 seconds (19.56 k allocations: 1.016 MiB)

So at least for me it is definitely faster, going from 25 s to 5 s feels worth it to me. If you mean by "scatter is going fast" even better than this, then I can't make out a difference.

Do I understand you correctly, the way to improve this would be add a snoopfile in which I need to write the things I want to have compiled?

Edit: Including plot(rand(100)) in the runtest file brings this down to

julia> @time using Plots
  0.001461 seconds (811 allocations: 45.156 KiB)

julia> @time plot(rand(100))
  3.752079 seconds (8.26 M allocations: 409.763 MiB, 6.07% gc time)

Not quite sure why it's not zero now, but hey, still better than before ;-)