search

JuliaLang / Distributed.jl

Create and control multiple Julia processes remotely for distributed computing. Ships as a Julia stdlib.
https://docs.julialang.org/en/v1/stdlib/Distributed/
MIT License
20 stars 8 forks source link

Loading/precompiling packages requires them to be available on Distributed workers #80

Open Uroc327 opened 2 years ago

Uroc327 commented 2 years ago

I have the julia REPL running on my local machine and I use Distributed to connect to a headless remote server via ssh. I want to use these workers to do some @distributed computations and then plot the results on my local REPL, e.g. using GLMakie.

As GLMakie requires X11 and graphics drivers to be installed, it is only installed on my local machine but not on the server. When I try to include GLMakie or call it after having connected to the headless server, the call fails.

Example:

using GLMakie
using Distributed

addprocs([...]) # This should connect to the headless server

scatter(randn(10), randn(10))

This fails with the error message

Errors encountered while load FileIO.File{FileIO.DataFormat{:PNG}, String}("/home/Uroc327/.julia/packages/Makie/lgPZh/assets/icons/icon-128.png").
All errors:
===========================================
On worker 2:
ArgumentError: Package PNGFiles [f57f5aa1-a3ce-4bc8-8ab9-96f992907883] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

Stacktrace:
 [1] _require
   @ ./loading.jl:1012
 [2] require
   @ ./loading.jl:936
 [3] JuliaLang/julia#1
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/Distributed.jl:79
 [4] JuliaLang/julia#103
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:274
 [5] run_work_thunk
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:63
 [6] run_work_thunk
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:72
 [7] JuliaLang/julia#96
   @ ./task.jl:411
===========================================
ArgumentError: Package ImageMagick [6218d12a-5da1-5696-b52f-db25d2ecc6d1] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

===========================================

Fatal error:
Error showing value of type Makie.FigureAxisPlot:
ERROR: On worker 2:
ArgumentError: Package PNGFiles [f57f5aa1-a3ce-4bc8-8ab9-96f992907883] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

Stacktrace:
 [1] _require
   @ ./loading.jl:1012
 [2] require
   @ ./loading.jl:936
 [3] JuliaLang/julia#1
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/Distributed.jl:79
 [4] JuliaLang/julia#103
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:274
 [5] run_work_thunk
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:63
 [6] run_work_thunk
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:72
 [7] JuliaLang/julia#96
   @ ./task.jl:411
Stacktrace:
  [1] macro expansion
    @ ./task.jl:388 [inlined]
  [2] _require_callback(mod::Base.PkgId)
    @ Distributed /builddir/julia-1.6.1/usr/share/julia/stdlib/v1.6/Distributed/src/Distributed.jl:76
  [3] JuliaLang/julia#1
    @ ~/.julia/packages/ImageIO/CzrzY/src/ImageIO.jl:54 [inlined]
  [4] lock(f::ImageIO.var"#1#2"{Base.PkgId}, l::ReentrantLock)
    @ Base ./lock.jl:187
  [5] checked_import
    @ ~/.julia/packages/ImageIO/CzrzY/src/ImageIO.jl:53 [inlined]
  [6] load(f::FileIO.File{FileIO.DataFormat{:PNG}, String}; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ ImageIO ~/.julia/packages/ImageIO/CzrzY/src/ImageIO.jl:59
  [7] load(f::FileIO.File{FileIO.DataFormat{:PNG}, String})
    @ ImageIO ~/.julia/packages/ImageIO/CzrzY/src/ImageIO.jl:59
  [8] #invokelatest#2
    @ ./essentials.jl:708 [inlined]
  [9] invokelatest
    @ ./essentials.jl:706 [inlined]
 [10] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::FileIO.Formatted; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:219
 [11] action
    @ ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:197 [inlined]
 [12] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::Symbol, ::String; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:185
 [13] action
    @ ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:185 [inlined]
 [14] load(::String; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:113
 [15] load
    @ ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:110 [inlined]
 [16] _broadcast_getindex_evalf
    @ ./broadcast.jl:648 [inlined]
 [17] _broadcast_getindex
    @ ./broadcast.jl:621 [inlined]
 [18] getindex
    @ ./broadcast.jl:575 [inlined]
 [19] copy
    @ ./broadcast.jl:922 [inlined]
 [20] materialize
    @ ./broadcast.jl:883 [inlined]
 [21] icon()
    @ Makie ~/.julia/packages/Makie/lgPZh/src/Makie.jl:262
 [22] GLMakie.Screen(; resolution::Tuple{Int64, Int64}, visible::Bool, title::String, kw_args::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ GLMakie ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:349
 [23] Screen
    @ ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:303 [inlined]
 [24] global_gl_screen
    @ ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:249 [inlined]
 [25] global_gl_screen(resolution::Tuple{Int64, Int64}, visibility::Bool, tries::Int64)
    @ GLMakie ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:394
 [26] global_gl_screen
    @ ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:393 [inlined]
 [27] backend_display
    @ ~/.julia/packages/GLMakie/TfSB1/src/display.jl:2 [inlined]
 [28] display(scene::Scene)
    @ Makie ~/.julia/packages/Makie/lgPZh/src/display.jl:59
 [29] #display#915
    @ ~/.julia/packages/Makie/lgPZh/src/display.jl:53 [inlined]
 [30] display
    @ ~/.julia/packages/Makie/lgPZh/src/display.jl:53 [inlined]
 [31] #display#914
    @ ~/.julia/packages/Makie/lgPZh/src/display.jl:52 [inlined]
 [32] display(fap::Makie.FigureAxisPlot)
    @ Makie ~/.julia/packages/Makie/lgPZh/src/display.jl:52
 [33] (::REPL.var"#do_respond#61"{Bool, Bool, REPL.var"#72#82"{REPL.LineEditREPL, REPL.REPLHistoryProvider}, REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::Any, ok::Bool)
    @ REPL /builddir/julia-1.6.1/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:798
Stacktrace:
  [1] handle_error(e::CompositeException, q::Base.PkgId, bt::Vector{Union{Ptr{Nothing}, Base.InterpreterIP}})
    @ FileIO ~/.julia/packages/FileIO/u9YLx/src/error_handling.jl:61
  [2] handle_exceptions(exceptions::Vector{Tuple{Any, Union{Base.PkgId, Module}, Vector{T} where T}}, action::String)
    @ FileIO ~/.julia/packages/FileIO/u9YLx/src/error_handling.jl:56
  [3] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::FileIO.Formatted; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:228
  [4] action
    @ ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:197 [inlined]
  [5] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::Symbol, ::String; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:185
  [6] action
    @ ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:185 [inlined]
  [7] load(::String; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:113
  [8] load
    @ ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:110 [inlined]
  [9] _broadcast_getindex_evalf
    @ ./broadcast.jl:648 [inlined]
 [10] _broadcast_getindex
    @ ./broadcast.jl:621 [inlined]
 [11] getindex
    @ ./broadcast.jl:575 [inlined]
 [12] copy
    @ ./broadcast.jl:922 [inlined]
 [13] materialize
    @ ./broadcast.jl:883 [inlined]
 [14] icon()
    @ Makie ~/.julia/packages/Makie/lgPZh/src/Makie.jl:262
 [15] GLMakie.Screen(; resolution::Tuple{Int64, Int64}, visible::Bool, title::String, kw_args::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ GLMakie ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:349
 [16] Screen
    @ ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:303 [inlined]
 [17] global_gl_screen
    @ ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:249 [inlined]
 [18] global_gl_screen(resolution::Tuple{Int64, Int64}, visibility::Bool, tries::Int64)
    @ GLMakie ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:394
 [19] global_gl_screen
    @ ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:393 [inlined]
 [20] backend_display
    @ ~/.julia/packages/GLMakie/TfSB1/src/display.jl:2 [inlined]
 [21] display(scene::Scene)
    @ Makie ~/.julia/packages/Makie/lgPZh/src/display.jl:59
 [22] #display#915
    @ ~/.julia/packages/Makie/lgPZh/src/display.jl:53 [inlined]
 [23] display
    @ ~/.julia/packages/Makie/lgPZh/src/display.jl:53 [inlined]
 [24] #display#914
    @ ~/.julia/packages/Makie/lgPZh/src/display.jl:52 [inlined]
 [25] display(fap::Makie.FigureAxisPlot)
    @ Makie ~/.julia/packages/Makie/lgPZh/src/display.jl:52
 [26] (::REPL.var"#do_respond#61"{Bool, Bool, REPL.var"#72#82"{REPL.LineEditREPL, REPL.REPLHistoryProvider}, REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::Any, ok::Bool)
    @ REPL /builddir/julia-1.6.1/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:798

When including GLMakie after having connected to the remote machine,

[...]
addprocs([...])
using GLMakie

the error message is also of the type is required but does not seem to be installed.

julia> versioninfo()
Julia Version 1.6.1
Commit 6aaedecc44* (2021-04-23 05:59 UTC)
Platform Info:
  OS: Linux (x86_64-unknown-linux-gnu)
  CPU: Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake)

julia> ] st
      Status `~/.julia/environments/v1.6/Project.toml`
  [6e4b80f9] BenchmarkTools v1.3.1
  [1e616198] COSMO v0.8.5
  [13f3f980] CairoMakie v0.7.5
  [79afa230] Continuables v1.0.1
  [f65535da] Convex v0.15.1
  [f68482b8] Cthulhu v1.6.1
  [a93c6f00] DataFrames v1.3.2
  [31c24e10] Distributions v0.25.53
  [634d3b9d] DrWatson v2.9.1
  [2fe49d83] Expectations v1.8.0
  [7a1cc6ca] FFTW v1.4.6
  [41a02a25] Folds v0.2.7
  [e9467ef8] GLMakie v0.5.5
  [c3a54625] JET v0.4.6
  [033835bb] JLD2 v0.4.22
  [4076af6c] JuMP v0.23.2
  [b964fa9f] LaTeXStrings v1.3.0
  [b4c46c6c] LinuxPerf v0.3.2
  [2ab3a3ac] LogExpFunctions v0.3.10
  [bdcacae8] LoopVectorization v0.12.103
  [23992714] MAT v0.10.3
  [b8f27783] MathOptInterface v1.1.1
  [eff96d63] Measurements v2.7.1
  [01bcebdf] Nonconvex v1.0.2
  [5fb14364] OhMyREPL v0.5.12
  [429524aa] Optim v1.6.2
  [e4faabce] PProf v2.1.3
  [9b87118b] PackageCompiler v2.0.5
  [91a5bcdd] Plots v1.27.3
  [c3e4b0f8] Pluto v0.18.4
  [7f904dfe] PlutoUI v0.7.38
  [92933f4c] ProgressMeter v1.7.2
  [295af30f] Revise v3.2.1
  [f2b01f46] Roots v1.4.0
  [276daf66] SpecialFunctions v2.1.4
  [90137ffa] StaticArrays v1.4.3
  [2913bbd2] StatsBase v0.33.16
  [f3b207a7] StatsPlots v0.14.33
  [0c5d862f] Symbolics v4.3.0
  [a759f4b9] TimerOutputs v0.5.16
  [37b6cedf] Traceur v0.3.1
  [28d57a85] Transducers v0.4.73
  [1986cc42] Unitful v1.11.0
  [e88e6eb3] Zygote v0.6.37

] instantiate and ] precompile executed successfully before all of this.

This was JuliaPlots/Makie.jl/issues/1817 .