julia_setup suddenly no longer working with a system image

[PaulWik]

We have been using a system image with julia_setup without problem for several months, but it suddenly started failing in our daily pipelines today. Further investigation has revealed that the issue is specific to trying to run with a system image, because we do not get the same problem when we just call JuliaCall::julia_setup() (meaning we can load a julia environment for use from inside R and then happily call julia functions that way).

It looks like something to do with RCall, and in particular with trying to precompile DataFrames (the latest version of which was released yesterday, which could tie in with our pipelines succeeding yesterday and then failing today).

Do you have any ideas why this would only cause an issue when trying to use a system image? Or any suggestions of things we could try or look at?

Because the problem seemed to relate to RCall, I did try with useRCall = FALSE but that didn't solve the problem

>   julia <-
+  JuliaCall::julia_setup(sysimage_path = sysImagePath)
Julia version 1.6.3 at location /opt/julia/bin will be used.
Loading setup script for JuliaCall...
WARNING: could not import DataAPI.allcombinations into DataFrames
WARNING: could not import DataAPI.metadata into DataFrames
WARNING: could not import DataAPI.metadatakeys into DataFrames
WARNING: could not import DataAPI.metadata! into DataFrames
WARNING: could not import DataAPI.deletemetadata! into DataFrames
WARNING: could not import DataAPI.emptymetadata! into DataFrames
WARNING: could not import DataAPI.colmetadata into DataFrames
WARNING: could not import DataAPI.colmetadatakeys into DataFrames
WARNING: could not import DataAPI.colmetadata! into DataFrames
WARNING: could not import DataAPI.deletecolmetadata! into DataFrames
WARNING: could not import DataAPI.emptycolmetadata! into DataFrames
WARNING: could not import Compat.keepat! into DataFrames
WARNING: could not import Compat.stack into DataFrames
ERROR: LoadError: LoadError: UndefVarError: metadatasupport not defined
Stacktrace:
  [1] getproperty(x::Module, f::Symbol)
    @ Base ./Base.jl:26
  [2] top-level scope
    @ ~/.julia/packages/DataFrames/XM9o2/src/other/metadata.jl:8
  [3] include(mod::Module, _path::String)
    @ Base ./Base.jl:386
  [4] include(x::String)
    @ DataFrames ~/.julia/packages/DataFrames/XM9o2/src/DataFrames.jl:1
  [5] top-level scope
    @ ~/.julia/packages/DataFrames/XM9o2/src/DataFrames.jl:172
  [6] include
    @ ./Base.jl:386 [inlined]
  [7] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{St                                                                                              ring}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
    @ Base ./loading.jl:1235
  [8] top-level scope
    @ none:1
  [9] eval
    @ ./boot.jl:360 [inlined]
 [10] eval(x::Expr)
    @ Base.MainInclude ./client.jl:446
 [11] top-level scope
    @ none:1
in expression starting at /root/.julia/packages/DataFrames/XM9o2/src/other/metadata.jl:8
in expression starting at /root/.julia/packages/DataFrames/XM9o2/src/DataFrames.jl:1
ERROR: LoadError: Failed to precompile DataFrames [a93c6f00-e57d-5684-b7b6-d8193f3e46c0] to /root/.julia/compiled/v1.6/DataFrames/jl_4NIE7W.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::Base.TTY, internal_stdout::Base.TTY, ignore_loaded_modules::Bool)
    @ Base ./loading.jl:1385
  [3] compilecache(pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1329
  [4] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1043
  [5] require(uuidkey::Base.PkgId)
    @ Base ./loading.jl:936
  [6] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:923
  [7] include
    @ ./Base.jl:386 [inlined]
  [8] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{St                                                                                              ring}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
    @ Base ./loading.jl:1235
  [9] top-level scope
    @ none:1
 [10] eval
    @ ./boot.jl:360 [inlined]
 [11] eval(x::Expr)
    @ Base.MainInclude ./client.jl:446
 [12] top-level scope
    @ none:1
in expression starting at /root/.julia/packages/RCall/6kphM/src/RCall.jl:2
LoadError("/usr/lib64/R/library/JuliaCall/julia/setup.jl", 16, ErrorException("Failed to precompile RCall [6f49c342-dc21-5d91-9882-a32aef131414                                                                                              ] to /root/.julia/compiled/v1.6/RCall/jl_8IIiOe.")) Error in .julia$cmd(paste0(Rhomeset, "Base.include(Main,\"", system.file("julia/setup.jl",                                                                                                :
  Error happens when you try to execute command ENV["R_HOME"] = "/usr/lib64/R";Base.include(Main,"/usr/lib64/R/library/JuliaCall/julia/setup.jl                                                                                              ") in Julia.
                        To have more helpful error messages,
                        you could considering running the command in Julia directly

Session Info

```r R version 4.1.3 (2022-03-10) Platform: x86_64-suse-linux-gnu (64-bit) Running under: openSUSE Leap 15.3 Matrix products: default BLAS/LAPACK: /usr/lib64/openblas-serial/libopenblas.so.0 locale: [1] LC_CTYPE=en_GB.UTF-8 LC_NUMERIC=C [3] LC_TIME=en_GB.UTF-8 LC_COLLATE=en_GB.UTF-8 [5] LC_MONETARY=en_GB.UTF-8 LC_MESSAGES=en_GB.UTF-8 [7] LC_PAPER=en_GB.UTF-8 LC_NAME=en_GB.UTF-8 [9] LC_ADDRESS=en_GB.UTF-8 LC_TELEPHONE=en_GB.UTF-8 [11] LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=en_GB.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base ```

[PaulWik]

Thanks to some kind help from @bkamins, this issue is because of a mismatch in the versions of DataAPI and Compat we had in our system image and the julia environment that JuliaCall created when we called julia_setup.

I therefore think that what happened was that after the release of DataFrames.jl when we tried to call julia_setup in a fresh/clean docker image, it ended up pulling in that latest version of DataFrames.jl, and this was incompatible with those old versions of DataAPI and Compat that we had in the system image, hence the problems and error messages.

Now that I have updated those packages in the system image, the problem has gone away.

[bkamins]

Now that I have updated those packages in the system image, the problem has gone away.

Great! I hope the packages serve you well.