Gadfly breaks convert(Array, DArray) on OS X Yosemite

[felixjung]

Hi,

I've run into a very serious issue with Gadfly after upgrading my Mac to OS X 10.10 Yosemite. However, the issue might be completely unrelated to this OS upgrade.

I'm on the Julia 0.3.1 binary for OS X and have tested this both using Gadfly 0.3.9 and the master branch.

Here is a description of the issue: I rely on parallel computations and store results in distributed arrays (DArrays). Once a DArray has been computed, I convert it to a regular Array for further use in my code.

Unfortunately, this conversion has stopped working and I have managed to track down the issue to Gadfly being loaded and Julia using more than 1 process. Here is a minimal working example:

## Load Gadfly
using Gadfly # If this line is present the below code will fail
             # when nprocs() > 1

## Work with DArray. These computations are completely unrelated to Gadfly.
addprocs(1)     # Add parallel cores (will trigger the issue upon convert() call)

darray = drandn((20, 20, 2), workers(), [1, 1, nworkers()])    # Create a DArray
array = convert(Array, darray)                                 # Convert DArray to Array (FAILS)

The thrown error is the following:

julia> include("/Users/Felix/Desktop/gadfly_darray_bug.jl")
Worker 2 terminated.ERROR: ProcessExitedException()
 in wait at /Applications/Julia-0.3.1.app/Contents/Resources/julia/lib/julia/sys.dylib (repeats 2 times)
 in wait at task.jl:48
 in sync_end at /Applications/Julia-0.3.1.app/Contents/Resources/julia/lib/julia/sys.dylib
 in convert at darray.jl:319
 in convert at abstractarray.jl:310
 in include at /Applications/Julia-0.3.1.app/Contents/Resources/julia/lib/julia/sys.dylib
 in include_from_node1 at /Applications/Julia-0.3.1.app/Contents/Resources/julia/lib/julia/sys.dylib
while loading /Users/Felix/Desktop/gadfly_darray_bug.jl, in expression starting on line 7
Worker 3 terminated.

To clearify: the code fails when I load Gadfly (using Gadfly) AND use n > 1 cores (addprocs(k) # k >= 1). It does not matter, if the DArray is constructed only by workers rather than all processes. The code will fail even when nprocs() = 2 and nworkers() = 1.

I'd really appreciate quick help with this. It might be a Julia bug on Yosemite, but given the direct involvement of Gadfly, I chose to post this here.

Thanks,

Felix

PS: here's my system configuration:

[felixjung]

Maybe I should loop in some people, @StefanKarpinski, @staticfloat

[darwindarak]

I think I'm running into a related problem with a single process Julia 0.3.3 running master branch of Gadfly on Mac OS X Yosemite, except it's not just with conversion between Array and DArray.
In my case, if I load Gadfly before reading in a file using HDF5/JLD, an error pops up complaining about missing methods for convert:

julia> using Gadfly

julia> using HDF5, JLD

julia> load("./kinparams.jld")
ERROR: `convert` has no method matching convert(::Type{Int64...}, ::Uint64)
 in convert at base.jl:13
 in convert at base.jl:21
 in read at /Users/darwin/.julia/v0.3/HDF5/src/JLD.jl:301
 in read_ref at /Users/darwin/.julia/v0.3/HDF5/src/JLD.jl:441
 in jlconvert! at /Users/darwin/.julia/v0.3/HDF5/src/jld_types.jl:431
 in read_scalar at /Users/darwin/.julia/v0.3/HDF5/src/JLD.jl:327
 in read at /Users/darwin/.julia/v0.3/HDF5/src/JLD.jl:299
 in read at /Users/darwin/.julia/v0.3/HDF5/src/JLD.jl:284
 in anonymous at /Users/darwin/.julia/v0.3/HDF5/src/JLD.jl:932
 in jldopen at /Users/darwin/.julia/v0.3/HDF5/src/JLD.jl:205
 in load at /Users/darwin/.julia/v0.3/HDF5/src/JLD.jl:931

The file loads fine if I don't use Gadfly though.

[timholy]

The latter seems likely to be https://github.com/JuliaLang/julia/issues/8631. You can test by using the procedure outlined in https://github.com/JuliaLang/Color.jl/issues/68.

[darwindarak]

You're right! The same problem is triggered if I do using Color instead. Sorry for the noise.

[felixjung]

@timholy so by your assessment, my issue is related to something else?

[timholy]

No clue. You, too, could try pinning Color and see what happens.

[timholy]

@darwindarak, it's not noise at all. This is a pretty unfortunate problem.

[timholy]

Try Pkg.update() and see if this problem is fixed. If you've pinned Color, you'll have to say Pkg.free("Color") first.

[felixjung]

Thanks for the update. Unfortunately, the problem persists after running Pkg.update() and making sure all packages are free. Here's the output from my above minimal example:

Worker 2 terminated.ERROR: ProcessExitedException()
 in wait at /Applications/Julia-0.3.1.app/Contents/Resources/julia/lib/julia/sys.dylib (repeats 2 times)
 in wait at task.jl:48
 in sync_end at /Applications/Julia-0.3.1.app/Contents/Resources/julia/lib/julia/sys.dylib
 in convert at darray.jl:319
 in convert at abstractarray.jl:310