search

JuliaImages / Images.jl

An image library for Julia
http://juliaimages.org/
Other
535 stars 142 forks source link

Build segfaults on OS X #188

Closed jwmerrill closed 10 years ago

jwmerrill commented 10 years ago

I'm on OS X 10.9.5, XCode Version 6.0.1 (6A317), and Julia Version 0.3.0 (2014-08-20 20:43 UTC).

This may be related to upgrading to XCode 6, which the OS helpfully did for me without asking...

signal (11): Segmentation fault: 11
MagickQueryConfigureOption at /Users/jwm/.julia/v0.3/Homebrew/deps/usr/lib/libMagickWand-6.Q16.dylib (unknown line)
anonymous at no file:0
jl_apply at /Users/jwm/src/julia/src/./julia.h:980
jl_eval_module_expr at /Users/jwm/src/julia/src/toplevel.c:140
jl_toplevel_eval_flex at /Users/jwm/src/julia/src/toplevel.c:386
jl_parse_eval_all at /Users/jwm/src/julia/src/toplevel.c:542
jl_load at /Users/jwm/src/julia/src/toplevel.c:579
include at ./boot.jl:245
jl_apply at /Users/jwm/src/julia/src/./julia.h:980
include_from_node1 at ./loading.jl:128
jl_apply at /Users/jwm/src/julia/src/./julia.h:980
jl_apply at /Users/jwm/src/julia/src/interpreter.c:59
eval at /Users/jwm/src/julia/src/interpreter.c:207
jl_toplevel_eval_flex at /Users/jwm/src/julia/src/toplevel.c:494
jl_toplevel_eval_flex at /Users/jwm/src/julia/src/toplevel.c:441
jl_f_top_eval at /Users/jwm/src/julia/src/builtins.c:404
evalfile at loading.jl:168
jl_apply at /Users/jwm/src/julia/src/./julia.h:980
anonymous at pkg/entry.jl:615
cd at ./file.jl:20
jl_apply at /Users/jwm/src/julia/src/./julia.h:980
build! at pkg/entry.jl:614
build at pkg/entry.jl:626
jl_apply at /Users/jwm/src/julia/src/gf.c:1416
resolve at ./pkg/entry.jl:384
edit at pkg/entry.jl:24
jl_apply at /Users/jwm/src/julia/src/gf.c:1416
anonymous at task.jl:340
jl_apply at /Users/jwm/src/julia/src/task.c:427
start_task at /Users/jwm/src/julia/src/task.c:426
start_task at /Users/jwm/src/julia/src/task.c:426
julia_trampoline at /Users/jwm/src/julia/src/init.c:1007
fish: Job 1, 'julia' terminated by signal SIGSEGV (Address boundary error)
timholy commented 10 years ago

It suggests your ImageMagick installation is borked. If you're not relying on ImageMagick, this is "harmless" but obviously not acceptable. Do you already know that there is something wrong with your ImageMagick install? Does imread(someimage, Images.ImageMagick) work for you?

jwmerrill commented 10 years ago

I can't try imread(someimage, Images.ImageMagick) because this error blocks installing the Images package for me.

jwmerrill commented 10 years ago

Sorry, I probably should have reported a little more context around this error. It happens when I do Pkg.add("Images")

julia> Pkg.add("Images")
INFO: Installing BinDeps v0.3.5
INFO: Installing Homebrew v0.1.10
INFO: Installing Images v0.4.12
INFO: Installing SHA v0.0.3
INFO: Installing SIUnits v0.0.2
INFO: Installing TexExtensions v0.0.2
INFO: Installing URIParser v0.0.2
INFO: Installing Zlib v0.1.7
INFO: Building Homebrew
remote: Counting objects: 10, done.
remote: Compressing objects: 100% (6/6), done.
Unpacking objects: 100% (10/10), done.
remote: Total 10 (delta 8), reused 6 (delta 4)
From https://github.com/Homebrew/homebrew
   399de3b..5e99f01  master     -> origin/master
HEAD is now at 5e99f01 ledger: update 3.0.3 bottle.
HEAD is now at 73c6cfe Bottle cbc
INFO: Building Images

signal (11): Segmentation fault: 11
MagickQueryConfigureOption at /Users/jwm/.julia/v0.3/Homebrew/deps/usr/lib/libMagickWand-6.Q16.dylib (unknown line)
anonymous at no file:0
jl_apply at /Users/jwm/src/julia/src/./julia.h:980
jl_eval_module_expr at /Users/jwm/src/julia/src/toplevel.c:140
jl_toplevel_eval_flex at /Users/jwm/src/julia/src/toplevel.c:386
jl_parse_eval_all at /Users/jwm/src/julia/src/toplevel.c:542
jl_load at /Users/jwm/src/julia/src/toplevel.c:579
include at ./boot.jl:245
jl_apply at /Users/jwm/src/julia/src/./julia.h:980
include_from_node1 at ./loading.jl:128
jl_apply at /Users/jwm/src/julia/src/./julia.h:980
jl_apply at /Users/jwm/src/julia/src/interpreter.c:59
eval at /Users/jwm/src/julia/src/interpreter.c:207
jl_toplevel_eval_flex at /Users/jwm/src/julia/src/toplevel.c:494
jl_toplevel_eval_flex at /Users/jwm/src/julia/src/toplevel.c:441
jl_f_top_eval at /Users/jwm/src/julia/src/builtins.c:404
evalfile at loading.jl:168
jl_apply at /Users/jwm/src/julia/src/./julia.h:980
anonymous at pkg/entry.jl:615
cd at ./file.jl:20
jl_apply at /Users/jwm/src/julia/src/./julia.h:980
build! at pkg/entry.jl:614
build at pkg/entry.jl:626
jl_apply at /Users/jwm/src/julia/src/./julia.h:980
resolve at ./pkg/entry.jl:384
edit at pkg/entry.jl:24
jl_apply at /Users/jwm/src/julia/src/gf.c:1416
anonymous at task.jl:340
jl_apply at /Users/jwm/src/julia/src/task.c:427
start_task at /Users/jwm/src/julia/src/task.c:426
start_task at /Users/jwm/src/julia/src/task.c:426
julia_trampoline at /Users/jwm/src/julia/src/init.c:1007
fish: Job 1, 'julia' terminated by signal SIGSEGV (Address boundary error)
timholy commented 10 years ago

The segfault occurs at the very last step of installation, and doesn't affect anything other than 1 line of deps/deps.jl (writing the version number of your ImageMagick installation). So you can go ahead and try to use Images, and see what happens.

lucasb-eyer commented 10 years ago

The following is a wild guess.

I think this line in MagickQueryConfigureOption will crash if an option which doesn't exist is being queried. I think so because the GetConfigureInfoList called above doesn't seem to return NULL when there's 0 matching options, but just sets number_options to 0 (and returns a list whose first entry is NULL), which isn't checked by MagickQueryConfigureOption. So maybe some ImageMagick versions on Mac OS X don't come with the LIB_VERSION_NUMBER option? Maybe we should be using either VERSION or LIB_VERSION? (Source) I couldn't find an authoritative source as for the valid values. I think it's worth a try, but I don't know how to try it; how do I access libwand in the REPL?

timholy commented 10 years ago

Ah! That's a very interesting guess. And easy to check: see the comment at the top of this code for how to list all available options.

lucasb-eyer commented 10 years ago

Ah nice, you even wrapped that!

@jwmerrill could you please copy-paste the output of:

  1. Images.LibMagick.queryoptions("*")
  2. Images.LibMagick.queryoption("VERSION")
  3. Images.LibMagick.queryoption("LIB_VERSION")
  4. Images.LibMagick.queryoption("LIB_VERSION_NUMBER")
jwmerrill commented 10 years ago 
julia> print(Images.LibMagick.queryoptions("*"))
ASCIIString[
  "CC",
  "CFLAGS",
  "CODER_PATH",
  "CONFIGURE",
  "CONFIGURE_PATH",
  "COPYRIGHT",
  "CPPFLAGS",
  "CXX",
  "CXXFLAGS",
  "DEFS",
  "DELEGATES",
  "DISTCHECK_CONFIG_FLAGS",
  "DOCUMENTATION_PATH",
  "EXEC-PREFIX",
  "EXECUTABLE_PATH",
  "FEATURES",
  "FEATURES",
  "FILTER_PATH",
  "HOST",
  "INCLUDE_PATH",
  "LDFLAGS",
  "LIB_VERSION",
  "LIB_VERSION_NUMBER",
  "LIBRARY_PATH",
  "LIBS","NAME",
  "NAME",
  "PCFLAGS",
  "PREFIX",
  "QuantumDepth",
  "QuantumDepth",
  "RELEASE_DATE",
  "SHARE_PATH",
  "SHAREARCH_PATH",
  "SVN_REVISION",
  "TARGET_CPU",
  "TARGET_OS",
  "TARGET_VENDOR",
  "VERSION",
  "WEBSITE"
]
julia> print(Images.LibMagick.queryoption("VERSION"))
6.8.9
julia> print(Images.LibMagick.queryoption("LIB_VERSION"))
0x689
julia> print(Images.LibMagick.queryoption("LIB_VERSION_NUMBER"))
6,8,9,1
lucasb-eyer commented 10 years ago

So that proves my guess wrong. Now the big question is why did the last line not crash? I have no idea.

timholy commented 10 years ago

Yeah, that makes no sense whatsoever; how the heck did that line succeed when the build segfaulted??

@jwmerrill, can I ask you to try Pkg.build("Images") several times in a row? Does it always segfault? What if you insert a sleep(1.0) before this line?

jwmerrill commented 10 years ago

It segfaults every time, even with the sleep. Here's the contents of my deps.jl, on the off chance that that's helpful:

macro checked_lib(libname, path)
        (dlopen_e(path) == C_NULL) && error("Unable to load \n\n$libname ($path)\n\nPlease re-run Pkg.build(package), and restart Julia.")
        quote const $(esc(libname)) = $path end
    end
@checked_lib libwand "/Users/jwm/.julia/v0.3/Homebrew/deps/usr/lib/libMagickWand-6.Q16.dylib"
function __init__()
    ENV["MAGICK_CONFIGURE_PATH"] = joinpath("/Users/jwm/.julia/v0.3/Homebrew/deps/usr/Cellar/imagemagick/6.8.9-1","lib","ImageMagick","config-Q16")
    ENV["MAGICK_CODER_MODULE_PATH"] = joinpath("/Users/jwm/.julia/v0.3/Homebrew/deps/usr/Cellar/imagemagick/6.8.9-1", "lib","ImageMagick","modules-Q16","coders")
end
jwmerrill commented 10 years ago

I was wondering if this line might be the secret sauce that allows the ccall to not segfault, but I tried manually doing

julia> include("deps.jl")
__init__ (generic function with 1 method)

julia> ccall((:MagickWandGenesis, libwand), Void, ())

julia> ccall((:MagickQueryConfigureOption, libwand), Ptr{Uint8}, (Ptr{Uint8},), "LIB_VERSION_NUMBER")

signal (11): Segmentation fault: 11

and got the same segfault.

I can, however, do

julia> using Images

julia> Images.LibMagick.libwand
"/Users/jwm/.julia/v0.3/Homebrew/deps/usr/lib/libMagickWand-6.Q16.dylib"

julia> ccall((:MagickQueryConfigureOption, Images.LibMagick.libwand), Ptr{Uint8}, (Ptr{Uint8},), "LIB_VERSION_NUMBER")
Ptr{Uint8} @0x00007f8c2d8c0600

so it seems like there has to be some other kind of initialization that runs during using Images that is not running during the build.

timholy commented 10 years ago

Hmm, you're right that just including deps.jl doesn't do the call to MagickWandGenesis.

What if you change the MagickQueryConfigureOption to something more harmless, like https://github.com/timholy/Images.jl/blob/a98e2939ac88983e1eb30e8c9eefc6e8d7de2e88/src/ioformats/libmagickwand.jl#L288. (You don't have to do anything with the output, just see whether it segfaults.)

staticfloat commented 10 years ago

I'm commenting here to follow along, in case this ends up being a problem with the binaries. No debugging wisdom to share at the moment however. ;)

timholy commented 10 years ago

Glad to have you with us, @staticfloat!

Upon close analysis, there are some puzzling things about the timing of events, but I may have the solution. From my reading, here's the sequence of what happens:

So here's my suspicion: without first running the __init__ defined in your deps.jl file, the ccall to MagickWandGenesis is actually failing, but because that call returns void there's no hint that it failed. So change these lines to

const filewand = joinpath(splitdir(@__FILE__)[1], "src", "ioformats", "libmagickwand.jl")
@show filewand  # this is just to see whether I've screwed up the path manipulation above
include(filewand)
vstr0 = LibMagick.queryoption("LIB_VERSION_NUMBER")
vstr = string("v\"", join(split(vstr0, ',')[1:3], '.'), "\"")

EDIT: then see whether Pkg.build("Images") works.

jwmerrill commented 10 years ago

I had to change line 1 of your patch to this:

const filewand = normpath(joinpath(splitdir(@__FILE__)[1], "..", "src", "ioformats", "libmagickwand.jl"))

but that just netted me this:

julia> Pkg.build("Images")
INFO: Building Homebrew
HEAD is now at 3c9dabb Update for the real pypy 2.4.0 final
HEAD is now at 73c6cfe Bottle cbc
INFO: Building Images
filewand => "/Users/jwm/.julia/v0.3/Images/src/ioformats/libmagickwand.jl"
===============================[ ERROR: Images ]================================

ColorTypes not defined
while loading /Users/jwm/.julia/v0.3/Images/src/ioformats/libmagickwand.jl, in expression starting on line 3
while loading /Users/jwm/.julia/v0.3/Images/deps/build.jl, in expression starting on line 78

================================================================================

================================[ BUILD ERRORS ]================================

WARNING: Images had build errors.

 - packages with build errors remain installed in /Users/jwm/.julia/v0.3
 - build a package and all its dependencies with `Pkg.build(pkg)`
 - build a single package by running its `deps/build.jl` script

================================================================================

I think more of the module system is necessary for loading that file.

jwmerrill commented 10 years ago

So this seems to work:

module CheckVersion
include("deps.jl")()
p = ccall((:MagickQueryConfigureOption, libwand), Ptr{Uint8}, (Ptr{Uint8},), "LIB_VERSION_NUMBER")
vstr = string("v\"", join(split(bytestring(p), ',')[1:3], '.'), "\"")
open("deps.jl", "a") do file
    write(file, "const libversion = $vstr\n")
end

I.e. calling the result of loading deps.jl.

Total cargo culting on my part, though--I don't really know the meaning of what I'm doing there.

jwmerrill commented 10 years ago

I guess that __init__ function is the last thing in the file, so including the file returns that function.

Amusingly, after

open("deps.jl", "a") do file
    write(file, "const libversion = $vstr\n")
end

runs, __init__ is no longer the last value in deps.jl.

timholy commented 10 years ago

Looks like we collided on fix commit; sorry, I should have left that to you. (But it did need to be wrapped in @osx_only, since no other platform has __init__ defined in deps.jl.)

lucasb-eyer commented 10 years ago

Though that now makes me wonder why it initially failed when called at runtime?

timholy commented 10 years ago

I'm pretty puzzled by that one, too.

rennis250 commented 10 years ago

I'm a bit confused. Was this a complete fix? I now receive this, even after nuking .julia:

=============================[ ERROR: Images ]==============================

__init__ not defined
while loading /Users/rje/.julia/v0.4/Images/deps/build.jl, in expression starting on line 77

============================================================================

OS X 10.9.5, Xcode 6.0.1 (6A317), Julia Version 0.4.0-dev+1289

Tried to do some debugging, but no luck.

Best, Rob

timholy commented 10 years ago

What are the contents of deps/deps.jl?

rennis250 commented 10 years ago 
macro checked_lib(libname, path)
        (dlopen_e(path) == C_NULL) && error("Unable to load \n\n$libname ($path)\n\nPlease re-run Pkg.build(package), and restart Julia.")
        quote const $(esc(libname)) = $path end
    end
@checked_lib libwand "/usr/local/lib/libMagickWand-6.Q16.dylib"

Might it have something to do with the last line referring to my system-wide install of ImageMagick?

Best, R

Edit: No, I suspect that doesn't make sense... That should be perfectly fine.

timholy commented 10 years ago

That part is great. The curious bit is that you don't have the definition of __init__ shown in https://github.com/timholy/Images.jl/issues/188#issuecomment-56418102. What does your build.jl look like? Are you somehow bypassing Homebrew.jl?

timholy commented 10 years ago

See 13d360380f4b8b7f9be99593436a57afda52579e. Can you do Pkg.checkout("Images"); Pkg.update(); Pkg.build("Images") and see if it works now?

rennis250 commented 10 years ago

Perfecto! All good here again. Curious that it somehow doesn't exist following the @osx_only check... Is there something about the macro that runs init() in a different scope or doesn't check the surrounding namespace?

Best, Rob

timholy commented 10 years ago

No, the problem was that it was following the @osx_only check, and therefore it tried to run __init__. On your system, likely because you have a system-wide installation of ImageMagick, the build script decided it wasn't necessary to write the __init__ function to deps.jl.

rennis250 commented 10 years ago

Ah, okay, I see. Thanks, again.

R