search

JuliaImages / Images.jl

An image library for Julia
http://juliaimages.org/
Other
529 stars 141 forks source link

imresize spuriously causes `NaN`s with 4d image #763

Open staticfloat opened 5 years ago

staticfloat commented 5 years ago

With a 4-dimensional array:

m = randn(250,202,3,1)
imresize(m, (277, 224, 3, 1))

Running the imresize() multiple times shows NaN's randomly appearing. If I eliminate the fourth dimension, this works just fine:

imresize(m[:,:,:,1], (277, 224, 3))

I assume this is because imresize() is attempting to interpolate across the 3rd axis, and it's reaching out of bounds or something like that. If that is the case, it would be nice if I had a way to specify the dimensions across which I wanted to resize (which I am attempting to do by passing in an explicit tuple of sizes).

staticfloat commented 5 years ago

I should note the "4d image" I have is actually a tensor representing a 250 x 202 image with 3 channels, and there is only 1 within this current "batch".

johnnychen94 commented 5 years ago

For more information, I tried this in two of my computers with

any(imresize(m, (277, 224, 3, 1)) .== NaN)

For MacOS I didn't observe NaN, i.e., get result as false

Julia Version 1.0.0
Commit 5d4eaca0c9 (2018-08-08 20:58 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin14.5.0)
  CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, skylake)

Interpolation Version: v0.10.5

For Linux: I get Segmentation fault on my machine (imresize(m[:,:,:,1], (277, 224, 3)) won't)

Julia Version 1.0.0
Commit 5d4eaca0c9 (2018-08-08 20:58 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2678 v3 @ 2.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, haswell)

Interpolation Version: v0.10.5
signal (11): Segmentation fault
in expression starting at no file:0
getindex at ./array.jl:732 [inlined]
interp_getindex at /home/jc/.julia/packages/Interpolations/gioj0/src/Interpolations.jl:290 [inlined]
macro expansion at /home/jc/.julia/packages/Interpolations/gioj0/src/Interpolations.jl:274 [inlined]
interp_getindex at /home/jc/.julia/packages/Interpolations/gioj0/src/Interpolations.jl:265 [inlined]
macro expansion at /home/jc/.julia/packages/Interpolations/gioj0/src/Interpolations.jl:274 [inlined]
interp_getindex at /home/jc/.julia/packages/Interpolations/gioj0/src/Interpolations.jl:265 [inlined]
macro expansion at /home/jc/.julia/packages/Interpolations/gioj0/src/Interpolations.jl:274 [inlined]
interp_getindex at /home/jc/.julia/packages/Interpolations/gioj0/src/Interpolations.jl:265 [inlined]
macro expansion at /home/jc/.julia/packages/Interpolations/gioj0/src/Interpolations.jl:274 [inlined]
interp_getindex at /home/jc/.julia/packages/Interpolations/gioj0/src/Interpolations.jl:265 [inlined]
_getindex at /home/jc/.julia/packages/Interpolations/gioj0/src/Interpolations.jl:230 [inlined]
getindex at ./abstractarray.jl:905 [inlined]
BSplineInterpolation at /home/jc/.julia/packages/Interpolations/gioj0/src/b-splines/indexing.jl:8 [inlined]
imresize! at /home/jc/.julia/packages/ImageTransformations/CGe4u/src/resizing.jl:265
imresize! at /home/jc/.julia/packages/ImageTransformations/CGe4u/src/resizing.jl:240 [inlined]
imresize at /home/jc/.julia/packages/ImageTransformations/CGe4u/src/resizing.jl:214
unknown function (ip: 0x7f43aeb36f36)
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1829
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2182
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:324
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:428
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:363 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:686
jl_interpret_toplevel_thunk_callback at /buildworker/worker/package_linux64/build/src/interpreter.c:799
unknown function (ip: 0xfffffffffffffffe)
unknown function (ip: 0x7f43a3e41c7f)
unknown function (ip: 0x2)
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:808
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:787
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/builtins.c:622
eval at ./boot.jl:319
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2182
eval_user_input at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:85
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:117 [inlined]
#28 at ./task.jl:259
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2182
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1536 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:268
unknown function (ip: 0xffffffffffffffff)
Allocations: 40634301 (Pool: 40616850; Big: 17451); GC: 92
fish: “julia” terminated by signal SIGSEGV (Address boundary error)
zygmuntszpak commented 5 years ago

With 

   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.7.0 (2018-08-08 06:46 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-w64-mingw32

after a single execution of 

m = randn(250,202,3,1)
imresize(m, (277, 224, 3, 1))

I get the following on Windows

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (i
n their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x15c43239 -- * at .\float.jl:399 [inlined]
macro expansion at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:274 [in
lined]
interp_getindex at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:265 [in
lined]
macro expansion at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:274 [in
lined]
interp_getindex at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:265 [in
lined]
macro expansion at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:274 [in
lined]
interp_getindex at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:265 [in
lined]
macro expansion at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:274 [in
lined]
interp_getindex at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:265 [in
lined]
_getindex at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:230 [inlined]
getindex at .\abstractarray.jl:913 [inlined]
BSplineInterpolation at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\b-splines\indexing.j
l:8 [inlined]
imresize! at C:\Users\Spock\.julia\packages\ImageTransformations\CGe4u\src\resizing.jl:265
in expression starting at C:\Users\Spock\.julia\dev\Images\src\debug.jl:111
getindex at .\array.jl:732 [inlined]
interp_getindex at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:290 [in
lined]
macro expansion at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:274 [in
lined]
interp_getindex at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:265 [in
lined]
macro expansion at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:274 [in
lined]
interp_getindex at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:265 [in
lined]
macro expansion at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:274 [in
lined]
interp_getindex at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:265 [in
lined]
macro expansion at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:274 [in
lined]
interp_getindex at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:265 [in
lined]
_getindex at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:230 [inlined]
getindex at .\abstractarray.jl:913 [inlined]
BSplineInterpolation at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\b-splines\indexing.j
l:8 [inlined]
imresize! at C:\Users\Spock\.julia\packages\ImageTransformations\CGe4u\src\resizing.jl:265
imresize! at C:\Users\Spock\.julia\packages\ImageTransformations\CGe4u\src\resizing.jl:240 [inlined]
imresize at C:\Users\Spock\.julia\packages\ImageTransformations\CGe4u\src\resizing.jl:214
unknown function (ip: 0000000015C41BA5)
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:1829
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2182
do_call at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:324
eval_value at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:428
eval_stmt_value at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:363 [in
lined]
eval_body at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:682
jl_interpret_toplevel_thunk_callback at /home/Administrator/buildbot/worker/package_win64/build/src\
interpreter.c:799
unknown function (ip: FFFFFFFFFFFFFFFE)
unknown function (ip: 0000000012553C5F)
unknown function (ip: 0000000000000000)
jl_toplevel_eval_flex at /home/Administrator/buildbot/worker/package_win64/build/src\toplevel.c:831
jl_parse_eval_all at /home/Administrator/buildbot/worker/package_win64/build/src\ast.c:841
include_string at .\loading.jl:1002
include_string at C:\Users\Spock\.julia\packages\CodeTools\hB4Hy\src\eval.jl:30
unknown function (ip: 0000000015C0879A)
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2182
#117 at C:\Users\Spock\.julia\packages\Atom\7rQ1O\src\eval.jl:94
withpath at C:\Users\Spock\.julia\packages\CodeTools\hB4Hy\src\utils.jl:30
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:1829
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2182 [inlined]
jl_apply at /home/Administrator/buildbot/worker/package_win64/build/src\julia.h:1538 [inlined]
jl_invoke at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:56
withpath at C:\Users\Spock\.julia\packages\Atom\7rQ1O\src\eval.jl:46 [inlined]
#116 at C:\Users\Spock\.julia\packages\Atom\7rQ1O\src\eval.jl:93 [inlined]
with_logstate at .\logging.jl:397
with_logger at .\logging.jl:493 [inlined]
#115 at C:\Users\Spock\.julia\packages\Atom\7rQ1O\src\eval.jl:92 [inlined]
hideprompt at C:\Users\Spock\.julia\packages\Atom\7rQ1O\src\repl.jl:85
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:1829
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2182
macro expansion at C:\Users\Spock\.julia\packages\Atom\7rQ1O\src\eval.jl:91 [inlined]
#114 at .\task.jl:85
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2182
jl_apply at /home/Administrator/buildbot/worker/package_win64/build/src\julia.h:1538 [inlined]
start_task at /home/Administrator/buildbot/worker/package_win64/build/src\task.c:268
Allocations: 42834304 (Pool: 42827894; Big: 6410); GC: 90

I'm not actually sure how to check which version of Interpolations I have installed, because ] st does not show it.

johnnychen94 commented 5 years ago

@zygmuntszpak It's listed in ~/.julia/environments/v1.0/Manifest.toml

timholy commented 5 years ago

Yikes!

If you launch julia with --check-bounds=yes, you get this:

julia> imresize(m, (277, 224, 3, 1))
ERROR: BoundsError: attempt to access 250×202×3×1 Array{Float64,4} at index [2, 2, 2, 0]
Stacktrace:
 [1] getindex at ./array.jl:732 [inlined]
 [2] interp_getindex at /home/tim/.julia/dev/Interpolations/src/Interpolations.jl:290 [inlined]
 [3] macro expansion at /home/tim/.julia/dev/Interpolations/src/Interpolations.jl:274 [inlined]
 [4] interp_getindex at /home/tim/.julia/dev/Interpolations/src/Interpolations.jl:265 [inlined]
 [5] macro expansion at /home/tim/.julia/dev/Interpolations/src/Interpolations.jl:274 [inlined]
 [6] interp_getindex at /home/tim/.julia/dev/Interpolations/src/Interpolations.jl:265 [inlined]
 [7] macro expansion at /home/tim/.julia/dev/Interpolations/src/Interpolations.jl:274 [inlined]
 [8] interp_getindex at /home/tim/.julia/dev/Interpolations/src/Interpolations.jl:265 [inlined]
 [9] macro expansion at /home/tim/.julia/dev/Interpolations/src/Interpolations.jl:274 [inlined]
 [10] interp_getindex at /home/tim/.julia/dev/Interpolations/src/Interpolations.jl:265 [inlined]
 [11] _getindex at /home/tim/.julia/dev/Interpolations/src/Interpolations.jl:230 [inlined]
 [12] getindex at ./abstractarray.jl:905 [inlined]
 [13] BSplineInterpolation at /home/tim/.julia/dev/Interpolations/src/b-splines/indexing.jl:8 [inlined]
 [14] imresize!(::Array{Float64,4}, ::Interpolations.BSplineInterpolation{Float64,4,Array{Float64,4},Interpolations.BSpline{Interpolations.Linear},NTuple{4,Base.OneTo{Int64}}}) at /home/tim/.julia/packages/ImageTransformations/CGe4u/src/resizing.jl:265
 [15] imresize! at /home/tim/.julia/packages/ImageTransformations/CGe4u/src/resizing.jl:240 [inlined]
 [16] imresize(::Array{Float64,4}, ::NTuple{4,Int64}) at /home/tim/.julia/packages/ImageTransformations/CGe4u/src/resizing.jl:214
 [17] top-level scope at none:0

I haven't had time to look into whether it would be better to fix this in Interpolations or in ImageTransformations (or both). Presumably throwing an error when any dimension has size 1 would be undesirable, right?

staticfloat commented 5 years ago

If I ask imresize() to go from a size of (A, B, C, D) -> (A/2, B/2, C, D), I feel like it shouldn't matter what C and D are; if they're the same then imresize() should just be a no-op along those axes. (I realize that may not be how imresize() is implemented, but that would be nice, IMO)

timholy commented 5 years ago

I'm a bit short on time, but if anyone wants to tackle this the fix here (or rather, in ImageTransformations) is to use NoInterp for those dimensions.

johnnychen94 commented 4 years ago

It now throws a more informative error, but still, this needs to be fixed in ImageTransformation.jl

julia> m = randn(250,202,3,1);
julia> imresize(m, (277, 224, 3, 1))
ERROR: ArgumentError: size (250, 202, 3, 1) is inconsistent with BSpline(Interpolations.Linear()), use NoInterp along singleton dimensions
Stacktrace:
 [1] (::Interpolations.var"#err_singleton#11")(::Array{Float64,4}, ::Interpolations.BSpline{Interpolations.Linear}) at /home/jc/.julia/packages/Interpolations/0zZ6m/src/b-splines/b-splines.jl:58
 [2] BSplineInterpolation at /home/jc/.julia/packages/Interpolations/0zZ6m/src/b-splines/b-splines.jl:65 [inlined]
 [3] interpolate(::Type{Float64}, ::Type{Float64}, ::Array{Float64,4}, ::Interpolations.BSpline{Interpolations.Linear}) at /home/jc/.julia/packages/Interpolations/0zZ6m/src/b-splines/b-splines.jl:133
 [4] interpolate at /home/jc/.julia/packages/Interpolations/0zZ6m/src/b-splines/b-splines.jl:152 [inlined]
 [5] imresize! at /home/jc/.julia/packages/ImageTransformations/7wC0C/src/resizing.jl:256 [inlined]
 [6] imresize(::Array{Float64,4}, ::NTuple{4,Int64}) at /home/jc/.julia/packages/ImageTransformations/7wC0C/src/resizing.jl:231
 [7] top-level scope at REPL[3]:1
scimas commented 4 years ago

Is there any workaround for this?

scimas commented 4 years ago

I was looking for functionality similar to PyTorch's nn.Upsample to use with Flux and although this isn't a complete replication of it, ~it works for my purposes~ fails to work with zygote.

using Interpolations

struct Upsampler
    ratio::Tuple{<:Real,<:Real}
end

function (up::Upsampler)(x)
    itp = interpolate(x, (BSpline(Linear()), BSpline(Linear()), NoInterp(), NoInterp()))
    ws = (size(x, 1) - 1) / (size(x, 1) * up.ratio[1] - 1)
    hs = (size(x, 2) - 1) / (size(x, 2) * up.ratio[2] - 1)
    itp(1:ws:size(x, 1), 1:hs:size(x, 2), 1:size(x, 3), 1:size(x, 4))
end

function Upsampler(r1, r2)
    r1 >= 1 || throw(DomainError(r1, "ratio must be >= 1"))
    r2 >= 1 || throw(DomainError(r2, "ratio must be >= 1"))
    Upsampler((r1, r2))
end

@functor Upsampler

up = Upsampler(2, 1.5);
x = rand(Float32, 4, 4, 3, 1);
out = up(x);
@assert size(out) == (8, 6, 3, 1)

It assumes the input is a 4d array with size (Width, Height, Channels, Batchsize), as expected by Flux, and performs linear interpolation.

In case someone stumbles upon this issue with a similar use case. Also let me know if any other better method is possible.

johnnychen94 commented 4 years ago

@scimas probably you want to subscribe https://github.com/FluxML/Flux.jl/pull/1136