search

JuliaMath / Interpolations.jl

Fast, continuous interpolation of discrete datasets in Julia
http://juliamath.github.io/Interpolations.jl/
Other
533 stars 109 forks source link

ImageTransformations issues #434

Closed SomTambe closed 3 years ago

SomTambe commented 3 years ago

I am writing backward passes for ImageTransformations.warp! and other Image agnostic functions. I have been solving most of the problems related to adjoints for Images, but after all I have gotten stuck on the Interpolations part.

I have arrived at this moment where I have this - 

julia> gradient((x,y)->rgb(sum(warp_me(x,y))), src2, h)
mn = [-348.0480263428803, -74.61757209029507]
mx = [-150.32291782157546, 116.97325016644439]
mn = [-348.0480263428803, -74.61757209029507]
mx = [-150.32291782157546, 116.97325016644439]
typeof(Δ) = FillArrays.Fill{NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}}, 2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}
size(Δ) = (200, 193)
out_p.offsets = (-350, -76)
typeof(Δ) = OffsetMatrix{NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}}, Matrix{NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}}}}
ERROR: MethodError: no method matching getfirst(::RGB{N0f8})

I looked up the getfirst functions but they were literally giving out the first elements of a tuple. So I defined it accordingly for RGB -

julia> Interpolations.getfirst(f::RGB{T}) where T = f.r

Now it went ahead and gave me this -

julia> gradient((x,y)->rgb(sum(warp_me(x,y))), src2, h)
mn = [-348.0480263428803, -74.61757209029507]
mx = [-150.32291782157546, 116.97325016644439]
mn = [-348.0480263428803, -74.61757209029507]
mx = [-150.32291782157546, 116.97325016644439]
typeof(Δ) = FillArrays.Fill{NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}}, 2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}
size(Δ) = (200, 193)
out_p.offsets = (-350, -76)
typeof(Δ) = OffsetMatrix{NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}}, Matrix{NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}}}}
ERROR: MethodError: no method matching getrest(::RGB{N0f8})

So this was also obvious for me, I defined it similarly for RGB - 

julia> Interpolations.getrest(f::RGB{T}) where T = (f.g, f.b)

I thought this would be enough, but it sure was not enough -

julia> gradient((x,y)->rgb(sum(warp_me(x,y))), src2, h)
mn = [-348.0480263428803, -74.61757209029507]
mx = [-150.32291782157546, 116.97325016644439]
mn = [-348.0480263428803, -74.61757209029507]
mx = [-150.32291782157546, 116.97325016644439]
typeof(Δ) = FillArrays.Fill{NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}}, 2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}
size(Δ) = (200, 193)
out_p.offsets = (-350, -76)
typeof(Δ) = OffsetMatrix{NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}}, Matrix{NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}}}}
ERROR: MethodError: no method matching inbounds_index(::N0f8, ::Tuple{Int64, Int64}, ::Float64, ::Interpolations.FilledExtrapolation{RGB{N0f8}, 2, Interpolations.BSplineInterpolation{RGB{N0f8}, 2, Matrix{RGB{N0f8}}, BSpline{Linear{Throw{OnGrid}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, BSpline{Linear{Throw{OnGrid}}}, RGB{N0f8}}, ::Tuple{Float64, Float64})

I am completely stuck at this moment. If the stacktrace helps, I would give it in the thread upon request, and will also give the other functions as well as the adjoints I have written for them. But if anyone could help me as to how I can go ahead from here, it would be really great!

I'd like to know in the first place why was I getting the getfirst error, because I have observed that Interpolations was Images-compatible. There must be some place I must be going south, and I can't figure it out.

johnnychen94 commented 3 years ago

Because ImageTransformations.warp involves many other types and functions, e.g., SVector, Transformations, and a for loop. It would be clearer if you can provide a reproducible and minimal example so that we can figure out if it is an Interpolations issue or an ImageTransformations issue, or whatever.

See #337 as an example. For example, does something as simple as l = itp(x).r enough to reproduce this issue?

SomTambe commented 3 years ago

Yes. Here is how to reproduce it. This is the file with the code - https://gist.github.com/SomTambe/5309dd7dff491d8770237799bfbfa9e1

julia> src2 = imresize(src_img,ratio=1/4)

julia> gradient((x,y)->rgb(sum(warp_me(x,y))), src2, h)

I hope this helps.

I am using the master branch of Interpolations.

DhairyaLGandhi commented 3 years ago

What's the dev branch of Interpolations?

SomTambe commented 3 years ago

What's the dev branch of Interpolations?

Sorry, I meant the master branch.

SomTambe commented 3 years ago

I digged a bit deeper, and I found over here, according to what inbounds_index is defined, the first index should always be of the type Interpolations.Flag. But I have defined the getfirst and getlast functions which return Float64s.

How should I define getfirst and getrest functions according to the current scenario? I have little idea about the DataType Interpolations.Flag. cc. @DhairyaLGandhi @johnnychen94

mkitti commented 3 years ago

getfirst should just pop off the first element. getrest should return the remaining elements. It's an old iteration method from Scheme/Lisp. See car and cdr. I don't think there is a getlast involved here.

https://github.com/JuliaMath/Interpolations.jl/blob/af35e779a7c39e540555d2c35bc1b2ccde242bc2/src/utils.jl#L17-L20

mkitti commented 3 years ago

The full stacktrace would be helpful.

SomTambe commented 3 years ago

Yup sorry, it should have been getrest instead of getlast (as in here), my apologies.

The full stacktrace is here - 

ERROR: MethodError: no method matching inbounds_index(::Float64, ::Tuple{Int64, Int64}, ::Float64, ::Interpolations.FilledExtrapolation{RGB{N0f8}, 2, Interpolations.BSplineInterpolation{RGB{N0f8}, 2, Matrix{RGB{N0f8}}, BSpline{Linear{Throw{OnGrid}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, BSpline{Linear{Throw{OnGrid}}}, RGB{N0f8}}, ::Tuple{Float64, Float64})
Closest candidates are:
  inbounds_index(::Throw, ::Any, ::Any, ::Any, ::Any) at /home/somvt/.julia/dev/Interpolations/src/extrapolation/extrapolation.jl:111
  inbounds_index(::Periodic, ::Any, ::Any, ::Any, ::Any) at /home/somvt/.julia/dev/Interpolations/src/extrapolation/extrapolation.jl:115
  inbounds_index(::Reflect, ::Any, ::Any, ::Any, ::Any) at /home/somvt/.julia/dev/Interpolations/src/extrapolation/extrapolation.jl:116
  ...
Stacktrace:
  [1] inbounds_position(eflag::Tuple{Float64, Float64}, bounds::Tuple{Tuple{Int64, Int64}, Tuple{Int64, Int64}}, x::Tuple{Float64, Float64}, etp::Interpolations.FilledExtrapolation{RGB{N0f8}, 2, Interpolations.BSplineInterpolation{RGB{N0f8}, 2, Matrix{RGB{N0f8}}, BSpline{Linear{Throw{OnGrid}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, BSpline{Linear{Throw{OnGrid}}}, RGB{N0f8}}, xN::Tuple{Float64, Float64})
    @ Interpolations ~/.julia/dev/Interpolations/src/extrapolation/extrapolation.jl:103
  [2] gradient
    @ ~/.julia/dev/Interpolations/src/extrapolation/extrapolation.jl:72 [inlined]
  [3] (::Interpolations.var"#pullback#84"{Interpolations.FilledExtrapolation{RGB{N0f8}, 2, Interpolations.BSplineInterpolation{RGB{N0f8}, 2, Matrix{RGB{N0f8}}, BSpline{Linear{Throw{OnGrid}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, BSpline{Linear{Throw{OnGrid}}}, RGB{N0f8}}, Tuple{Float64, Float64}})(Δy::ChainRulesCore.Tangent{Any, NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}}})
    @ Interpolations ~/.julia/dev/Interpolations/src/chainrules/chainrules.jl:9
  [4] ZBack
    @ ~/.julia/packages/Zygote/zowrf/src/compiler/chainrules.jl:77 [inlined]
  [5] (::Zygote.var"#188#189"{Tuple{Tuple{Nothing, Nothing}}, Zygote.ZBack{Interpolations.var"#pullback#84"{Interpolations.FilledExtrapolation{RGB{N0f8}, 2, Interpolations.BSplineInterpolation{RGB{N0f8}, 2, Matrix{RGB{N0f8}}, BSpline{Linear{Throw{OnGrid}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, BSpline{Linear{Throw{OnGrid}}}, RGB{N0f8}}, Tuple{Float64, Float64}}}})(Δ::NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}})
    @ Zygote ~/.julia/packages/Zygote/zowrf/src/lib/lib.jl:194
  [6] (::Zygote.var"#1689#back#190"{Zygote.var"#188#189"{Tuple{Tuple{Nothing, Nothing}}, Zygote.ZBack{Interpolations.var"#pullback#84"{Interpolations.FilledExtrapolation{RGB{N0f8}, 2, Interpolations.BSplineInterpolation{RGB{N0f8}, 2, Matrix{RGB{N0f8}}, BSpline{Linear{Throw{OnGrid}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, BSpline{Linear{Throw{OnGrid}}}, RGB{N0f8}}, Tuple{Float64, Float64}}}}})(Δ::NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}})
    @ Zygote ~/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:59
  [7] Pullback
    @ ~/.julia/packages/ImageTransformations/xYRLH/src/ImageTransformations.jl:37 [inlined]
  [8] Pullback
    @ ~/work/juliatests/test_warp.jl:34 [inlined]
  [9] (::typeof(∂(λ)))(Δ::NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}})
    @ Zygote ~/.julia/packages/Zygote/zowrf/src/compiler/interface2.jl:0
 [10] Pullback
    @ ~/work/juliatests/test_warp.jl:40 [inlined]
 [11] (::typeof(∂(λ)))(Δ::NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}})
    @ Zygote ~/.julia/packages/Zygote/zowrf/src/compiler/interface2.jl:0
 [12] (::Zygote.var"#561#565")(f::typeof(∂(λ)), δ::NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}})
    @ Zygote ~/.julia/packages/Zygote/zowrf/src/lib/array.jl:226
 [13] (::Base.var"#4#5"{Zygote.var"#561#565"})(a::Tuple{typeof(∂(λ)), NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}}})
    @ Base ./generator.jl:36
 [14] iterate
    @ ./generator.jl:47 [inlined]
 [15] collect
    @ ./array.jl:678 [inlined]
 [16] map
    @ ./abstractarray.jl:2383 [inlined]
 [17] (::Zygote.var"#560#564"{OffsetMatrix{typeof(∂(λ)), Matrix{typeof(∂(λ))}}})(Δ::OffsetMatrix{NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}}, Matrix{NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}}}})
    @ Zygote ~/.julia/packages/Zygote/zowrf/src/lib/array.jl:226
 [18] (::Zygote.var"#2572#back#566"{Zygote.var"#560#564"{OffsetMatrix{typeof(∂(λ)), Matrix{typeof(∂(λ))}}}})(Δ::OffsetMatrix{NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}}, Matrix{NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}}}})
    @ Zygote ~/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:59
 [19] Pullback
    @ ~/work/juliatests/test_warp.jl:40 [inlined]
 [20] (::typeof(∂(λ)))(Δ::OffsetMatrix{NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}}, Matrix{NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}}}})
    @ Zygote ~/.julia/packages/Zygote/zowrf/src/compiler/interface2.jl:0
 [21] (::Zygote.var"#41#42"{typeof(∂(λ))})(Δ::OffsetMatrix{NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}}, Matrix{NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}}}})
    @ Zygote ~/.julia/packages/Zygote/zowrf/src/compiler/interface.jl:41
 [22] (::var"#Δ_warp!#13"{Interpolations.FilledExtrapolation{RGB{N0f8}, 2, Interpolations.BSplineInterpolation{RGB{N0f8}, 2, Matrix{RGB{N0f8}}, BSpline{Linear{Throw{OnGrid}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, BSpline{Linear{Throw{OnGrid}}}, RGB{N0f8}}, AffineMap{SMatrix{2, 2, Float64, 4}, SVector{2, Float64}}, var"#α#10"{var"#λ#9"{Interpolations.FilledExtrapolation{RGB{N0f8}, 2, Interpolations.BSplineInterpolation{RGB{N0f8}, 2, Matrix{RGB{N0f8}}, BSpline{Linear{Throw{OnGrid}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, BSpline{Linear{Throw{OnGrid}}}, RGB{N0f8}}, AffineMap{SMatrix{2, 2, Float64, 4}, SVector{2, Float64}}}}, OffsetMatrix{RGB{N0f8}, Matrix{RGB{N0f8}}}})(Δ::FillArrays.Fill{NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}}, 2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}})
    @ Main ~/work/juliatests/test_warp.jl:63
 [23] ZBack
    @ ~/.julia/packages/Zygote/zowrf/src/compiler/chainrules.jl:77 [inlined]
 [24] Pullback
    @ ~/.julia/packages/ImageTransformations/xYRLH/src/warp.jl:88 [inlined]
 [25] (::typeof(∂(warp)))(Δ::FillArrays.Fill{NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}}, 2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}})
    @ Zygote ~/.julia/packages/Zygote/zowrf/src/compiler/interface2.jl:0
 [26] Pullback
    @ ~/.julia/packages/ImageTransformations/xYRLH/src/warp.jl:87 [inlined]
 [27] (::typeof(∂(warp)))(Δ::FillArrays.Fill{NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}}, 2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}})
    @ Zygote ~/.julia/packages/Zygote/zowrf/src/compiler/interface2.jl:0
 [28] Pullback
    @ ~/.julia/packages/ImageTransformations/xYRLH/src/warp.jl:106 [inlined]
 [29] (::typeof(∂(#warp#51)))(Δ::FillArrays.Fill{NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}}, 2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}})
    @ Zygote ~/.julia/packages/Zygote/zowrf/src/compiler/interface2.jl:0
 [30] #188
    @ ~/.julia/packages/Zygote/zowrf/src/lib/lib.jl:194 [inlined]
 [31] #1689#back
    @ ~/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:59 [inlined]
 [32] Pullback
    @ ~/.julia/packages/ImageTransformations/xYRLH/src/warp.jl:105 [inlined]
 [33] (::typeof(∂(warp)))(Δ::FillArrays.Fill{NamedTuple{(:r, :g, :b), Tuple{Float64, Float64, Float64}}, 2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}})
    @ Zygote ~/.julia/packages/Zygote/zowrf/src/compiler/interface2.jl:0
 [34] Pullback
    @ ./REPL[14]:1 [inlined]
 [35] (::typeof(∂(#18)))(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/zowrf/src/compiler/interface2.jl:0
 [36] (::Zygote.var"#41#42"{typeof(∂(#18))})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/zowrf/src/compiler/interface.jl:41
 [37] gradient(f::Function, args::AffineMap{RotMatrix2{Float64}, SVector{2, Float64}})
    @ Zygote ~/.julia/packages/Zygote/zowrf/src/compiler/interface.jl:59
 [38] top-level scope
    @ REPL[14]:1
 [39] top-level scope
    @ ~/.julia/packages/CUDA/3VnCC/src/initialization.jl:81
DhairyaLGandhi commented 3 years ago

Hey so I was playing around with a couple things here, and noticed -

julia> Interpolations.gradient(etp, 128,128)
2-element SVector{2, RGB{Float64}} with indices SOneTo(2):
 RGB{Float64}(0.1877378741327218,-0.2520153083508244,-0.14480481623229252)
 RGB{Float64}(0.06388650061740186,-0.16431235982818448,-0.24256451766987147)

julia> Interpolations.gradient(etp, 128,129)
ERROR: MethodError: no method matching getfirst(::RGB{N0f8})
Closest candidates are:
  getfirst(::Interpolations.Flag) at /home/dhairyalgandhi/.julia/packages/Interpolations/QLbbu/src/utils.jl:17
  getfirst(::Tuple) at /home/dhairyalgandhi/.julia/packages/Interpolations/QLbbu/src/utils.jl:18
Stacktrace:
 [1] _tcollect
   @ ~/.julia/packages/Interpolations/QLbbu/src/utils.jl:23 [inlined]
 [2] tcollect(f::typeof(Interpolations.etpflag), itp::Interpolations.FilledExtrapolation{RGB{Float64}, 2, Interpolations.BSplineInterpolation{RGB{Float64}, 2, Matrix{RGB{Float64}}, BSpline{Linear{Throw{OnGrid}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, BSpline{Linear{Throw{OnGrid}}}, RGB{N0f8}})
   @ Interpolations ~/.julia/packages/Interpolations/QLbbu/src/utils.jl:22
 [3] gradient(::Interpolations.FilledExtrapolation{RGB{Float64}, 2, Interpolations.BSplineInterpolation{RGB{Float64}, 2, Matrix{RGB{Float64}}, BSpline{Linear{Throw{OnGrid}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, BSpline{Linear{Throw{OnGrid}}}, RGB{N0f8}}, ::Int64, ::Int64)
   @ Interpolations ~/.julia/packages/Interpolations/QLbbu/src/extrapolation/extrapolation.jl:71
 [4] top-level scope
   @ REPL[43]:1
 [5] top-level scope

So if I try to get a gradient for an extrapolation that is outside the range of the size, we get this error. Is that expected?

SomTambe commented 3 years ago

I'm closing this issue since everything has been taken care of.