search

slimgroup / JUDI.jl

Julia Devito inversion.
https://slimgroup.github.io/JUDI.jl
MIT License
98 stars 31 forks source link

Unable to run FWI with SPG and subsampling #218

Closed kerim371 closed 8 months ago

kerim371 commented 8 months ago

Hi,

I tried to run Viking Graben example with subsampling_factor=10 and got error (it works fine without subsampling though):

ERROR: LoadError: PyError ($(Expr(:escape, :(ccall(#= /home/kerim/Documents/Colada/r/julia-1.9/.julia/packages/PyCall/KLzIO/src/pyfncall.jl:43 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'ValueError'>
ValueError('Override `us_u(t_sub, x, y)` is incompatible with overrides `[damp(x, y), irho(x, y), m(x, y), rcvu(time, p_rcvu), srcu(time, p_srcu), u(t, x, y), us_u(t_sub, x, y)]`')
  File "/home/kerim/Documents/Colada/r/julia-1.9/.julia/packages/JUDI/JEsVr/src/pysource/interface.py", line 359, in J_adjoint
    return J_adjoint_standard(model, src_coords, wavelet, rec_coords, recin,
  File "/home/kerim/Documents/Colada/r/julia-1.9/.julia/packages/JUDI/JEsVr/src/pysource/interface.py", line 478, in J_adjoint_standard
    rec, u, Iu, _ = ffunc(model, src_coords, rec_coords, wavelet, save=True, nlind=nlind,
  File "/home/kerim/Documents/Colada/r/julia-1.9/.julia/packages/JUDI/JEsVr/src/pysource/propagators.py", line 77, in forward
    summary = op(**kw)
  File "/home/kerim/Documents/Colada/r/python-install/lib/python3.9/site-packages/devito/operator/operator.py", line 761, in __call__
    return self.apply(**kwargs)
  File "/home/kerim/Documents/Colada/r/python-install/lib/python3.9/site-packages/devito/operator/operator.py", line 827, in apply
    args = self.arguments(**kwargs)
  File "/home/kerim/Documents/Colada/r/python-install/lib/python3.9/site-packages/devito/operator/operator.py", line 668, in arguments
    args = self._prepare_arguments(**kwargs)
  File "/home/kerim/Documents/Colada/r/python-install/lib/python3.9/site-packages/devito/operator/operator.py", line 544, in _prepare_arguments
    raise ValueError("Override `%s` is incompatible with overrides `%s`" %

Stacktrace:
  [1] pyerr_check
    @ ~/Documents/Colada/r/julia-1.9/.julia/packages/PyCall/KLzIO/src/exception.jl:75 [inlined]
  [2] pyerr_check
    @ ~/Documents/Colada/r/julia-1.9/.julia/packages/PyCall/KLzIO/src/exception.jl:79 [inlined]
  [3] _handle_error(msg::String)
    @ PyCall ~/Documents/Colada/r/julia-1.9/.julia/packages/PyCall/KLzIO/src/exception.jl:96
  [4] macro expansion
    @ ~/Documents/Colada/r/julia-1.9/.julia/packages/PyCall/KLzIO/src/exception.jl:110 [inlined]
  [5] #107
    @ ~/Documents/Colada/r/julia-1.9/.julia/packages/PyCall/KLzIO/src/pyfncall.jl:43 [inlined]
  [6] disable_sigint
    @ ./c.jl:473 [inlined]
  [7] __pycall!
    @ ~/Documents/Colada/r/julia-1.9/.julia/packages/PyCall/KLzIO/src/pyfncall.jl:42 [inlined]
  [8] _pycall!(ret::PyCall.PyObject, o::PyCall.PyObject, args::Tuple{PyCall.PyObject, Vararg{Matrix{Float32}, 4}}, nargs::Int64, kw::PyCall.PyObject)
    @ PyCall ~/Documents/Colada/r/julia-1.9/.julia/packages/PyCall/KLzIO/src/pyfncall.jl:29
  [9] _pycall!(ret::PyCall.PyObject, o::PyCall.PyObject, args::Tuple{PyCall.PyObject, Vararg{Matrix{Float32}, 4}}, kwargs::Base.Pairs{Symbol, Any, NTuple{13, Symbol}, NamedTuple{(:t_sub, :space_order, :checkpointing, :freq_list, :ic, :is_residual, :born_fwd, :nlind, :dft_sub, :f0, :return_obj, :misfit, :illum), Tuple{Int64, Int64, Bool, Nothing, String, Bool, Bool, Bool, Int64, Float32, Bool, PyCall.PyObject, Bool}}})
    @ PyCall ~/Documents/Colada/r/julia-1.9/.julia/packages/PyCall/KLzIO/src/pyfncall.jl:11
 [10] #pycall#112
    @ ~/Documents/Colada/r/julia-1.9/.julia/packages/PyCall/KLzIO/src/pyfncall.jl:80 [inlined]
 [11] pycall
    @ ~/Documents/Colada/r/julia-1.9/.julia/packages/PyCall/KLzIO/src/pyfncall.jl:80 [inlined]
 [12] #4
    @ ~/Documents/Colada/r/julia-1.9/.julia/packages/JUDI/JEsVr/src/JUDI.jl:82 [inlined]
 [13] (::JUDI.var"#1#2"{JUDI.var"#4#5"{Tuple{Float32, PyCall.PyArray, PyCall.PyObject, PyCall.PyObject}, Base.Pairs{Symbol, Any, NTuple{13, Symbol}, NamedTuple{(:t_sub, :space_order, :checkpointing, :freq_list, :ic, :is_residual, :born_fwd, :nlind, :dft_sub, :f0, :return_obj, :misfit, :illum), Tuple{Int64, Int64, Bool, Nothing, String, Bool, Bool, Bool, Int64, Float32, Bool, PyCall.PyObject, Bool}}}, PyCall.PyObject, Tuple{PyCall.PyObject, Vararg{Matrix{Float32}, 4}}}})()
    @ JUDI ~/Documents/Colada/r/julia-1.9/.julia/packages/JUDI/JEsVr/src/JUDI.jl:74
 [14] lock(f::JUDI.var"#1#2"{JUDI.var"#4#5"{Tuple{Float32, PyCall.PyArray, PyCall.PyObject, PyCall.PyObject}, Base.Pairs{Symbol, Any, NTuple{13, Symbol}, NamedTuple{(:t_sub, :space_order, :checkpointing, :freq_list, :ic, :is_residual, :born_fwd, :nlind, :dft_sub, :f0, :return_obj, :misfit, :illum), Tuple{Int64, Int64, Bool, Nothing, String, Bool, Bool, Bool, Int64, Float32, Bool, PyCall.PyObject, Bool}}}, PyCall.PyObject, Tuple{PyCall.PyObject, Vararg{Matrix{Float32}, 4}}}}, l::ReentrantLock)
    @ Base ./lock.jl:229
 [15] pylock(f::JUDI.var"#4#5"{Tuple{Float32, PyCall.PyArray, PyCall.PyObject, PyCall.PyObject}, Base.Pairs{Symbol, Any, NTuple{13, Symbol}, NamedTuple{(:t_sub, :space_order, :checkpointing, :freq_list, :ic, :is_residual, :born_fwd, :nlind, :dft_sub, :f0, :return_obj, :misfit, :illum), Tuple{Int64, Int64, Bool, Nothing, String, Bool, Bool, Bool, Int64, Float32, Bool, PyCall.PyObject, Bool}}}, PyCall.PyObject, Tuple{PyCall.PyObject, Vararg{Matrix{Float32}, 4}}})
    @ JUDI ~/Documents/Colada/r/julia-1.9/.julia/packages/JUDI/JEsVr/src/JUDI.jl:71
 [16] rlock_pycall(::PyCall.PyObject, ::Type{Tuple{Float32, PyCall.PyArray, PyCall.PyObject, PyCall.PyObject}}, ::PyCall.PyObject, ::Vararg{Any}; kw::Base.Pairs{Symbol, Any, NTuple{13, Symbol}, NamedTuple{(:t_sub, :space_order, :checkpointing, :freq_list, :ic, :is_residual, :born_fwd, :nlind, :dft_sub, :f0, :return_obj, :misfit, :illum), Tuple{Int64, Int64, Bool, Nothing, String, Bool, Bool, Bool, Int64, Float32, Bool, PyCall.PyObject, Bool}}})
    @ JUDI ~/Documents/Colada/r/julia-1.9/.julia/packages/JUDI/JEsVr/src/JUDI.jl:81
 [17] macro expansion
    @ ~/Documents/Colada/r/julia-1.9/.julia/packages/JUDI/JEsVr/src/TimeModeling/Modeling/misfit_fg.jl:50 [inlined]
 [18] macro expansion
    @ ./timing.jl:393 [inlined]
 [19] macro expansion
    @ ~/Documents/Colada/r/julia-1.9/.julia/packages/JUDI/JEsVr/src/JUDI.jl:141 [inlined]
 [20] multi_src_fg(model_full::JUDI.IsoModel{Float32, 2}, source::judiVector{Float32, Matrix{Float32}}, dObs::judiVector{Float32, Matrix{Float32}}, dm::Nothing, options::JUDIOptions, nlind::Bool, lin::Bool, misfit::Function)
    @ JUDI ~/Documents/Colada/r/julia-1.9/.julia/packages/JUDI/JEsVr/src/TimeModeling/Modeling/misfit_fg.jl:49
 [21] macro expansion
    @ ~/Documents/Colada/r/julia-1.9/.julia/packages/JUDI/JEsVr/src/TimeModeling/Modeling/propagation.jl:40 [inlined]
 [22] macro expansion
    @ ./timing.jl:393 [inlined]
 [23] macro expansion
    @ ~/Documents/Colada/r/julia-1.9/.julia/packages/JUDI/JEsVr/src/JUDI.jl:141 [inlined]
 [24] run_and_reduce(func::Function, #unused#::Nothing, nsrc::Int64, arg_func::JUDI.var"#276#277"{JUDIOptions, Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:nlind, :lin, :misfit), Tuple{Bool, Bool, typeof(myloss)}}}, JUDI.IsoModel{Float32, 2}, judiVector{Float32, Matrix{Float32}}, judiVector{Float32, Matrix{Float32}}, Nothing})
    @ JUDI ~/Documents/Colada/r/julia-1.9/.julia/packages/JUDI/JEsVr/src/TimeModeling/Modeling/propagation.jl:39
 [25] multi_src_fg!(G::PhysicalParameter{Float32, 2}, model::JUDI.IsoModel{Float32, 2}, q::judiVector{Float32, Matrix{Float32}}, dobs::judiVector{Float32, Matrix{Float32}}, dm::Nothing; options::JUDIOptions, kw::Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:nlind, :lin, :misfit), Tuple{Bool, Bool, typeof(myloss)}}})
    @ JUDI ~/Documents/Colada/r/julia-1.9/.julia/packages/JUDI/JEsVr/src/TimeModeling/Modeling/propagation.jl:108
 [26] multi_src_fg!
    @ ~/Documents/Colada/r/julia-1.9/.julia/packages/JUDI/JEsVr/src/TimeModeling/Modeling/propagation.jl:100 [inlined]
 [27] #multi_exp_fg!#255
    @ ~/Documents/Colada/r/julia-1.9/.julia/packages/JUDI/JEsVr/src/TimeModeling/Modeling/misfit_fg.jl:191 [inlined]
 [28] multi_exp_fg!
    @ ~/Documents/Colada/r/julia-1.9/.julia/packages/JUDI/JEsVr/src/TimeModeling/Modeling/misfit_fg.jl:191 [inlined]
 [29] fwi_objective!(G::PhysicalParameter{Float32, 2}, model::JUDI.IsoModel{Float32, 2}, q::judiVector{Float32, Matrix{Float32}}, dobs::judiVector{Float32, Matrix{Float32}}; options::JUDIOptions, kw::Base.Pairs{Symbol, typeof(myloss), Tuple{Symbol}, NamedTuple{(:misfit,), Tuple{typeof(myloss)}}})
    @ JUDI ~/Documents/Colada/r/julia-1.9/.julia/packages/JUDI/JEsVr/src/TimeModeling/Modeling/misfit_fg.jl:172
 [30] fwi_objective(model::JUDI.IsoModel{Float32, 2}, q::judiVector{Float32, Matrix{Float32}}, dobs::judiVector{Float32, Matrix{Float32}}; options::JUDIOptions, kw::Base.Pairs{Symbol, typeof(myloss), Tuple{Symbol}, NamedTuple{(:misfit,), Tuple{typeof(myloss)}}})
    @ JUDI ~/Documents/Colada/r/julia-1.9/.julia/packages/JUDI/JEsVr/src/TimeModeling/Modeling/misfit_fg.jl:137
 [31] objective_function(m_update::Matrix{Float64})
    @ Main ~/Documents/JUDI.jl/examples/field_examples/viking_graben_line12/fwi/fwi_spg.jl:196
 [32] (::SlimOptim.var"#objgrad!#11"{typeof(objective_function), result{Float32}})(g::Matrix{Float32}, x::Matrix{Float64})
    @ SlimOptim ~/Documents/Colada/r/julia-1.9/.julia/packages/SlimOptim/33Fm9/src/SPGSlim.jl:99
 [33] _spg(obj::Function, grad!::SlimOptim.var"#grad!#10"{typeof(objective_function), result{Float32}}, objgrad!::SlimOptim.var"#objgrad!#11"{typeof(objective_function), result{Float32}}, projection::SlimOptim.var"#projection#9"{typeof(proj), result{Float32}}, x::Matrix{Float32}, g::Matrix{Float32}, sol::result{Float32}, ls::Nothing, options::SlimOptim.SPG_params; callback::typeof(SlimOptim.noop_callback))
    @ SlimOptim ~/Documents/Colada/r/julia-1.9/.julia/packages/SlimOptim/33Fm9/src/SPGSlim.jl:172
 [34] spg(funObj::typeof(objective_function), x::Matrix{Float32}, funProj::typeof(proj), options::SlimOptim.SPG_params; ls::Nothing, callback::Function)
    @ SlimOptim ~/Documents/Colada/r/julia-1.9/.julia/packages/SlimOptim/33Fm9/src/SPGSlim.jl:103
 [35] spg(funObj::Function, x::Matrix{Float32}, funProj::Function, options::SlimOptim.SPG_params)
    @ SlimOptim ~/Documents/Colada/r/julia-1.9/.julia/packages/SlimOptim/33Fm9/src/SPGSlim.jl:89
 [36] top-level scope
    @ ~/Documents/JUDI.jl/examples/field_examples/viking_graben_line12/fwi/fwi_spg.jl:247
in expression starting at /home/kerim/Documents/JUDI.jl/examples/field_examples/viking_graben_line12/fwi/fwi_spg.jl:247

JUDI: master (v3.3.9) Devito: v4.8.3

kerim371 commented 8 months ago

And I just tried with Devito v4.8.2 and it works. So the problem is with newer Devito v4.8.3

mloubout commented 8 months ago

That's quite surprising, the tests (i.e test_all_options.jl) works fine with subsampling including when changing it to a factor of 10 not sure what the issue is, could you try. the master branch of devito?

kerim371 commented 8 months ago

@mloubout Devito master branch doesn't work either. Also I tried Viking Graben lbfgs example and got same error.

At the same time I tried modeling_basic_2D.jl example wich worked fine with f, g = fwi_objective(model0, q, dobs; options=opt) and subsampling_factor=10.

But with Devito v4.8.2 everything works.

mloubout commented 8 months ago

Can you try replacing

https://github.com/slimgroup/JUDI.jl/blob/eaec4307ae85822042d5cd4c30d299376b024ccc/src/pysource/fields.py#L132

by

int(np.ceil((nt + t_sub)/t_sub))

kerim371 commented 8 months ago

@mloubout hi,

I remember I've written the comment here after I've tried the replacement you proposed int(np.ceil((nt + t_sub)/t_sub)) but I don't see it here. Have you see it?

The point was that I could not reproduce the error anymore neither with nsave = nt // t_sub + 1 or int(np.ceil((nt + t_sub)/t_sub))

So that was very surprising for me I remember but I ve'tried many things but could not reproduce the error.

If any testing is needed I can try.

mloubout commented 8 months ago

I changed it as it seems a bit safer for some corner cases, lemme know if it pops up again