search

slimgroup / WISE.jl

WISE: full-Waveform variational Inference via Subsurface Extensions
https://doi.org/10.1190/geo2023-0744.1
MIT License
2 stars 0 forks source link

Getting an error when trying to use Summarized net #3

Closed advaitb closed 7 months ago

advaitb commented 7 months ago

@ziyiyin97 I could successfully replicate the environment on aws but I'm getting an error when trying to run summarized network at the G.forward step.

ERROR: LoadError: MethodError: no method matching backward(::CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, ::Nothing, ::FluxBlock)
Closest candidates are:
  backward(::AbstractArray{T, N}, ::AbstractArray{T, N}, ::FluxBlock; set_grad) where {T, N} at ~/.julia/packages/InvertibleNetworks/583RP/src/layers/layer_flux_block.jl:65
  backward(::AbstractArray{T, N}, ::AbstractArray{T, N}, ::AbstractArray{T, N}, ::AbstractArray{T, N}, ::ConditionalResidualBlock; set_grad) where {T, N} at ~/.julia/packages/InvertibleNetworks/583RP/src/conditional_layers/conditional_layer_residual_block.jl:113
  backward(::AbstractArray{T, N}, ::AbstractArray{T, N}, ::AbstractArray{T, N}, ::AbstractArray{T, N}, ::CouplingLayerBasic; set_grad) where {T, N} at ~/.julia/packages/InvertibleNetworks/583RP/src/layers/invertible_layer_basic.jl:122
  ...
Stacktrace:
  [1] _predefined_mode(::FluxBlock, ::Symbol, ::CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ InvertibleNetworks ~/.julia/packages/InvertibleNetworks/583RP/src/utils/neuralnet.jl:29
  [2] _predefined_mode(::FluxBlock, ::Symbol, ::CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, ::Vararg{Any})
    @ InvertibleNetworks ~/.julia/packages/InvertibleNetworks/583RP/src/utils/neuralnet.jl:27
  [3] (::InvertibleNetworks.var"#143#145"{InvertibleNetworks.var"#143#144#146"{FluxBlock}})(::CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ InvertibleNetworks ~/.julia/packages/InvertibleNetworks/583RP/src/utils/neuralnet.jl:40
  [4] (::InvertibleNetworks.var"#143#145"{InvertibleNetworks.var"#143#144#146"{FluxBlock}})(::CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, ::Vararg{Any})
    @ InvertibleNetworks ~/.julia/packages/InvertibleNetworks/583RP/src/utils/neuralnet.jl:40
  [5] backward(ΔX::CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, X::CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, Y::CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, S::SummarizedNet; Y_save::Nothing)
    @ InvertibleNetworks ~/.julia/packages/InvertibleNetworks/583RP/src/networks/summarized_net.jl:54
  [6] backward(ΔX::CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, X::CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, Y::CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, S::SummarizedNet)
    @ InvertibleNetworks ~/.julia/packages/InvertibleNetworks/583RP/src/networks/summarized_net.jl:52
  [7] _predefined_mode(::SummarizedNet, ::Symbol, ::CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, ::Vararg{CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ InvertibleNetworks ~/.julia/packages/InvertibleNetworks/583RP/src/utils/neuralnet.jl:29
  [8] _predefined_mode(::SummarizedNet, ::Symbol, ::CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, ::Vararg{CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}})
    @ InvertibleNetworks ~/.julia/packages/InvertibleNetworks/583RP/src/utils/neuralnet.jl:27
  [9] (::InvertibleNetworks.var"#143#145"{InvertibleNetworks.var"#143#144#146"{SummarizedNet}})(::CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, ::Vararg{CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ InvertibleNetworks ~/.julia/packages/InvertibleNetworks/583RP/src/utils/neuralnet.jl:40
 [10] (::InvertibleNetworks.var"#143#145"{InvertibleNetworks.var"#143#144#146"{SummarizedNet}})(::CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, ::Vararg{CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}})
    @ InvertibleNetworks ~/.julia/packages/InvertibleNetworks/583RP/src/utils/neuralnet.jl:40
 [11] top-level scope
    @ ~/Advait/wise_env/train_ofwi.jl:123
in expression starting at /home/ec2-user/Advait/wise_env/train_ofwi.jl:105

I can run the train script using just the CNF but the training is unstable and the loss doesn't decrease (infact it increases) after the first 50 iterations on the OpenFWI data. Here the script I'm currently using.

ziyiyin97 commented 7 months ago

I saw this issue was redirected to https://github.com/slimgroup/InvertibleNetworks.jl/issues/102 so I will close this one. But feel free to reopen.