Issues with running complete code

Hi, I am still running into the error, even after fixing the Windows issue. Could it be that there is an error with accessing the folder before it has even been created? This error only occurs for this particular folder.

Visualising realisations from the GaussianProcess/nuFixed model...
ERROR: LoadError: ArgumentError: No file exists at given path: C:\Users\poorv\Downloads\Y4S1\FYP\NeuralBayesEstimators\intermediates\GaussianProcess\nuFixed\parameter_configurations\test_xi.rda
Stacktrace:
 [1] checkpath_load
   @ C:\Users\poorv\.julia\packages\FileIO\BE7iZ\src\loadsave.jl:167 [inlined]
 [2] load(::String; options::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:convert,), Tuple{Bool}}})
   @ FileIO C:\Users\poorv\.julia\packages\FileIO\BE7iZ\src\loadsave.jl:110
 [3] loadwithoutdict(path::String, object_name::String)
   @ SpatialDeepSets C:\Users\poorv\Downloads\Y4S1\FYP\NeuralBayesEstimators\src\SpatialDeepSets.jl:157
 [4] Parameters(path::String)
   @ Main C:\Users\poorv\Downloads\Y4S1\FYP\NeuralBayesEstimators\src\GaussianProcess\nuFixed\Parameters.jl:29
 [5] top-level scope
   @ C:\Users\poorv\Downloads\Y4S1\FYP\NeuralBayesEstimators\src\VisualiseRealisations.jl:26

I would also like to touch on another issue: before I run your scripts, I have to manually change the line encoding of all the files under '/src' folder from CRLF to LF , else it gives this error:

C:\Users\poorv\Downloads\Y4S1\FYP\NeuralBayesEstimators>bash sh/all.sh
': not a valid identifier `R_HOME
sh/all.sh: line 3: $'\r': command not found
: invalid option4: set: -
set: usage: set [-abefhkmnptuvxBCHP] [-o option-name] [--] [arg ...]
sh/all.sh: line 5: $'\r': command not found

######## Setting up ############

sh/all.sh: line 9: $'\r': command not found
sh/all.sh: line 11: syntax error in conditional expression
'h/all.sh: line 11: syntax error near `]]
'h/all.sh: line 11: `if [[ ! -f data/RedSea/redseatemperature.rdata ]]

I am not sure if this has any implication, would it be possible to fix this too so that I do not have to change it after each round of pull?

I am a little short on time for my project, so I am hoping this problem could be solved soon. Do let me know if you will need more time. So sorry for the trouble

Best regards, Poorva

Hi,

I think the main issue is that you are running shell scripts (.sh files) using Windows. I have never used Windows, but my colleague says that this doesn't work well. Specifically for your first error, if you look at the shell script for the simulation studies (see here), in the case of the Gaussian process with fixed smoothness, the training parameters are constructed inside an if-statement. My colleague says this if-statement is probably not being treated properly by Windows, and hence the training parameters are not being constructed (which is why the file test_xi.rda does not exist at the given path).

In regards to your second issue, I think it also relates to the fact that the shell scripts are not designed for Windows. You'll see that we now have a folder bat that can be used by Windows users (see here for a description). Have you tried these .bat scripts?

You mention that you have a short turnaround for your project. We've made a package, NeuralEstimators, that implements the methodology described in this paper, and probably it would be faster for you to start from the package rather than this source code. I'm also happy to meet with you over Zoom if you'd like to discuss anything.

Best, Matt

Hi, It would be great if we could arrange a zoom meeting perhaps in the next few days, as I would like to clarify some things regarding the package use. What would your availability be in the coming week? Also would like to check, what could be the reason I am having this issue? Could it be an issue with Windows as well?

Precompiling project...
  ✗ Zygote
  ✗ Flux
  ✗ GraphNeuralNetworks
  ✗ NeuralEstimators
  86 dependencies successfully precompiled in 417 seconds (66 already precompiled)
  4 dependencies errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the packages

julia> import Pkg; Pkg.precompile()
Precompiling project...
  Progress [====================>                    ]  2/4
  Progress [====================>                    ]  2/4
  Progress [==============================>          ]  3/4
  ✗ Zygote
  ✗ Flux
  ✗ GraphNeuralNetworks
  ✗ NeuralEstimators
  0 dependencies successfully precompiled in 305 seconds (152 already precompiled)

ERROR: The following 1 direct dependency failed to precompile:

NeuralEstimators [38f6df31-6b4a-4144-b2af-7ace2da57606]

Failed to precompile NeuralEstimators [38f6df31-6b4a-4144-b2af-7ace2da57606] to C:\Users\poorv\.julia\compiled\v1.7\NeuralEstimators\jl_18C9.tmp.
ERROR: LoadError: MethodError: no method matching iterate(::Core.CodeInfo)
Closest candidates are:
  iterate(::Union{LinRange, StepRangeLen}) at C:\Users\poorv\AppData\Local\Programs\Julia-1.7.1\share\julia\base\range.jl:826
  iterate(::Union{LinRange, StepRangeLen}, ::Integer) at C:\Users\poorv\AppData\Local\Programs\Julia-1.7.1\share\julia\base\range.jl:826
  iterate(::T) where T<:Union{Base.KeySet{<:Any, <:Dict}, Base.ValueIterator{<:Dict}} at C:\Users\poorv\AppData\Local\Programs\Julia-1.7.1\share\julia\base\dict.jl:695
  ...
Stacktrace:
  [1] indexed_iterate(I::Core.CodeInfo, i::Int64)
    @ Base .\tuple.jl:92
  [2] pullback(::Function, ::Zygote.Context{false}, ::Int64, ::Vararg{Int64})
    @ Zygote C:\Users\poorv\.julia\packages\Zygote\WOy6z\src\compiler\interface.jl:44
  [3] pullback(::Function, ::Int64, ::Int64)
    @ Zygote C:\Users\poorv\.julia\packages\Zygote\WOy6z\src\compiler\interface.jl:42
  [4] gradient(::Function, ::Int64, ::Vararg{Int64})
    @ Zygote C:\Users\poorv\.julia\packages\Zygote\WOy6z\src\compiler\interface.jl:96
  [5] top-level scope
    @ C:\Users\poorv\.julia\packages\Zygote\WOy6z\src\precompile.jl:17
  [6] include(mod::Module, _path::String)
    @ Base .\Base.jl:418
  [7] include
    @ C:\Users\poorv\.julia\packages\Zygote\WOy6z\src\Zygote.jl:1 [inlined]
  [8] macro expansion
    @ C:\Users\poorv\.julia\packages\Requires\Z8rfN\src\Requires.jl:40 [inlined]
  [9] precompile()
    @ Zygote C:\Users\poorv\.julia\packages\Zygote\WOy6z\src\Zygote.jl:63
 [10] macro expansion
    @ C:\Users\poorv\.julia\packages\PrecompileTools\kmH5L\src\workloads.jl:85 [inlined]
 [11] top-level scope
    @ C:\Users\poorv\.julia\packages\Zygote\WOy6z\src\Zygote.jl:84
 [12] include
    @ .\Base.jl:418 [inlined]
 [13] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
    @ Base .\loading.jl:1318
 [14] top-level scope
    @ none:1
 [15] eval
    @ .\boot.jl:373 [inlined]
 [16] eval(x::Expr)
    @ Base.MainInclude .\client.jl:453
 [17] top-level scope
    @ none:1
in expression starting at C:\Users\poorv\.julia\packages\Zygote\WOy6z\src\precompile.jl:17
in expression starting at C:\Users\poorv\.julia\packages\Zygote\WOy6z\src\Zygote.jl:1
ERROR: LoadError: Failed to precompile Zygote [e88e6eb3-aa80-5325-afca-941959d7151f] to C:\Users\poorv\.julia\compiled\v1.7\Zygote\jl_7E1A.tmp.
Stacktrace:
  [1] error(s::String)
    @ Base .\error.jl:33
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, ignore_loaded_modules::Bool)
    @ Base .\loading.jl:1466
  [3] compilecache(pkg::Base.PkgId, path::String)
    @ Base .\loading.jl:1410
  [4] _require(pkg::Base.PkgId)
    @ Base .\loading.jl:1120
  [5] require(uuidkey::Base.PkgId)
    @ Base .\loading.jl:1013
  [6] require(into::Module, mod::Symbol)
    @ Base .\loading.jl:997
  [7] include
    @ .\Base.jl:418 [inlined]
  [8] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
    @ Base .\loading.jl:1318
  [9] top-level scope
    @ none:1
 [10] eval
    @ .\boot.jl:373 [inlined]
 [11] eval(x::Expr)
    @ Base.MainInclude .\client.jl:453
 [12] top-level scope
    @ none:1
in expression starting at C:\Users\poorv\.julia\packages\Flux\n3cOc\src\Flux.jl:1
ERROR: LoadError: Failed to precompile Flux [587475ba-b771-5e3f-ad9e-33799f191a9c] to C:\Users\poorv\.julia\compiled\v1.7\Flux\jl_70AD.tmp.
Stacktrace:
  [1] error(s::String)
    @ Base .\error.jl:33
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, ignore_loaded_modules::Bool)
    @ Base .\loading.jl:1466
  [3] compilecache(pkg::Base.PkgId, path::String)
    @ Base .\loading.jl:1410
  [4] _require(pkg::Base.PkgId)
    @ Base .\loading.jl:1120
  [5] require(uuidkey::Base.PkgId)
    @ Base .\loading.jl:1013
  [6] require(into::Module, mod::Symbol)
    @ Base .\loading.jl:997
  [7] include
    @ .\Base.jl:418 [inlined]
  [8] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base .\loading.jl:1318
  [9] top-level scope
    @ none:1
 [10] eval
    @ .\boot.jl:373 [inlined]
 [11] eval(x::Expr)
    @ Base.MainInclude .\client.jl:453
 [12] top-level scope
    @ none:1
in expression starting at C:\Users\poorv\.julia\packages\NeuralEstimators\2XgrW\src\NeuralEstimators.jl:1
Stacktrace:
 [1] pkgerror(msg::String)
   @ Pkg.Types C:\Users\poorv\AppData\Local\Programs\Julia-1.7.1\share\julia\stdlib\v1.7\Pkg\src\Types.jl:68
 [2] precompile(ctx::Pkg.Types.Context; internal_call::Bool, strict::Bool, warn_loaded::Bool, already_instantiated::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Pkg.API C:\Users\poorv\AppData\Local\Programs\Julia-1.7.1\share\julia\stdlib\v1.7\Pkg\src\API.jl:1362
 [3] precompile
   @ C:\Users\poorv\AppData\Local\Programs\Julia-1.7.1\share\julia\stdlib\v1.7\Pkg\src\API.jl:1013 [inlined]
 [4] #precompile#220
   @ C:\Users\poorv\AppData\Local\Programs\Julia-1.7.1\share\julia\stdlib\v1.7\Pkg\src\API.jl:1011 [inlined]
 [5] precompile()
   @ Pkg.API C:\Users\poorv\AppData\Local\Programs\Julia-1.7.1\share\julia\stdlib\v1.7\Pkg\src\API.jl:1011
 [6] top-level scope
   @ REPL[7]:1

Thank you for your help! Best Regards, Poorva Pandya

Hi Poorva, I've sent you a message on LinkedIn (I think it is you!). Otherwise could you please send me an email (msdale@uow.edu.au) and we can set up a meeting time.

Regarding the error; are you trying to install the current version of NeuralEstimators, or the version that was used for the reproducible code in this repository? If it's the former, you may want to try using a later version of Julia (e.g., Julia 1.9).

Best, Matt

Hi, after running this particular code chunk in Julia

J = 100
θ = sample(1)
Z = [simulate(θ, m, J)]
assessment = assess([θ̂], θ, Z)

I encountered this issue:

julia> assessment = assess([θ̂], θ, Z)
ERROR: AssertionError: The number of data sets in Z must be a multiple of the number of parameter vectors in θ
Stacktrace:
 [1] assess(estimators::Vector{…}, θ::LinearAlgebra.Adjoint{…}, Z::Vector{…}; estimator_names::Vector{…}, parameter_names::Vector{…}, ξ::Nothing, use_ξ::Bool, xi::Nothing, use_xi::Bool, use_gpu::Bool, verbose::Bool)
   @ NeuralEstimators C:\Users\poorv\.julia\packages\NeuralEstimators\2XgrW\src\assess.jl:206
 [2] assess(estimators::Vector{PointEstimator{…}}, θ::LinearAlgebra.Adjoint{Float64, Matrix{…}}, Z::Vector{Vector{…}})
   @ NeuralEstimators C:\Users\poorv\.julia\packages\NeuralEstimators\2XgrW\src\assess.jl:186
 [3] top-level scope
   @ REPL[29]:1
Some type information was truncated. Use show(err) to see complete types.

julia> show(err)
1-element ExceptionStack:
AssertionError: The number of data sets in Z must be a multiple of the number of parameter vectors in θ
Stacktrace:
 [1] assess(estimators::Vector{PointEstimator{DeepSet{Chain{Tuple{Dense{typeof(relu), Matrix{Float32}, Vector{Float32}}, Dense{typeof(relu), Matrix{Float32}, Vector{Float32}}}}, typeof(NeuralEstimators.elementwise_mean), Chain{Tuple{Dense{typeof(relu), Matrix{Float32}, Vector{Float32}}, Dense{typeof(identity), Matrix{Float32}, Vector{Float32}}}}}}}, θ::LinearAlgebra.Adjoint{Float64, Matrix{Float64}}, Z::Vector{Vector{Matrix{Float64}}}; estimator_names::Vector{String}, parameter_names::Vector{String}, ξ::Nothing, use_ξ::Bool, xi::Nothing, use_xi::Bool, use_gpu::Bool, verbose::Bool)
   @ NeuralEstimators C:\Users\poorv\.julia\packages\NeuralEstimators\2XgrW\src\assess.jl:206
 [2] assess(estimators::Vector{PointEstimator{DeepSet{Chain{Tuple{Dense{typeof(relu), Matrix{Float32}, Vector{Float32}}, Dense{typeof(relu), Matrix{Float32}, Vector{Float32}}}}, typeof(NeuralEstimators.elementwise_mean), Chain{Tuple{Dense{typeof(relu), Matrix{Float32}, Vector{Float32}}, Dense{typeof(identity), Matrix{Float32}, Vector{Float32}}}}}}}, θ::LinearAlgebra.Adjoint{Float64, Matrix{Float64}}, Z::Vector{Vector{Matrix{Float64}}})
   @ NeuralEstimators C:\Users\poorv\.julia\packages\NeuralEstimators\2XgrW\src\assess.jl:186
 [3] top-level scope
   @ REPL[29]:1

The previous lines were exactly the same as the example you have provided on your package documentation. I am not able to figure out what is causing this issue. Hoping to get a better understanding

A second issue I faced was:

julia> Z = [simulate(θ, m, J)]
ERROR: MethodError: no method matching simulate(::LinearAlgebra.Adjoint{Float64, Matrix{Float64}}, ::Int64, ::Int64)

Closest candidates are:
  simulate(::Any, ::Any)
   @ Main REPL[6]:1
  simulate(::P, ::Any, ::Integer; args...) where P<:Union{ParameterConfigurations, Matrix}
   @ NeuralEstimators C:\Users\poorv\.julia\packages\NeuralEstimators\2XgrW\src\simulate.jl:32

Stacktrace:
 [1] top-level scope
   @ REPL[18]:1

when i tried to run

J = 100
θ = sample(1)
Z = [simulate(θ, m, J)]
assessment = assess([θ̂], θ, Z)

I am not sure if I was required to run something else prior to this. Could you please explain to me what is J supposed to signify and how it is different from the previous function? I was facing difficulty in understand this bit in the documentation.

Thank you Best Regards, Poorva

Hi,

I realised that you were running an example given in the documentation, so I was able to run the code myself and see what the problem was. Sorry, there was a bug in the code that meant that example wasn't running properly. I've just updated the package; could you please pull it from GitHub and try again? Thanks.

For your second error, this was also a problem with the package (fixed now; sorry, it's been a while since I ran this example and it seems I broke backwards compatibility at some point). The error code tells you what went wrong: there was no method of simulate for objects of type LinearAlgebra.Adjoint{Float64, Matrix{Float64}}. This is because I had defined the method to dispatch on objects of type P<:Union{ParameterConfigurations, Matrix}, which does not include objects of type Adjoint (which are an AbstractMatrix but not a Matrix). This was easy to fix by changing the method definition for simulate to dispatch on objects of type P<:Union{ParameterConfigurations, AbstractMatrix}.

J represents the number of data sets simulated for each parameter vector. So, if you're using assess for looking at the bootstrap distribution of the estimator, J is the number of bootstrap replicates.

Cheers, Matt

Hi Matthew, Thank you for your help. The code you had given is now able to work just fine. I just wished to clarify a few more things. For my project, I wished to examine how sensitive the results are for different hyperparameters. Some of the suggestions given by my prof included number of prior samples drawn, using novel datasets, and optimising the neural network(ie. adding more layers/activation functions, etc)

I tried coming up with this to test on variable hidden layer counts.

using NeuralEstimators
using Flux
using Distributions
import NeuralEstimators: simulate
function sample(K)
    μ = rand(Normal(0, 1), K)
    σ = rand(Uniform(0.1, 1), K)
    θ = hcat(μ, σ)'
    θ = Float32.(θ)
    return θ
end
simulate(parameters, m) = [θ[1] .+ θ[2] .* randn(Float32, 1, m) for θ ∈ eachcol(parameters)]

function build_neural_network(input_dim, output_dim, n, activation, is_phi)

    layers = [Dense(input_dim, layer_width, activation)]

    for i in 1:n
        if is_phi
            push!(layers, Dense(layer_width, layer_width))
        else
            push!(layers, Dense(layer_width, layer_width, activation))
        end
    end

    if is_phi
        push!(layers, Dense(layer_width, output_dim))
    else
        push!(layers, Dense(layer_width, output_dim, activation))
    end

    return Chain(layers...)
end

# Example usage with n hidden layers
input_dim = 1
layer_width = 32
output_dim = 2
n = 3 
activation = relu  

ψ = build_neural_network(input_dim, layer_width, n, activation, false)
ϕ = build_neural_network(layer_width, output_dim,n,activation, true)

architecture = DeepSet(ψ, ϕ)

The NN for ψ is able to build successfully. However, for ϕ it returns this:

julia> ϕ = build_neural_network(layer_width, output_dim,n,activation, true)
ERROR: MethodError: Cannot `convert` an object of type
  Dense{typeof(identity),Array{Float32,2},Array{Float32,1}} to an object of type
  Dense{DataType,Array{Float32,2},Array{Float32,1}}

Closest candidates are:
  convert(::Type{T}, ::T) where T
   @ Base Base.jl:84

Stacktrace:
 [1] push!(a::Vector{Dense{DataType, Matrix{…}, Vector{…}}}, item::Dense{typeof(identity), Matrix{Float32}, Vector{Float32}})
   @ Base .\array.jl:1118
 [2] build_neural_network(input_dim::Int64, output_dim::Int64, n::Int64, activation::Function, is_phi::Bool)
   @ Main .\REPL[101]:7
 [3] top-level scope
   @ REPL[109]:1
Some type information was truncated. Use `show(err)` to see complete types.

After some searching, it seems like might be a compatibility issue with NeuralEstimators package. Is there any way to mitigate this? Else, could I perhaps get an alternative idea on how I could go about testing on different hyperparameters

So sorry for the trouble. I could also arrange a meeting on Monday if it is easier to discuss that way Thanks for your help! Poorva

Hi Poorva,

Glad to hear that the code is now working. And it sounds like an interesting project, there are many hyperparameters involved with these neural likelihood-free methods. I'll be interested to hear about your findings.

The error arises because the object Dense(layer_width, layer_width) is of a different type to the object Dense(input_dim, layer_width, activation). In particular, when you write:

layers = [Dense(input_dim, layer_width, activation)]

the object layers is initialised with type Vector{T}, where T is {DataType,Array{Float32,2},Array{Float32,1}}. Any object added to layers with push!() must be of the type T, and this is where the code breaks, because Dense(layer_width, layer_width) has a different type.

To understand this better, consider the following toy example:

x = [2.0]            # initialise as a Vector{Float64}
push!(x, 3.0)        # works, because 3.0 is of type Float64
push!(x, 3.0f0)      # works, because 3.0f0, which is of type Float32, can be converted to Float64
push!(x, [3.0, 3.0]) # error, because we can't convert a vector of numbers to Float64

x = []               # initialise as a Vector{Any}, that is, a vector that can store objects of any type
push!(x, 3.0)        # works
push!(x, 3.0f0)      # works
push!(x, [3.0, 3.0]) # works

The easiest solution is to modify your function as:

function build_neural_network(input_dim, output_dim, n, activation, is_phi)

    layers = []

    for i in 1:n
        push!(layers, Dense(layer_width, layer_width, activation))
    end

    if is_phi
        push!(layers, Dense(layer_width, output_dim))
    else
        push!(layers, Dense(layer_width, output_dim, activation))
    end

    return Chain(layers...)
end

Note that I've changed the code slightly: the hidden layers of ϕ should contain non-linear activation functions as well.

Best, Matt

Hi Matt, Apologies for the oversight. The code seems to work fine after the changes, thanks for your help! I currently have tried experiment with different sample sizes(m) and different layer counts for ψ and ϕ. You mentioned that there are many hyperparameters involved in these methods. Could you perhaps give me some examples of hyperparameters I could look into, If possible some that I could easily experiment with using the functions of the package themselves? Thanks again! Best regards, Poorva

Hi Poorva, No worries, error messages from Julia can be a bit cryptic at first.

I'm not sure that I would classify the sample size as part of the hyperparameters: the sample size is dictated by the application. I suppose that you might be interested in seeing how well the estimator can extrapolate to sample sizes that differ to those used during training, like we did in our paper. But it might be simpler to keep the sample size fixed in your experiments.

I would say that the main hyperparameters, in order of importance, are:

the number of parameter vectors used in the training set (K),
neural-network architecture hyperparameters (width and depth of the networks, activation functions, etc.),
the learning rate used during training.

You can modify all of these using the package (you might also be interested in the helper function initialise_estimator).

Best, Matt

Hi Matt, Sure, I will look into these parameters. Thank you so much for your help! Really appreciate it :) Best Regards, Poorva Pandya

msainsburydale / NeuralBayesEstimators

Issues with running complete code #3