Feedback

[rafaqz]

@briochemc @ConnectedSystems ModelParameters.jl and InteractModels.jl are pretty much ready to use now.

I have built them into GrowthMaps.jl and DynamicGrids.jl/DynamicGridsInteract.jl for display of params in the REPL, parameter flattening and for auto generating control interfaces. So it's in a working state. But it would be great to get some early feedback from you both, as you indicated wanting to use this once it's ready.

It has some behaviours I have discussed previously:

• @briochemc units are handled and merged with values as for FieldMetadata.jl - there is a withunits method that will apply the units to the val, range, bounds etc fields. They just don't serialize/deserialze properly to csv yet because of that Unitful.jl issue.
• @ConnectedSystems you should be able to load/save parameters, bounds etc to CSV.

There is a demo of the InteractModel in the tests, just using NamedTuple but that could be any object: https://github.com/rafaqz/ModelParameters.jl/blob/master/InteractModels/test/runtests.jl

And there are some docs, not sure if they are adequate yet.

[briochemc]

This looks very nice! And I guess it answers parts or whole of that other issue I cc'ed you on Metadata.jl!

I have zero feedback yet (because I have to run) but it looks very exciting and I promise I will give it a go! Are there examples that use Optim already out there? Or some DifferentialEquations model? Or some CliMA stuff?

[rafaqz]

I don't know that much about CLIMA, I just use it as an example of the kind of models this is designed for - complex physical/environmental models that have heirarchical components, or like DynamicGrids/GrowthMaps having lists of components. They seem to have a fairly complicated but standardised method of handling parameters and variables in the models.

But Dispersal.has used Flatten forever to automatically make parameters available to Optim - and it still just uses Flatten directly because I haven't swapped it over.

To run an optim model you need to use another wrapper object/inherit from AbstractModel and define a functor method for it, like this one: https://github.com/cesaraustralia/Dispersal.jl/blob/master/src/optimisation/optimisation.jl#L98-L113

But that could actually be simplified here, now that I think about it. We could have a LossModel functor:

struct LossModel{F,P} <: AbstractModel 
    f::F
    parent::P
end
function (model::LossModel)(params)
    model.val = params
    model.f(stripparams(parent(model)))
end

Then to use it with Optim:

parametriser = LossModel(model) do updated_model
    output = run_my_model(updated_model)
    return somelossfunction(output) # returns loss value for Optim.jl or similar
end
lower = collect(first(parametiser.bounds))
upper = collect(last(parametiser.bounds))
init_vals = collect(parametriser.val)
result = Optim.optimize(parametriser, lower, upper, init_vals, SAMIN())

Saves a bit of work I guess, not much but it at least formalises the process and would have good docs on how to do this.

[briochemc]

OK I got to play for a few mintues with it, and I have some initial feedback!

I see the

update!(model, df)

API, but is it possible to create a model from just a DataFrame? Would it make sense to add a constructor for

Model(df::DataFrame)

to work out of the box? [edit: maybe it should be Model(df::Tables.AbstractColumns) instead? Not sure]

Relatedly, one thing I really liked with FieldMetadata.jl was creating up the parameters in a table-like form. What would be the closest thing to recreating that functionality? Take a case where one has many fields, like,e.g.,

@initial_value @units @flattenable @limits @description struct MyParams{X,Y,...}
    σ_X::X | 30.0 | Mmol/yr | true  |  (0,1)  | "Magnitude of XXX"
    σ_Y::Y |  5.0 | Gmol/yr | false | (-∞,+∞) | "Magnitude of YYY"
    # and so on...

which, with ModelParameters.jl, would look more like

Base.@kwdef struct MyModel{X,Y,...}
    σ_X::X = Param(30.0, units=Mmol/yr,  optimizable=true, bounds= (0,1) , description="Magnitude of XXX")
    σ_Y::Y = Param( 5.0, units=Gmol/yr, optimizable=false, bounds=(-∞,+∞), description="Magnitude of YYY")
    # and so on...
end

This is looking good to me — don't get me wrong — but I think I prefer how the FieldMetadata version looks. (Less repetition in a more table-like input.) Is that something you thought about?

[rafaqz]

Thanks for this. There are totally some tradeoffs. The tables do look nice, but you can only abuse bitwise "or" like that inside a macro!

Half of the idea was also to get rid of those macros, because of the initial shock factor a new person has reading the code, especially when you start scaling tools to working in an organization, and your models are already really complicated. I was using Mixers.jl and FieldMetadata.jl and it just looked like another language entirely. But there could be an option where you use one macro on the whole struct and it does the Param part for you. For me the clunkyness is actually an asset right now!!

With loading directly from a Dataframe, the problem is you don't know what the model structure is. I think maybe when they add metadata to the Tables.jl interface we could include the model type there and work out a way of reconstructing the whole model from the type and parameters? But it would still need to be serialised, maybe to JSON or something.

Maybe adding a new type for non-optimized fields like Support or something could be useful too, so you can save those to the dataframe too, but not get them with params.

There are probably quite a few things to work through still with this approach, and it might not be any better than FielMetadatata.jl if you don't have to share it with other people.

It's also annoying to have things like description in the Param object - you could feasbly use both Param and FieldMetadata.jl together for things like that.

[ConnectedSystems]

Hi @rafaqz

Great to see this move towards maturity.

I'll be attempting to replace my own homegrown effort for Agtor.jl with this (as an aside, I'm curious - did I influence ModelParameters.jl in any way?). I was trying to find the time to make more use of Flatten for it, but this is way better.

Some issues should crop up in moving to ModelParameters but I suspect it will take me a few weeks to come back with any thing specific. One immediate question that comes to mind is "does this support categorical parameters?"

Thanks for all your (awesome) work

[rafaqz]

Quite a lot! The discussions we had about import/export of params from CSV was actually where the idea for ModelParameters came from. It just took a while to work out how to do it and get it written.

In terms of categorical params, I think you can pretty much use whatever you want for parameters, it returns a tuple so there is not type limitation. Including them in optimization may be another story, I've never done that so not sure what limitations there may be.

If you need to pull them out separately to the other params, you could subtype AbstractParam with CatParam - so your param will be included in params(model), but you can also do Flatten.flatten(obj, CatParam) and Flatten.flatten(obj, Param) if you need them to be separate. Some of those things we could go in the package when the use-case is clearer.

We could also make a shorthand to do that kind of querying for certain types of AbstractParam, without needing to use Flatten directly, like `params(CatParams, obj) could just do that. Or something.

There will probably be some shortcomings and things we need to add, don't hesitate to make issues. I only really have two quite similar use-cases to base my decisions on so far (GrowthMaps.jl and DynamicGrids.jl) so it's hard to know how general things are.

[rafaqz]

@ConnectedSystems Agtor looks interesting too. What kind of projects are you doing with it?

At cesar we do a bunch of agriculture-related scenario modeling with DynamicGrids.jl - (that partly supports all of these packages). We look and pest incursions, management feedbacks, and things like pest / parasite interactions and pesticide genetic resistance. It's cell-base rather that actor-based, so more for bigger-scales with way less detail than what Agtor appears to do.

[ConnectedSystems]

Ah hope you can excuse my Julia code. I'm definitely still learning the language so things could be better.

This is going to sound like marketing material but Agtor came out of my experiences with a catchment-scale modelling project wherein an integrated model (in that specific case, the integrated model represented interactions between the local hydrology, groundwater system, dam operation, and climatic factors and of course agricultural management/operation).

One bottleneck was the computational burden which certainly limited the scope of the the scale of model uncertainty, sensitivity, and exploratory analyses that could have been conducted. Having cumbersome interfaces and the interactions being handled in Python were other factors, so I started Agtor to try to address these computational aspects and to streamline the workflow in support of the modeling concerns (e.g. uncertainty/sensitivity analysis) in these integrated modeling contexts (and single-system modeling too!).

We're currently discussing how to progress this but one thing I'd like to do with Agtor is to better represent socio-cultural change and considerations such as changing on-farm water management practices, or level of consideration for environmental water needs (some farmers may prefer to forego increased water entitlements for example) - hence the focus on an actor-based approach.

[briochemc]

With loading directly from a Dataframe, the problem is you don't know what the model structure is. I think maybe when they add metadata to the Tables.jl interface we could include the model type there and work out a way of reconstructing the whole model from the type and parameters? But it would still need to be serialised, maybe to JSON or something.

I was thinking about this again and I'm not sure I see what's missing in the DataFrame about the model structure? Surely I'm missing something but could you help me see it?

[rafaqz]

If your model is just one NamedTuple then you may have everything you need to define it in a DataFrame. But my models are normally a tuple of structs with some nested structs also holding parameters, or just one complicated struct that holds other nested structs. There is no way of rebuilding these composed structures from the DataFrame of parameters - you need the actual object to reconstruct with the values from the dataframe. The dataframe just holds the parameters with no knowledge of where they came from in the composite object.

[briochemc]

So you need subsubmodels to not be able to reconstruct, right?

[rafaqz]

Yes, or your model is a struct (we don't know which one), or it's a NamedTuple with some fields that are not parameters.

We could load NamedTuple of just parameters, one issue with that is there is no limitation of models having all different field names - but I guess that would be a test of wether it could even be a simple NamedTuple model.

I pretty much always have submodels that dispatch to their own methods, or at least a single struct, so it's not a use-case for me. But if it is for you we can work out how to include that.

[briochemc]

Oh I see now! I had completely missed that Submodel2 is part of Submodel3 in the examples on the ReadMe! Thanks!

[ConnectedSystems]

I have an approach that addresses this, but I'm not sure how easily it can be adapted for ModelParameters.

In my case, I load the model structure from a series of yaml files which defines the parameter type and values (e.g. here's an example of a crop definition). My AgParameter (which I hope to replace with ModelParameters) have a "name" field. The definition gets walked through and the names (really an ID) updated to reflect the hierarchical structure. The structure of the name was inspired by (but does not exactly follow) the Basic Modeling Interface approach of using Standard Names.

So for the crop type above, it's field name (yield_per_ha for example) might be expanded out to:

Zone-ZoneName___CropField-FieldName___Crop-Canola~yield_per_ha

Which follows the pattern "ObjectType-ObjectName~FieldName", with each object separated by triple underscores ('___') and the order implies the hierarchical structure.

In my envisioned process, the DataFrame of generated parameter values have these unique names as its columns and its a "simple" matter of flattening the model collating all the ModelParameters and updating their values based on their names.

I have this working with my own implementation - see example here. Hopefully it's clear enough for you to understand the process.

As I see it, to get this style of workflow going with ModelParameters, we just need to subtype ModelParameters with a "name" or "ID" field (or add this into the current spec) and proceed as described above but there's likely some unseen complication as I haven't had the chance to look at ModelParameters in detail yet.

[rafaqz]

Ok that's an interesting way to do it.

One complication is dealing with nested structures - how would you define a parameter nested inside three layers of structs/tuples? It seems to me that whatever we use it needs to handle nesting to represent a lot of models, which doesn't work so well for a dataframe, but is easier in yaml or just julia code.

I guess I've been looking at the tables interface in ModelParameters as being slightly orthogonal to the problem of saving the entire model structure - just getting the parameters for passing to/from optimisers, for tables in papers, for saving/loading multiple parametrisations of the same model, etc. is a useful thing, but it avoids taking on the whole problem of serialising any kind of model structure accurately and entirely. I still just use a julia script for that.

But we could probably push things a lot further than that, like serializing to yaml or something - but maybe that's another package!

[ConnectedSystems]

how would you define a parameter nested inside three layers of structs/tuples?

In my approach, it generates the representation for me, it doesn't really matter how many layers/levels their are in the hierarchical relationship as it's really just [Object hierarchy]~[field]. That first part could be N layers deep.

In my case the model structure is initially mapped out by loading the YAML spec, but I also have methods to walk through a defined model as well.

If I were to adapt your example in the README, for example:

Base.@kwdef struct SubModel1{Λ,X}
    λ::Λ = Param(0.8, bounds=(0.2, 0.9))
    x::X = Submodel2()
end

Base.@kwdef struct Submodel2{Γ, d}
    γ::Γ = Param(1e-3, bounds=(1e-4, 1e-2))
    x::d = Submodel3()
end

Base.@kwdef struct SubModel3{Γ}
    γ::Γ = Param(0.15, bounds=(0.1, 0.8))
end

model = Model((SubModel1()))

The underlying parameter list would look something like:

Parameter, default_val, min_val, max_val
SubModel1~λ, 0.8, 0.2, 0.9
SubModel1___Submodel2~γ, 1e-3, 1e-4, 1e-2
SubModel1___SubModel2___SubModel3~γ, 0.15, 0.1, 0.8

One problem is that for deeply nested structures the names will get quite long. As a Symbol I guess it doesn't matter so much, but when exported to CSV or other text-based format it could become problematic. Fortunately I've yet to experience this issue...

Regarding serializing the model itself, I too avoid that for the moment. For one, the structure is already defined in YAML as I said, but for ease of reproducibility I plan to just store a copy of the model in JLD2 format along with the parameter samples used.

[rafaqz]

Ah ok you just join the nested types with __for each layer of nesting. I think that's a workable solution.

In some of my plant models there is 7 or 8 (or even more) layers of nesting because the photosynthesis model is nested inside the growth model, and both have a bunch of nesting already. But as no-one actually has to write it out its not too bad I guess, just put it in the last column of the csv!

I was serializing to julia kwarg constructors using https://github.com/rafaqz/Codify.jl for a while, but it's a bit of a hack. JLD2 is what we mostly use for DynamicGrids.jl models, although we had to be careful to only use our own or stable Base types, otherwise package updates were breaking our .jld2 files when we wanted to update project packages.

[briochemc]

A maybe cleaner-looking workaround would be to add tree symbols, like

┌──────────────┬───────┬───────┬────────────────┐
│ component    │ field │   val │         bounds │
├──────────────┼───────┼───────┼────────────────┤
│ Submodel1    │     α │   0.8 │     (0.2, 0.9) │
│ Submodel1    │     β │   0.5 │     (0.7, 0.4) │
│ SubModel3    │     λ │   0.8 │     (0.2, 0.9) │
│ └──Submodel2 │     γ │ 0.001 │ (0.0001, 0.01) │
└──────────────┴───────┴───────┴────────────────┘

[briochemc]

So you could add them automatically when converting to a DataFrame, and then use them and strip them when building the model from a DataFrame? And if none of these symbols appear, it would still return a "flat" model I guess?

[briochemc]

@rafaqz maybe you want to chime in https://discourse.julialang.org/t/nicer-parameter-indexing/50704? I think a small example of using SciML would be great! Do you think you could make one? Even better would be an example that works with ModelingToolkit.jl too! (In particular with the @parameters macro)

[rafaqz]

I think ModelParameters is for solving a problem that they don't really have... as that is a DSL already and the parameters are already in a vector. Model parameters is more for simplifying more heterogeneous models where you don't have a DSL, and define your components using structs and methods of them.

Maybe it could be used for more complicated DiffEq models where you can't use the DSL...

I also find the DiffEq macros deeply confusing, and don't understand how much of that example works :rofl: