Separate hyperparameters

[TTitscher]

Hyperparameters are parameters that are not direct input to the model error computation, but are rather used outside (, right?!) One example would be parameters of the noise distribution. For the current VB implementation, they are a separate input to the method and for sampling, they are used in the transformation of the model error to a likelihood function.

Having those, conceptually, in two separate objects, allows us to define the problem independent of the inference method used. In some "simple" cases -- in my experience the majority though -- a small wrapper around the problem definition or the inference methods can lead to a drop-in replacement of the methods. So without any effort, you can start your investigations with a scipy.optimize, validate with sampling and switch to VB for performance or larger problem size.

[joergfunger]

On the one hand, I agree that they are separate objects, since they are not required in the forward model. On the other hand, the structure is quite similar to the model parameters (so there can be hyperparameters local or shared by different data sets/experiments). The structure can even be used for scipy optimize or pyMC3 as shown here. At least from my own experience with the framework, it is not very intuitive yet - maybe that's due to the lack of a general concept we have not yet agreed on or documented and thus adding another parallel hyperparameter layer might increase the complexity again, even though after some online discussions the code is now much clearer. The usecase shown in the link above is currently working (still needs some debugging) with pyMC3, the next step would then be to get it to work with VB. Maybe that would help to get an agreement on how we should use the framework.

[TTitscher]

I try to summarize some internal discussion with @joergfunger

• The library does require/provide a way to deal with hyper parameters. That is user business.

• A straight-forward option would be for the user to create a hyper-parameter-list, add it to the "problem" and maybe even set it latent. That way, those parameters are treated like model error parameters.

  hyper_prm, hyper_key = ParameterList(), "hyper"
  hyper_prm.define("sigma")
  existing_latent.define_parameter_list(hyper_prm, hyper_key)
  existing_latent.add("global_sigma_to_infer", sigma, hyper_key) # or whatever the current syntax is

• Another idea already implemented adds the hyper parameters to all existing model parameters.

  for prm_list in prm_lists:
  prm_list.define("sigma")
  existing_latent.add_by_name("sigma") # to make all len(prm_lists) sigmas shared

• Instead of this uniform treatment by the library, there will instead, for convenience, be an automatic conversion from a "VariationalBayesProblem" to a sampling problem or general "loglike_f". This will always be possible as the "easy" distributions of VB (MVN or Gamma prior and posterior for parameters and noise, respectively) should be present in any other inference tool and each output of VB (arranged in noise groups) is associated to one term in a loglike_f.

[TTitscher]

IMO this is solved with the LatentParameters and NoiseModel classes.