Variables, parameters and observables

This replaces #603. We introduce standard ways of parametrising a lattice, varying any quantity and looking at any resulting quantity. There are 3 main classes:

Variables

Variables are references to any scalar quantity. AT includes two predefined variable classes referring to scalar attributes of lattice elements:

an ElementVariable is associated to an element object, and acts on all occurences of this object. But it will not affect any copy, neither shallow nor deep, of the original object,
a RefptsVariable is not associated to an element object, but to an element location in a Lattice. It acts on any copy of the initial lattice. A ring argument must be provided to the set and get methods to identify the lattice.

Variable referring to other quantities may be created by:

Deriving the Variable base class. Usually this consist in overloading the abstract methods _setfun and _getfun,
Using the CustomVariable class, accepting user-defined get and set functions.

Parameters

Parameters are objects of class Param which can be used as Element attributes instead of numeric values.

Arithmetic combinations of parameters create new read-only parameters of class ParamBase, whose value is permanently kept up-to-date. This is useful to introduce correlation between attributes of different elements.

Presently the use of parameters is limited to Element attributes, but it may me extended in the future (ex: Lattice attributes).

Observables

Observables provide a unified way accessing a large quantity of figures resulting from various computations on lattices. They may be used in parameter scans, matching, response matrices…

AT provides a number of specific observables sharing a common interface, inherited from the Observable base class. They are:

OrbitObservable,
GlobalOpticsObservable: tunes, damping times…,
LocalOpticsObservable: β, η…,
MatrixObservable: T_ij…,
TrajectoryObservable: x, p_x…,
EmittanceObservable: ε_x…,
LatticeObservable: attributes of lattice elements,
GeometryObservable

An Observable has optional target, weight and bounds attributes for matching. After evaluation, it has the following main properties:

value
weighted_value: value / weight
deviation: value - target
weighted_deviation: (value - target)/weight
residual: ((value - target)/weight)**2

Custom Observables may be created by providing the adequate evaluation function.

For evaluation, observables must be grouped in an ObservableList which optimises the computation, avoiding redundant function calls. ObservableList provides the evaluate method, and the values, deviations, residuals and sum_residuals properties, among others.

The documentation for the branch is available here. Please read and comment.

The new classes introduce name conflicts with existing Variable, ElementVariable classes. They also change the interface of matching. Keeping two slightly different matching functions is not desirable, so the idea is to move from the old scheme to the new one and replace the match function. The proposed strategy is:

Starting with this PR, the access to the new classes and functions is:
```
from at.future import match, ElementVariable, ...
```
The old classes are still available without any change with:
```
from at import match, ElementVariable, ...
```
or using at.match(...), at.ElementVariable(...).
At some point (version 1.0?), we will switch the default to the new scheme, and the old one will need something like:
```
from at.deprecated import match, ElementVariable, ...
```

Dear @lfarv,

this is a very big pull request (26 files!). I have no idea how to "test" it completely.

There are many new classes, many new features. How to test each of them in realistic conditions?

I may test the matching for example.

The known breaking of backward compatibility is also a major issue for me. I stopped using matlab AT for broken backward compatibility. May be some other users will stop using pyAT for this broken backward compatibility. There are big competitors out there, such as XSuite... May be new names could be a better solution?

I am looking at the notebooks in docs/p/notebooks. VERY USEFUL to get trough all this work!

Observable.ipynb

It seams the user has to remember a quite large number of *Observables. I find this not so user friendly. More examples could be of help. Else switching between the various modes with keywords in the same Observable definition (r_in swithes to Trajectory, etc..)?
The r_in argument to evaluate() could instead be an argument of TrajectoryObservable? This allows to have different r_in for different Observables.
The dp argument is global for all observables. However it may happen that we want to observe a quantity and the same at different dp, during the same observation. Also dp seems to belong better at the single *Observable level.
The call LocalOpticsObservable([33, 101], phase_advance,... Will compute internally at all locations between 0 and 101? if not the phase_advances may be incorrect (may be this has been solved in an other pullrequests?).
Can statfun argument be used also with LocalOpticsObservable? (I see myself using this LocalOpticsObservable very often)
can regexp be used to specify LocalOpticsObservable locations?

parameters.ipynb

I think that this is the natural user expectation

p5 = Param(-0.3, name='QD strength')
qd1.PolynomB[1] = p5

I think that it would be wise to make the above lines work as expected from the user (exactly as in the example!) rather than giving a warning. There are several "work arrounds" explained in the notebook. It remains a bit disappointing. Why going trough all this development if we may not do what we wanted to do when this was started (easy, user friendly, parameterized matching)? I think that the qd1.PolynomB[1] = Param(-0.3) would have to work in the final version.

I like the parameter history.

variables.ipynb

seems incomplete, here is only the header

_testvariables.ipynb

Why is it needed to make VariableLists and ObservableList? Could they simply be python lists?
```
variables = VariableList([param1])
constraints = ObservableList(hmba_lattice, [obs1])
```
clearly method1, exploiting parameters looks much simpler than method2 creating ElementShifter. The second may be nevertheless useful for more complex optimizations.

These notebook examples are really a good way to understand what the code does. Could we have more examples for the use of other classes mentioned in this pull request? MartrixObservable, GlobalLatticeObservable, Trajectory and Gemoetry observables?

thank you

best regards

Simone

Hi @simoneliuzzo. Very nice review of the present state of things, thanks !

I'll give today the "easy" answers, I'll go step by step later for the rest.

Variables

variables.ipynb

seems incomplete

Work in progess, there is now a better version

_testvariables.ipynb

Why is it needed to make VariableLists and ObservableList? Could they simply be python lists?

Just to add vectorised methods corresponding to the individual ones. Examples: VariableList.set(values) to apply a vector of values, ObservableList.values to get a list of all Observable values, ObservableList.sum_residuals, etc. Plus pretty-printing of ObservableLists.

Observables

It seams the user has to remember a quite large number of *Observables. I find this not so user friendly. More examples could be of help. Else switching between the various modes with keywords in the same Observable definition (r_in swithes to Trajectory, etc..)?

This is a design decision. When reading the code, it makes clear what each object does, and it makes each class more efficient by avoiding "switch ... case..." tests. What could be done would be to add a kind of factory function on top, which would create the right class based on ... what? A keyword, a flag ? that's not easier to remember. That's why the notebook goes through all classes to give an example of each.

Could we have more examples for the use of other classes mentioned in this pull request? MartrixObservable, GlobalLatticeObservable, Trajectory and Gemoetry observables?

observables.ipynb gives one example of each of all the available Observables. But it can easily be modified, suggestions are welcome!

Can statfun argument be used also with LocalOpticsObservable? (I see myself using this LocalOpticsObservable very often)

Of course it can! See in the given examples

LocalOpticsObservable(at.Monitor, 'beta', plane='v', statfun=np.amax)

to get the maximum vertical β on monitors.

The call LocalOpticsObservable([33, 101], phase_advance,... Will compute internally at all locations between 0 and 101? if not the phase_advances may be incorrect (may be this has been solved in an other pullrequests?).

This is triggered by the use_integer=True keyword. See the LocalOpticsObservable help for its explanation.

can regexp be used to specify LocalOpticsObservable locations?

Not now. You can always do it in two steps

refs = ring.get_bool_index("pattern",  regexp=True)
pbs = LocalOpticsObservable(refs, 'beta', ...)

It can easily be added. I'll do it. But now, regexp is still not accepted in the dozens of optics functions using a refpts argument. So why accepting it here?

Hello @lfarv, thanks for reviving this. I must admit that this is the result of long discussions and many iterations on github and private emails and II have completely lost track and forgot what were the decision taken.

I completely agree with @simoneliuzzo that this is too big to properly review and the github discussion on such large PR will quickly become a total mess. I would really appreciate that this PR is split as follows (I think this was already requested in the previous PR): 1-variables 2-Observables 3-Parameters 4-matching

We can then validate each development gradually and potentially improve the next ones using the review comments. So let's start with variables, I already have several points to discuss on that part. I wait for the new PR to send them/

@swhite2401: the split is done.

However, variables and parameters are so intricate that any change on parameters in the proposed step 3 would change again the modules approved in step 1. So we go for: 1- variables and parameters 2- observables 3- matching 4- response matrices

atcollab / at