Compute features before dataselection

[pascalau]

I'm currently working on providing better information to the user which parts of a log contain useful information to estimate the parameters based on Fisher Information. I have a first prototype working on a very old branch with a hacky solution (See pr-fisher-information-preview). To calculate the metrics it is necessary to calculate the features (X matrix) before the data selection screen. I would therefore propose the following changes:

• Calculate the features before the data selection takes place and store the features in the dataframe as well.
• Pass the dataframe to the optimizer as a whole with instructions for the optimizer on how to assemble X and y
• Move the functionality of the data selection from the data handler to the dynamics_model/model_template.

This would allow for the following pipeline:

• Dataloader loads the data and resamples it. Hands the data in form of a dataframe to the model.
• Model calculates features based on the dataframe and stores them back into the dataframe. The same applies to the y values. The model also fills out a dictionary to which y value the feature has a contribution.
• Model calls feature selection and crops the dataframe to only contain the relevant data.
• The dataframe gets passed to the optimizer with the aforementioned dictionary and locally constructs the regression matrix and performs the parameter estimation. This allows to operate on multiple smaller matrices instead of constructing a large mostly sparse matrix which is more difficult to solve for the lapack backend.

The advantages of this approach are:

• The data is contained in a single object and not scattered in multiple objects. This should make it a lot easier to implement other features in the future to preprocess data or something similar.
• It is a lot easier to keep the memory footprint of the program low, especially when working on large logs since there should not be any unnecessary copies of data.

[pascalau]

I have a first functioning prototype on the branch pr-improved-dataflow. The prototype first saves all features in the data frame and uses the function assemble_regression_matrices() to create the X,y objects.

[manumerous]

That sounds great! Do you plan to include the fisher information into the visual dataframe selector at some at some point? It would be very cool and straight forward to visualize the fisher information metric in the visual dataframe selector in an additional subplot. Feel free to reach out to me if you have any questions on that. :)

[pascalau]

See #176

[Jaeyoung-Lim]

Closing as addressed in https://github.com/ethz-asl/data-driven-dynamics/pull/176