Design data transformation interface.

[bradyzp]

As we approach a point where we can begin to process data, we need to figure out how to handle transformations on data series.

Prerequisites: Issue #32 'Join gravity data with trajectory data': Do we want to merge the two pandas DataFrames, perhaps after performing a synchronization on them? If we merge and perhaps enable the user to trim the ends (junk data) into a new DF ready for processing while keeping references to the original data, this might pave the way for further processing.

Issue #35 'Line Selection': Should we enable the actual splitting of lines after selection - i.e. plot each line segment on it's own, and allow filters/transformations to be applied individually? User could then presumably select the line they are interested in on the Project Tree View for plotting and manipulation.

---

How to apply arbitrary filters/transformations to data? Some initial thoughts/ideas:

• Maintain an ordered list of transformations that are then applied to the data before plotting/export. This would make it easier to add and remove filters/transformations without performing destructive operations on the underlying data.

• Create a wrapper Class for pre-processed DataFrames (Issue #32 & #35) allowing the application of filters etc. to specific segments, with each segment maintaining its own state. Class could provide easy interface to get the processed data for plotting & export. e.g.

  class DataWrapper:
  def __init__(self, frame: DataFrame):
      self.df = frame
      self.transforms = {}
  
  def apply_transform(self, transform: Callable):
      # Allow application of transformation function (Callable)
      self.transforms[uid] = transform
      ...
      # Return uid of transform for future reference
      return uid
  
  def remove_transform(self, uid):
      del self.transforms[uid]
      ...
  
  def data(self):
      # Apply transforms
      ...
      return TransformedDataFrame

[cbertinato]

Do we want to merge the two pandas DataFrames, perhaps after performing a synchronization on them?

I think this is the way to go. Sync first, then merge. The data were are dealing with, even for long flights, isn't huge, so I think we can afford to produce a separate DataFrame for the merged data as I think we had discussed a while ago.

Should we enable the actual splitting of lines after selection - i.e. plot each line segment on it's own, and allow filters/transformations to be applied individually?

If I get your meaning, that we would break out DataFrames after line selection on which the user could then operate, then I think that this is a good model. Perhaps one could operate on a line-by-line basis, or apply to all lines in a flight.

[cbertinato]

I would like to build this feature with the flexibility to eventually implement custom transform blocks. Toward that end, we will need to define how the user would interact with this feature. Some thoughts on that here:

• standard transforms like filters, derivatives, corrections (column-wise addition/subtraction) can be built-in, available to be added in a transform chain
• custom transforms need to be pickle-able
• custom transforms can be added by specifying a function, inputs, and outputs
• composite transforms: model as a transform chain, or allow use of transforms within a transform function, such as derivatives?
• transforms and transform chains take a DataFrame and return a DataFrame

Transform chains consist of callable objects that are instances of the transform class. The transform class is a container for the function to be applied.

[bradyzp]

Closing this as resolved in commit f9646044c528d26610cfedcaaa6100a5c7942548 Further discussion in issue #65