User Applied Filters and Custom Transformations

[kkappler]

Story

As a data processor I want to apply some operation to my time series and then store the output time series in an MTH5.

These operations might be of two flavors:

1. Some linear filter that I can characterize using the existing filters in mt_metadata.
2. Some operator (maybe defined within a method, perhaps non-linear)

I would like to add an annotation to my MTH5 so that when I come back to open the file later (or share it with another person) there is a record of the process that was applied.

In case (1) I should be able to describe my filter using one of the pre-existing containers (e.g. PoleZeroFilter, TimeDelay, etc.).
In case (2), I may not have completely characterized the operator in terms of its complex response – the operator may be not even have a simple inverse, or at least not a known one. At a minimum I would like to describe the process with a keyword or name, any parameters that were applied to the operator, and the order in which it was applied (if I did several of these transformations).

These filters are in general not removed by default when calibrating the data, i.e. calling remove_instrument_response.

Background:

Currently, MTH5 offers a Filters Group, that can be used to keep an archive of filter operations during data acquisition. Typical examples of these filters are the transducer response, analog to digital converter, or really, any other stage of linear analog or digital operation on the data before it is archived.

Under the Station > Run > Channel level of the MTH5, each channel is associated with calibration filter information via a list of keywords, which specify the filters, and a corresponding list of booleans, describing if the archived data have the filter applied, or not applied.

The Calibration stage of processing uses these information to determine how to transform the data into physical units from what is stored in the MTH5.

Potential Solution:

Requirements:

• Provides a record of additional filters applied to a channel
• If these are linear, we can represent it as PoleZero, FIR, coefficient, etc.
• Can also be defined by a method filter_operator(data, *args, **kwargs)

Ideally, we would augmenting the filters_list of ChannelResponse with the additional user_applied filters

• Solution (1) Allow augmenting of ChannelResponse with additional filters

  • Case (1): Working with "standard linear filters"

  • use the existing classes in mt_metadata

  • Set their “applied” booleans to True.

  • This would require additional logic in remove_channel_response because we would generally not want to remove these filters during calibration.

  • We could add an attribute to the FilterBase called user_applied which defaults to False, and then make user_applied filters get ignored by default in calibration / remove_intstrument_response.

  • i.e. calibration has a step get_list_of_filters_to_remove(self, include_decimation=False, include_delay=False), this could be modified to get_list_of_filters_to_remove(self, include_decimation=False, include_delay=False, include_user_applied=False)

  • when user applying custom filters they would tag user_applied=True

  • Note that remove_instrument_response does support a general kwarg filters_list, which the user can leverage if they want to unapply their filters.

  • Case (2): Custom (possibly non-linear, applied via algorithm), applied filters

  • Create a new class for these that extends FilterBase

  • For now this could be SpecialFilter

    • this would have user_applied=True by default
    • would not have things like obspy_mapping, calibration_date, and may not have a complex_response,
    • may rely on name and description to tell the user about itself.
    • may have standard properties like units in, units_out,

This should at least support a record of a filtering workflow.

The other way around this would be to create an attribute, similar to ChannelReponse, called say UserAppliedFilters, or similar that lives at the same level as ChannelReponse, but that seems more complicated.

[kkappler]

To prototype this, most changes will be in mt_metadata, but mth5 ts_filters will also need an update.

Steps:

• [x] 1. Add user_applied attr to FilterBase.

• The default value is false.

• I set this as required=true, it seemed simpler than to add a method to FilterBase class user_applied that tries to return self.user_applied, and if it gets and attribute error, returns False

  {
  "user_applied": {
      "type": "bool",
      "required": true,
      "style": "name",
      "units": null,
      "description": "If True, filter was generated by user, not calibration metadata.",
      "options": [],
      "alias": [],
      "example": true,
      "default": false
  },

• [x] 2. Update mt_metadata channel_response.py:

• Functions get_indices_of_filters_to_remove, and get_list_of_filters_to_remove both need kwargs include_user_applied=False

• Logic in both the functions needs updating to ignore filters with user_applied=True

• [ ] 3. Update mth5 remove_instrument_response

  • takes kwarg include_user_applied=None
  • call to get_list_of_filters_to_remove updated with this kwarg filters_to_remove = self.channel_response.get_list_of_filters_to_remove( include_decimation=include_decimation, include_delay=include_delay) include_user_applied=include_user_applied

• [ ] 4. update tests