search

simpeg / aurora

software for processing natural source electromagnetic data
MIT License
13 stars 2 forks source link

Python data structure for writing to TF.xml #79

Closed kkappler closed 2 years ago

kkappler commented 3 years ago

aurora/pipelines/process_mth5.py has a function called process_mth5_run This function refrerences a class called TransferFunctionCollection() TransferFunctionCollection() is where the outputs at each decimation level are aggregated.

The local_run_obj and remote_run_obj, and the config need to be merged with TransferFunctionCollection to complete a data structure that can pipe to XML

kkappler commented 2 years ago

A class from mt_metadata is being implemented for this. The main processing call process_mth5_run was returning a tf_collection object. I am changing this to normally return mt_metadata.transfer_functions.core.TF

In order to not break existing tests, I added a kwarg to process_mth5_run called return_collection=True.
The default behaviour will be to return a TransferFunctionCollection object, but if return_collection=False a mt_metadata.transfer_functions.core.TF will be returned. Once the tests have been updated, we can then change the kwargs default to False.

To get the prototype running on the parkfield test I needed to add hz to workaround a KeyError in the merge_decimation_levels. We may want to add a 4-channel and 5-channel version of the parkfield test. The four channel case would have only horizontal sensors, and the 5-ch case has hz and pulls data from NCEDC.

Robustified merge_decimation_levels to return nan for tipper if no hz -- This is not very clean code - needs a review

kkappler commented 2 years ago

Before closing this issue, review this commit. It started off with some fairly complicated merging but later I found an easier way. However, the complicated xarray merging stuff probably made it into the commit, albeit unused in the normal codebase, but it would be good to review and flag any outstanding issues here.

Specifically, I was trying to add a single channel xarray to an existing one and was finding merge was not working for single channel. I could add multiple (two) channels, overwrite one of them and drop the other, but this is hacky. Better to illuminate clearly the syntax for adding a single channel and raise an issue on xarray of this is not a clean and easy thing to do.

commit f08a23dd6c749b8d2f4be581d207b1a21151f01a Author: Karl N. Kappler magnetotellurics@gmail.com Date: Sat Sep 25 16:35:41 2021 -0700

TF export

-added the ability to return mt_metadata.transfer_functions.core.TF from process
_mth5_run via an export_tf method
-modified parkfield run test so that hz is included
-needed to source data from NCEDC rather than IRIS to get hz data
-this was needed to workaround a KeyError in merging the TF collection
-Factored and robustified merge_decimation_levels against not having hz
-fixed a minor bug in emtf_z_file_writer so that now only the processed channels
 are listed in the z-file header channel list, rather than all channels in the m
th5
-added methods to merge and reformat the transfer function collection for castin
g to mt_metadata.transfer_functions.core.TF
-some of these methods are not pretty - in particular a work around for not havi
ng tipper data present was requires for both tf and cov_nn

[Issue: #79]
kkappler commented 2 years ago

actually, it looks like the hacky code I reference in previous comment never made it into the commit so I am closing this issue.