tf kernel

[kkappler]

ProcessingSummary Data Frame exists, it is built by TransferFunctionKernel (TFK)

Next steps:

• [x] Modify process_mth5 to use tfk. In future, it may be passed as an argument, instead of config, and dataset, but probably best to keep passing config and tfk dataset separately (like it is now) and then TFK gets built as first step in process_mth5. This will leave the existing ipynbs as backward compatible.

• [x] Move processing_config.validate_processing(tfk_dataset) into TFK

• [x] move initialize_config method into TFK and out of process_mth5

• [\] Modify tfk_dataset.initialize_dataframe_for_processing(mth5_objs) so that RunTS is not packed into df, only the run reference and the mth5 obj ** The way things are now, looping on decimation level, there is no clean way to loop over run without intermediate storage of decimated time series, otherwise we will need to redecimate the data for the 1st, 2nd, 3rd, and Nth decimation level all the way from zero. Better to await implementation of FC and or dec level data structures in mth5.

• [\] Review the processing main loop; It should be done in a way that is groupby [survey, station, run, start, end], that way, every individual RunTS will be loaded only the one time ** See previous point - without storage we will take a serious performance hit in compute cycles.

  • [X] Use processing_summary as the main loop, and check that row is valid (per #103) before processing
    • The spirit of this task is implemented, if not the letter. We now loop over only decimation levels that are asserted to be valid (i.e. there are "enough" Fourier coefficients in the time series)

• [X] Fix all tests so that they are passing with this new method

• [\] Modify all ipynb so that they are passing with this new method

  • Not explicitly checked, but the TFK is implemented inside process_mth5 so there should be no change in behaviour of these notebooks.

[kkappler]

Extra Items

• [x] Time permitting, add the tasks from issue #159 so PKD tests are cleaner
  • [x] remove reference to SAO-only test config
  • [x] Replace all references to h5 files with pkd_sao_test_00.h5
  • [x] Create method called ensure_data_exists() which makes PKD data and handles reporting of NCEDC issues
• [x] Move the saving of json files out of config creator, and make a method of Processing

Once that is integrated, review whether to make tfk an input, or to give tfk a process() method, that basically is the same as process_mth5.

[codecov[bot]]

Codecov Report

Merging #242 (afc0e5f) into main (d9932f0) will increase coverage by 1.38%. The diff coverage is 89.71%.

@@            Coverage Diff             @@
##             main     #242      +/-   ##
==========================================
+ Coverage   77.53%   78.92%   +1.38%     
==========================================
  Files          99      103       +4     
  Lines        5493     5798     +305     
==========================================
+ Hits         4259     4576     +317     
+ Misses       1234     1222      -12     

Impacted Files	Coverage Δ
aurora/config/config_creator.py	90.62% <ø> (-1.27%)	:arrow_down:
aurora/config/metadata/decimation_level.py	81.17% <ø> (ø)
aurora/pipelines/transfer_function_helpers.py	87.17% <ø> (-1.46%)	:arrow_down:
aurora/time_series/frequency_band.py	68.11% <ø> (-0.46%)	:arrow_down:
setup.py	0.00% <ø> (ø)
aurora/pipelines/time_series_helpers.py	75.18% <50.00%> (-0.39%)	:arrow_down:
...ra/test_utils/synthetic/make_processing_configs.py	75.55% <51.85%> (+5.18%)	:arrow_up:
aurora/sandbox/mth5_channel_summary_helpers.py	82.75% <60.00%> (-13.67%)	:arrow_down:
aurora/general_helper_functions.py	61.29% <71.42%> (+1.29%)	:arrow_up:
aurora/test_utils/parkfield/make_parkfield_mth5.py	80.85% <75.00%> (-0.24%)	:arrow_down:
... and 24 more

... and 2 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.