Process Musgraves Dataset

[kkappler]

For the "2030" Geophysics Community Connect Workshop

Started processing Musgraves on Gadi, using a notebook modelled after process_synthetic.ipynb

Here are the encountered issues:

• [x] Permissions Error, cannot open in Append Mode The musgraves data are readable but not writeable. Copying the data to my scratch is a last resort, better to modfiy aurora to open in "r" mode, unless save_fc is True. This can be handled by logic in TFK, and processing_config.
• open in read-mode was applied to aurora v0.3.4 patch, we can use the old datasets for now. We should still rebuild the datasets if time permit but we don't need to for the workshop.
• [x] Run label in tfk.dataset.df (ultimately from mth5 channel summary) doesn't match run_obj.metadata.id This is likely because the data were created using an older version of mth5. Rebuilding the data with latest release should fix. Otherwise we can workaround in process_mth5 with:

  try:
  assert row.run_id == run_obj.metadata.id
  except AssertionError:
  print("WARNING Run ID in dataset_df does not match run_obj")
  print("WARNING Setting run_obj metadata to match dataset_df")
  run_obj.metadata.id = row.run_id

  right before make stfts. I have added this workaround to earthscope_tests branch and will merge into dev and then main depending on whether or not we rebuild the dataset

• This workaround has been applied to main branch in v0.3.4
• [x] Channel Nomenclature:

“bx”, “by”, “bz”, “ex”, “ey” has not been encountered before. Registering it properly would require a patch on mt_metadata. Alternatively we could allow "define on the fly" in aurora ... TBD

This raises the question of the role of channel nomenclature, which is to accommodate MTH5 where the channel labels deviate from the standard ["ex", "ey", "hx", "hy", "hz"].

• We do not wish to modify the hdf5, since we dont know where it is from.

• In the output TF we need to adopt a policy on nomenclature - agree with input h5 or standardize, or allow control. For now, we are returning the TF in the same coordinates as the input MTH5, but forcing HEXY format on the mt_metadata XML output.

• [x] Transfer function is computed but doesn't cast to mt_metatdata.transfer_function.core.TF Needs debugging ... this is likley associated with channel nomenclature, and should be diagnosable via examining aurora/tests/synthetic/test_processing.py and comparing behaviour of test_can_use_channel_nomenclature()

Once the above issues are tackled, we can then:

• [x] build an index table of all the musgraves data
• [x] use a tools similar to earthscope_tests widescale.py to iterate on the table in parallel For a first test:
  • [x] Extract from the dataframe table only the level1 data
  • [x] process as single station to see what fraction of data result in xml of 89 files run, all resulted in XML in 1m21s For a Second Test
  • [x] Given an L1 dataframe, search for all combinations of RR and Local Station
  • instead of doing combinatorics I just used the station pairs that Nigel shared.
  • [x] Add remote_path, remote_station_id to a new (melted) dataframe
  • [x] Iterate over RR processing
  • [x] Time testing with n_processors, etc.

[kkappler]

This may get re-opened for a second pass, as it would be good to compare the TF values obtained by BIRRP and aurora, but the pandarallel processing ran for SS and RR so we will have an example for the workshop. Closing for now.