Add a test dataset to validate against EMTF

[kkappler]

Relates to #14

It maybe nice to make an integrated test out of this. When a commit is done this test can be run and if the aurora TFs are not np.close() to the EMTF ones the commit is flagged

Earthscope Test

[kkappler]

Devlopement of plan for working the earthscope data is here

House the data at IRIS at have an integrated test that generates and processes an mth5

[kkappler]

@kujaku11 to wrap these data as an mth5

[kkappler]

mth5 success. Next step is test run through aurora driver

[kkappler]

@kujaku11 I tried loading the mth5 that we created on Friday but I get an error when I try to read it:

m = MTH5() m.open_mth5(SYNTHETIC_DATA, mode="r") mth5.utils.exceptions.MTH5Error: Input file is not a valid MTH5 file

if you pull you'll see that i updated the file creator and tried adding a function to access the data. I couldn't find the string "open_mth5" in the mth5 repo with an "r" on it. I can access the file if i set mode to "w", but when I try to

m.get_station("mt001") i get KeyError: "Unable to open object (object 'mt001' doesn't exist)"

and when try: m.get_run("mt001", "001") i get: raise MTH5Error(f"Could not find {station_name}/{run_name}") mth5.utils.exceptions.MTH5Error: Could not find mt001/001

[kujaku11]

@kkappler Try opening the file with an mode="a", for append that way you can read and write an existing file.

If you open with "w" it will overwrite the existing file. Looks like trying to open with "r" returns an error which is a bug, I'll add an issue to MTH5.

[kujaku11]

@kkappler added MTH5 issue #36 and fixed it. The updated code is now in master, which has been merged with experiment to get ready to release a beta version on wednesday.

If you open a file in mode="r" a warning is raised to alert the user that the file is not writeable.

[kkappler]

OK, I've got the run_ts time series up and ready to go, I can window, taper, and Fourier transform.

Next step is calibration. We will need to check with Gary what filters we need to add, for now I thought I would just add a dummy coefficient filter that multiplies by 1.

@kujaku11 I tried from mt_metadata.timeseries.filters.coefficient_filter import CoefficientFilter coeff_filter = CoefficientFilter() m.survey_group.metadata.filters["filter1"] = coeff_filter but the dictionary doesn't pick up an entry.

Is there an example way to:

1. without an xml, populate the survey filters with some values?
2. To associate each channel with that filter as well? You can feel free to add examples to: aurora/data/emtf_synthetic/make_mth5_from_asc.py inside the method create_mth5_synthetic_file()

[kujaku11]

@kkappler you should be able to do something like:

from mt_metadata.timeseries.filters import CoefficientFilter
cf = CoefficientFilter()
cf.units_in = "digital counts"
cf.units_out = "millivolts"
cf.gain = 1.0
cf.name = "unit_conversion"
cf
Out[38]: 
{
    "coefficient_filter": {
        "calibration_date": "1980-01-01",
        "gain": 1.0,
        "name": "unit_conversion",
        "type": "coefficient",
        "units_in": "digital counts",
        "units_out": "millivolts"
    }
}

cf_group = m.filters_group.add_filter(cf)

[kkappler]

Rudimentary tools for comparison (plot and z-file interface ) exist. This issue is basically closed but we still need to get agreement as exact as possible. The steps involved to do that include:

• refine z-file writer with declination, FORTRAN style numerics, correct carriage returns etc.
• step-by-step through the FORTRAN processing parameters so that the metadata in aurora zss file agrees with that in the FORTRAN (number of windows, window function, band setup, etc)
• Get an ascii dump of the matlab .zss file. (it doesn't load in Octave) iris-mt-scratch/iris_mt_scratch/egbert_codes-20210121T193218Z-001/egbert_codes/matlabPrototype_10-13-20/test/syntheticTests/TS1_zss.mat

[kkappler]

The synthetic test dataset is added and integrated has tests