search

simpeg / aurora

software for processing natural source electromagnetic data
MIT License
14 stars 2 forks source link

Parkfield calibration test works on local data but not on newly downloaded #161

Closed kkappler closed 2 years ago

kkappler commented 2 years ago

Noticed that test_calibrate_parkfield.py is failing in gitactions tests. Ran locally and it passed. Moved my existing data/pkd_test_00.h5 out of its folder and re-ran the test, resulting in re-downloading the data and got a failure.

So, it seems to matter which pkd_test_00.h5 is used. The old one dates back to Nov 23, 2021.

Maybe something changed in mth5 or mt_metadata?

kujaku11 commented 2 years ago

I'll have a look tomorrow. Its probably an mth5 thing.

On Sat, Mar 26, 2022, 1:44 PM kkappler @.***> wrote:

Noticed that test_calibrate_parkfield.py is failing in gitactions tests. Ran locally and it passed. Moved my existing data/pkd_test_00.h5 out of its folder and re-ran the test, resulting in re-downloading the data and got a failure.

So, it seems to matter which pkd_test_00.h5 is used. The old one dates back to Nov 23, 2021.

Maybe something changed in mth5 or mt_metadata?

— Reply to this email directly, view it on GitHub https://github.com/simpeg/aurora/issues/161, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQA5BRJVJ3F5WB4HGRR533VB5ZMFANCNFSM5RXUNZBQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

kujaku11 commented 2 years ago

@kkappler It looks like the issue is that ZPK filters pulled from NCDEC are in units of Tesla, whereas the lookup table is in nT.

If the units are converted to nT then everything looks good. So I'm not sure where the change occurred if it was originally in nT and or there was some conversion somewhere that has been skipped now. But the tests pass in mt_metadata and MTH5 of reading in a filter and outputting the same filter.

In aurora branch issue_153 I added a few print statements to aurora\tests\parkfield\calibration_helpers.py and 

if channel.channel_response_filter.units_in.lower() in ["t", "tesla"]:
            print("WARNING: Expecting nT but got T")
            pz_calibration_response *= 1E-9
            print("\tConverting units to nT")

Now not sure how this affects the pipeline, but should just check the units on the filters and all will be good.

kkappler commented 2 years ago

Thanks @kujaku11 - I see now that I introduced this bug on dev branch recently. I am testing a fix now.

It was because I was bypassing:

if stage.input_units == "T":
    stage.input_units = "nT"
    stage.stage_gain *= 1e-9