search

pysat / pysatNASA

pysat support for NASA Instruments
BSD 3-Clause "New" or "Revised" License
21 stars 6 forks source link

GUVI imaging updates #198

Closed aburrell closed 8 months ago

aburrell commented 1 year ago

Description

Addresses #197 by adding a custom concat_data method to the TIMED GUVI instrument. This requires the pysat pull request: https://github.com/pysat/pysat/pull/1131 to work.

Type of change

Please delete options that are not relevant.

How Has This Been Tested?

Load TIMED GUVI data for the sdr-imaging tag. Check and see that the Auroral data does not have the 'time' dimension, rather it has only the 'time_auroral' dimension (in addition to other appropriate dimensions). Also, the extent of the 'time_auroral' dimension should be limited to the period of time encompassed by 'time'.

import datetime as dt
import pysat
import pysatNASA

guvi = pysat.Instrument(inst_module=pysatNASA.instruments.timed_guvi, tag='sdr-imaging', inst_id='low_res')
guvi.load(date=dt.datetime(2005, 6, 29), use_header=True)
guvi['DISK_RECTIFIED_INTENSITY_DAY_AURORAL']

Yields:

<xarray.DataArray 'DISK_RECTIFIED_INTENSITY_DAY_AURORAL' (nCrossDayAur: 13, time_auroral: 5694, nchan: 5)>
array([[[7209.4736 , 4871.778  ,  590.9235 ,  730.5943 ,  717.60645],
        [6576.525  , 4667.561  ,  621.9355 ,  707.2645 ,  665.0437 ],
        [6809.1646 , 4822.3955 ,  596.235  ,  748.06116,  691.1846 ],
        ...,
        [4474.8916 , 2927.6484 ,  298.7434 ,  364.07367,  367.03128],
        [4596.163  , 3004.093  ,  346.05643,  425.2893 ,  366.12378],
        [4790.0947 , 3107.468  ,  339.42386,  416.37698,  412.0753 ]],

       [[7700.39   , 5367.413  ,  588.6728 ,  689.4741 ,  633.90454],
        [7502.0303 , 5343.9834 ,  563.9284 ,  690.71356,  622.1614 ],
        [7610.072  , 5227.0176 ,  579.8522 ,  672.1343 ,  624.2945 ],
        ...,
        [5129.152  , 3147.917  ,  272.6081 ,  374.8297 ,  332.13535],
        [4943.952  , 3176.0396 ,  301.06866,  390.83847,  325.5044 ],
        [5508.1694 , 3418.333  ,  295.7247 ,  376.09515,  322.656  ]],

       [[8324.27   , 5803.675  ,  538.3639 ,  673.33435,  598.7233 ],
        [8609.587  , 5865.27   ,  522.56   ,  644.01355,  619.9537 ],
        [8638.347  , 5787.5786 ,  531.3132 ,  659.1144 ,  591.5215 ],
        ...,
...
        [3811.3555 , 1985.8969 ,  318.80933,  288.66934,  318.9553 ],
        [3880.6448 , 2146.9849 ,  216.54195,  299.55103,  377.7116 ],
        [4197.2637 , 2367.9148 ,  255.80391,  360.47253,  444.9939 ]],

       [[5934.3174 , 4275.513  ,  729.7753 ,  916.8984 ,  999.6828 ],
        [6192.6875 , 4144.6416 ,  653.86615,  805.297  ,  850.64716],
        [6697.9395 , 4189.3247 ,  737.42444,  809.34186, 1050.4926 ],
        ...,
        [3662.6667 , 2056.6392 ,  273.36102,  287.7209 ,  266.2291 ],
        [3729.201  , 2185.1753 ,  240.62947,  369.76355,  333.78735],
        [4228.927  , 2228.7437 ,  278.9088 ,  358.12225,  396.88416]],

       [[6082.5977 , 4183.329  ,  782.99896,  914.4199 ,  966.175  ],
        [6344.678  , 3610.2083 ,  769.0373 ,  906.5941 ,  961.9961 ],
        [6586.852  , 4194.138  ,  791.0451 ,  772.94763,  916.8184 ],
        ...,
        [3534.3728 , 1967.3213 ,  320.33307,  286.80234,  449.809  ],
        [3753.4165 , 2096.4358 ,  278.61523,  340.76562,  396.22098],
        [3862.314  , 2027.921  ,  258.25385,  369.7472 ,  472.15524]]],
      dtype=float32)
Coordinates:
  * nchan                              (nchan) <U8 '121.6nm' ... 'LBHlong'
    PIERCEPOINT_DAY_LATITUDE_AURORAL   (nCrossDayAur, time_auroral) float32 4...
    PIERCEPOINT_DAY_LONGITUDE_AURORAL  (nCrossDayAur, time_auroral) float32 2...
    PIERCEPOINT_DAY_SZA_AURORAL        (nCrossDayAur, time_auroral) float32 3...
  * time_auroral                       (time_auroral) datetime64[ns] 2005-06-...
  * nCrossDayAur                       (nCrossDayAur) int64 0 1 2 3 ... 10 11 12

Also, there should be no warnings raised when loading the data, as there is meta data for all variables and cleaning applied to these data tags/inst IDs.

Finally, unit tests should be run locally against https://github.com/pysat/pysat/pull/1131 to ensure appropriate behaviour with updated pysat handling.

Test Configuration:

Checklist:

If this is a release PR, replace the first item of the above checklist with the release checklist on the wiki: https://github.com/pysat/pysat/wiki/Checklist-for-Release

jklenzing commented 1 year ago

What's the status of this pull? Is it waiting on a new pysat release?

aburrell commented 1 year ago

This requires the pysat pull request: https://github.com/pysat/pysat/pull/1131 to work.

Yes, it requires a pysat release.

jklenzing commented 11 months ago

Testing this branch against the pysat 3.2.0 RC, I see the following failures (for guvi):

Full error output at https://github.com/pysat/pysatNASA/actions/runs/7051616139/job/19194923688 Some of this has been documented in #218

jklenzing commented 11 months ago

@aburrell, I think to finish this one we just need to fix the test_load_with_pad test (see above comment). The other guvi error can be handled separately.

aburrell commented 8 months ago

@jklenzing I fixed both GUVI errors. The first one was a data-availability issue. I picked test days with better coverage and everything looks good.

jklenzing commented 8 months ago

TODO:

jklenzing commented 8 months ago

Running vs the RC, there is a segmentation fault in the ubuntu branch: https://github.com/pysat/pysatNASA/actions/runs/8163504069

Reran the branch, happened again. The develop branch passes without issue.

jklenzing commented 8 months ago

Fixed the segmentation fault issue. Applying the skip test flags to the high-res data only allows the new concat routines to be tested, while avoiding memory issues on GA. https://github.com/pysat/pysatNASA/actions/runs/8189899989

jklenzing commented 8 months ago
From the tests: Run with Stmts Miss Cover
pysat 3.1.0 78 27 65%
pysat 3.2.0rc6 78 5 94%