ENH: General Madrigal instrument

[aburrell]

Description

Partially addresses #1 by adding general support for pandas-supportable madrigal data sets.

Type of change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality or documentation)
• This change requires a documentation update

How Has This Been Tested?

Added new unit tests and examples:

import pysat
import pysatMadrigal as py_mad

# Download DMSP data from Madrigal
dmsp_abi = pysat.Instrument(inst_module=py_mad.instruments.madrigal_pandas,
                            tag='180', kindat='17110')
dmsp_abi.download(dt.datetime(2015, 12, 30), dt.datetime(2015, 12, 31),
                  user='Firstname+Lastname', password='email@address.com')
dmsp_abi.load(date=dt.datetime(2015, 12, 30))

Test Configuration

• Operating system: OS X Big Sur
• Version number: Python 3.8 and 3.9
• Any details about your local setup that are relevant: develop branch of pysat

Checklist:

• [x] Make sure you are merging into the develop (not main) branch
• [x] My code follows the style guidelines of this project
• [x] I have performed a self-review of my own code
• [x] I have linted the files updated in this pull request
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] I have made corresponding changes to the documentation
• [x] My changes generate no new warnings
• [x] I have added tests that prove my fix is effective or that my feature works
• [x] New and existing unit tests pass locally with my changes
• [x] Any dependent changes have been merged and published in downstream modules
• [x] Add a note to CHANGELOG.md, summarizing the changes

If this is a release PR, replace the first item of the above checklist with the release checklist on the pysat wiki: https://github.com/pysat/pysat/wiki/Checklist-for-Release

[rstoneback]

Data does load for Jan 1,

In [5]: dmsp_abi.load(2015, 1)
pysat WARNING: The generalized Madrigal data Instrument can't support instrument-specific cleaning.
<ipython-input-5-065c33fa990e>:1: DeprecationWarning: Meta now contains a class for global metadata (MetaHeader). Default attachment of global attributes to Instrument will be Deprecated in pysat 3.2.0+. Set `use_header=True` to remove this warning.
  dmsp_abi.load(2015, 1)

In [6]: dmsp_abi.data
Out[6]: 
                     year  month  day  hour  min  sec  recno  kindat  kinst      ut1_unix      ut2_unix  epowf        mlt  gdlat   glon  cgm_lat  cgm_long  eqb_qc_fl  eqb_prv_kp  eqb_prb_kp  sat_id   epowq
2015-01-01 00:00:53  2015      1    1     0    0   53      0   17110    180  1.420070e+09  1.420070e+09   62.9   4.070833  -50.7   80.4    -62.9     135.0        2.0        2.24       0.907      17  2171.0
2015-01-01 00:13:05  2015      1    1     0   13    5      1   17110    180  1.420071e+09  1.420071e+09   64.1  20.991389  -80.2  -27.0    -67.9      26.2        1.0        1.58       0.972      17  2171.0
2015-01-01 00:14:41  2015      1    1     0   14   41      2   17110    180  1.420071e+09  1.420071e+09   63.6   6.001944  -51.2  100.3    -65.8     161.0        2.0        1.87       0.928      18  2181.0
2015-01-01 00:27:21  2015      1    1     0   27   21      3   17110    180  1.420072e+09  1.420072e+09   64.6  21.463333  -79.2  -18.2    -67.6      30.3        1.0        1.28       0.992      18  2181.0
2015-01-01 00:35:49  2015      1    1     0   35   49      4   17110    180  1.420073e+09  1.420073e+09   63.0   4.343611  -51.3   78.2    -63.0     131.6        2.0        2.21       0.907      19  2191.0
.

[aburrell]

Long list of filename formats you added here!

Was super not happy to need to do that, but couldn't find another solution.

[rstoneback]

Long list of filename formats you added here!

Was super not happy to need to do that, but couldn't find another solution.

Makes sense. Thanks for the bummer repetitive work!

[aburrell]

It looks like something is not working right in the GitHub Actions user/password assignment, but I can't see anything that would make it go wrong 🐛

[rstoneback]

It looks like something is not working right in the GitHub Actions user/password assignment, but I can't see anything that would make it go wrong 🐛

Looks like the download flag is only set for a single tag # for the madrigal_pandas instrument. Other supported tags are thus marked as no_download. The no_download test in cls_instrument_library is looking for some kind of no download warning, but it doesn't pass through and user or password information when it calls download. The madrigal download has a check for user and password, leading to the error that is raised.

[aburrell]

🦥

Looks like the download flag is only set for a single tag # for the madrigal_pandas instrument.

Thanks for the whole explanation. Not sure how to set all the tags, but I'll figure it out.

[rstoneback]

[aburrell]

These tests are passing locally, but require the file format parsing changes available on the 'develop' branch of pysat. Because of this, I am adding a version cap for pysat and putting this merge on hold until the next pysat release.

[aburrell]

@rstoneback pinging for a re-review. Looks like the tests are probably going to pass given how long they've taken.

[rstoneback]

xarray released a new version https://twitter.com/xarray_dev/status/1550522574672523266 , on Friday, which breaks pysat tests and functionality.

[rstoneback]

The madrigal_pandas download isn't working. Getting an internal server error. Rerunning the tests didn't help.

[aburrell]

Hmm... was able to do a download in ipython. I'm going to try re-running just one job and see if it's perhaps the parallel requests that's an issue.

[aburrell]

Well, that didn't work. @rstoneback , what happens if you run the test suite locally?

[rstoneback]

Getting at least 1 failure locally so far. inst_dict26. When tests finish I'll decode that.

[rstoneback]

In total there were 29 test failures locally. madrigal_pandas was the Instrument with the download error. Some 500 Internal Server Errors and a variety of error message check issues.

[rstoneback]

In total there were 29 test failures locally. madrigal_pandas was the Instrument with the download error. Some 500 Internal Server Errors and a variety of error message check issues.

I unfortunately did not run these tests in my normal Python environment. Rerunning in the correct one.

[rstoneback]

In the correct environment I get only 2 errors, 1 for download and 1 for the remote_file_list, both for madrigal_pandas or inst_dict26.

[aburrell]

Can you see what the tag is?

[rstoneback]

Can you see what the tag is?

8105