OMNI (not OMNI2) Data Not Up to Date

[julia-claxton]

Hello! I have noticed that my OMNI data only goes through January 2022, no matter how often I update it. My OMNI2 data is up to date, however using OMNI2 data in calculation of L-shell (my use case for the OMNI data) results in an error saying that the OMNI2 data does not have Kp index. MWE is given below.

Minimal example to reproduce issue:

import spacepy.toolbox, spacepy.omni

spacepy.toolbox.update()
print("OMNI Data Range: ", end = "")
print(spacepy.omni.omnirange())

import datetime
import spacepy.coordinates, spacepy.irbempy

time = spacepy.time.Ticktock(datetime.datetime(2022, 5, 1, 0, 0, 0), 'UTC') # Arbitrarily chosen
position = spacepy.coordinates.Coords([2, 2, 2], 'GEI', 'car', units = ['Re', 'Re', 'Re'], use_irbem = False) # Arbitrarily chosen
spacepy.irbempy.get_Lm(time, position, [90]) # This line throws the 'data out of range' error

Error message/Traceback:

OMNI Data Range: (datetime.datetime(1963, 11, 27, 14, 0), datetime.datetime(2022, 1, 19, 23, 0))

Traceback (most recent call last):
  File "/Users/<username>/Downloads/mwe.py", line 12, in <module>
    spacepy.irbempy.get_Lm(time, position, [90])
  File "/Users/<username>/Library/Python/3.9/lib/python/site-packages/spacepy/irbempy/irbempy.py", line 1467, in get_Lm
    results = get_Lstar(ticks, loci, alpha, extMag=extMag, options=opts, omnivals=omnivals)
  File "/Users/<username>/Library/Python/3.9/lib/python/site-packages/spacepy/irbempy/irbempy.py", line 1663, in get_Lstar
    DALL = _get_Lstar(ticks, loci, alpha, extMag, options, omnivals, landi2lstar)
  File "/Users/<username>/Library/Python/3.9/lib/python/site-packages/spacepy/irbempy/irbempy.py", line 1367, in _get_Lstar
    d = prep_irbem(ticks, loci, alpha, extMag, options, omnivals)
  File "/Users/<username>/Library/Python/3.9/lib/python/site-packages/spacepy/irbempy/irbempy.py", line 1722, in prep_irbem
    omnivals = omni.get_omni(ticks)
  File "/Users/<username>/Library/Python/3.9/lib/python/site-packages/spacepy/omni.py", line 257, in get_omni
    raise ValueError('Requested dates are outside data range')
ValueError: Requested dates are outside data range

OS, Python version, and dependency version information:

macOS-13.4.1-arm64-arm-64bit
sys.version_info(major=3, minor=9, micro=18, releaselevel='final', serial=0)
numpy=1.26.1
scipy=1.10.1
matplotlib=3.8.0

I am using a MacBook Pro with an M1 (Apple silicon) processor.

Version of SpacePy

spacepy=0.4.1

Downloaded via pip and updated prior to running MWE script. I am running spacepy with Python 3.9 because that is the only way I could get spacepy to run properly on my machine.

[candeynasa]

VIRBO.org is no longer getting updates. OMNI data are retrievable from https://spdf.gsfc.nasa.gov/pub/data/omni/omni_cdaweb/ (CDF files) or via the CDAS and HAPI APIs from CDAWeb.

[julia-claxton]

I think it would be smart to have spacepy pull from that website if VIRBO is now out-of-date, since it will just get more out-of-date the more time goes on.

Either way, I'm not sure how to get spacepy to work with the provided CDFs for my purposes. The directory that L-shell calculations pull OMNI data from (/Users/<username>/.spacepy/data/qindenton_daily_files) are all .txt files on my machine, so .cdf won't work there. I have tried getting spacepy.irbempy.get_Lm() to use a more up-to-date database (e.g. OMNI2hourly), but that throws KeyErrors since the data is formatted differently than the default QDhourly. I tried reformatting the data myself but that didn't work either. Is there a better way to calculate L-shell past Jan. 2022?

[drsteve]

The "OMNI2" is taken directly from NASA's OMNI2 data set (the 2 distinguishes it from the ~70s era OMNI product). The "OMNI" data set used by SpacePy is a derived product that's more widely known - at least these days - as Qin-Denton data. Hence the QD in the QDhourly name.

As noted previously ViRBO were generating and serving that. After ViRBO ceased to generate the data, LANL picked up the processing and intermittently passed updates back to ViRBO to serve. At some point the processing and provision was passed to the Van Allen Probes ECT suite since they had support for a website and data processing, as well as a need for the data product. So the current update code pulls the older data from ViRBO and then pulls the hourly-resolution (text) files from the ECT website to build a combined data set.

The issue is that Van Allen Probes ended and post-mission support has effective ended too. So there's currently no current public-facing generation of the Qin-Denton data products past January 2022. They are used by the NASA MMS mission in generation of the MEC data product (at higher resolution, but we generate 1, 5 and 60 minute files), but the underlying files aren't served. We're currently looking for a more permanent home for the ongoing data processing to generate that data product as well as to serve it. SpacePy doesn't have funding to support ongoing processing and data hosting.

As a temporary fix I can grab the post-2022 files and find a spot to make them available. It'll probably end up being a dump on Zenodo or something similar. I'll update here once I have the best path for that figured out.

[candeynasa]

The QD data can be archived at SPDF; adding to CDAWeb would also need a version in CDF form with ISTP metadata. If only the OMNI values are needed, the ASCII files for OMNI are in the high and low res directories under https://spdf.gsfc.nasa.gov/pub/data/omni/ . I mentioned the OMNI CDFs since they are generally easier to load (such as with pycdf https://spacepy.github.io/pycdf.html ) or even easier direct from CDAWeb using its API via cdasws https://cdaweb.gsfc.nasa.gov/WebServices/REST/ .

[julia-claxton]

Hi, I'm sorry if this is a silly question. I am unsure how to download the OMNI data from the websites provided in such a way that spacepy can use them for L-shell calculations. The files don't appear to be formatted in the same way as the QDHourly data that spacepy uses, which causes spacepy to throw KeyErrors when it looks at the provided data that is formatted differently than the Qin-Denton data. I find myself still unable to use irbempy.get_Lm() for any time after Jan 2022 to get L-shell from position data.

What would be the best way going about getting L-shell from position & time after Jan 2022? If using the differently-formatted OMNI data, should I reformat it myself or is there a setting in spacepy I can use?

[jtniehof]

@julia-claxton, right now the best option is unfortunately to wait until Steve can get things squared away. The original OMNI CDF files at CDAWeb/SPDF do not have some of the derived quantities which are in the QD files, which is why we use the QD. The OMNI2 data, in addition to having a different set of names than OMNI/QD, also don't have those derived quantities, so effectively they can't be used as inputs to the IRBEM routines. See #617.

@candeynasa , if you're offering to host the QD files at SPDF, and hopefully even figure a means of routine production, we should talk. We need a low-friction way to keep these things up to date and it's been hampered by being on a volunteer basis.

[jtniehof]

@drsteve, @candeynasa, and others at SPDF have been working on hosting. In the meantime, Steve generated up-to-date files and I've processed them into the final parsed SpacePy database. That can be downloaded here. Place that .h5 file in the data directory inside the Spacepy DOT_FLN directory.

[julia-claxton]

That worked perfectly, thank you so much @jtniehof!