Units missing in ICON files

[aburrell]

Describe the bug For October 2019 units are missing from the ICON IVM output files.

To Reproduce

import datetime as dt
import pysat

stime = dt.datetime(2019, 10, 22)
etime = dt.datetime(2021, 1, 1)

icon = pysat.Instrument('icon', 'ivm', inst_id='a')
icon.load(date=stime)
icon.meta['Ion_Velocity_Z']

Yields:

Out[15]: 
FieldNam                              Ion velocity along IVM-z
LablAxis                              Ion velocity along IVM-z
long                                                       NaN
Valid_Min                                               -750.0
ScaleTyp                                                Linear
Valid_Max                                                750.0
units                                                         
long_name                             Ion velocity along IVM-z
notes        Velocity is relative to co-rotation. Positive-...
desc                                  Ion velocity along IVM-z
value_min                                               -750.0
value_max                                                750.0
fill                                                       NaN
children                                                  None
Name: Ion_Velocity_Z, dtype: object

And

icon.load(date=etime)
icon.meta['Ion_Velocity_Z']

Yields

FieldNam                              Ion velocity along IVM-z
Valid_Min                                               -750.0
LablAxis                              Ion velocity along IVM-z
ScaleTyp                                                Linear
Valid_Max                                                750.0
units                                                      m/s
long_name                             Ion velocity along IVM-z
notes        Cross-track velocity is relative to co-rotatio...
desc                                  Ion velocity along IVM-z
value_min                                               -750.0
value_max                                                750.0
fill                                                       NaN
long                                                       NaN
children                                                  None
Name: Ion_Velocity_Z, dtype: object

Expected behavior The units should be loaded for the files. If expected metadata is missing, a warning should be raised.

Desktop (please complete the following information):

• OS: OS X Big Sur
• Version 3.9
• Other details about your setup that could be relevant: develop branch of pysatNASA

Additional context Add any other context about the problem here.

[jklenzing]

This is a data issue. Oct 2019 has still only released v2. Later reprocessing (v6) starts on 11/16/2019. Not sure if these will be reprocessed or if these dates should just be dropped. @JonathonMSmith ?

[jklenzing]

https://cdaweb.gsfc.nasa.gov/pub/data/icon/l2/l2-7_ivm-a/2019/

[JonathonMSmith]

According to the ICON team, dates before day 320 of 2019 should not be used. This is because the satellite was still in checkout mode. Because of this, they have not, and will not be updating files from before that date. It is not intended for these files to be available on any public server. The current plan is to get them removed from SPDF and Berkeley where they exist currently. Not sure how long that will take.

[aburrell]

We could add a date check to the ICON clean functions. This way if anyone gets their hands on early ICON data they'd only be able to use it in a non-clean mode.

[jklenzing]

What about a version check?

[JonathonMSmith]

Well, I think the date range is a little more explicit for the cleaning routine because that's the decision from the ICON SOC as to where the good data starts. I do have an update that the data referred to in this issue should be taken down within a week.

[jklenzing]

Verified that the data is no longer on CDAWeb.

[jklenzing]

@aburrell, do we still want to put a date limiter in place? The old files are no longer on CDAWeb, so the question is how should we handle this for people who have already downloaded the data.

[aburrell]

I am ok not having a date limiter, I think it will be best for us to set the precedent that pysat is not going to keep a list of 'do-not-use' files. Instead we'll provide general data cleaning.

However, is there some way to warn the user that the file they've loaded doesn't have the expected format?

[jklenzing]

That's probably a good issue to add to the core code. If we run the file list, we can extract the latest version number, then warn if a file uses an older one. Will write up an issue there.

[aburrell]

That's probably a good issue to add to the core code

Yeah... but is there a general way to do that? Especially if the users haven't downloaded files recently? Happy to move the discussion to pysat, though.

[jklenzing]

It's tricky, and may be more effort than benefit. For ongoing missions (ICON, etc), presumably users will be downloading new data as it's released, so the version number would increase with time. This could be a good prompt that older data has been reprocessed. I don't think it will catch every case, but could be helpful for cases like this.

New issue at: https://github.com/pysat/pysat/issues/1049