search

cedardevs / feedback

An issue-only repository for centralizing all external feedback
0 stars 0 forks source link

Support "YYYY" and "YYYY-MM" dates #12

Closed chris-noaa-affiliate closed 4 years ago

chris-noaa-affiliate commented 4 years ago

Describe the solution you'd like Add support for "YYYY" and "YYYY-MM" dates. Of the 25,000 non-NCEI records, 2,000 records have dates of this format. Currently, these records fail to be indexed.

Example of "2015-12" at line 146 https://data.noaa.gov/waf/NOAA/nmfs/afsc/iso/xml/12672.xml

ajakz commented 4 years ago

@chris-noaa-affiliate The dates being parsed into OneStop for search are not located in the area where you've indicated date parsing failure. Where we are extracting the date information, however, the dates are:

<gmd:EX_TemporalExtent>
  <gmd:extent>
    <gml:TimePeriod gml:id="boundingTemporalExtent-1-1">
      <gml:description> | Currentness: publication</gml:description>
      <gml:beginPosition>1999</gml:beginPosition>
      <gml:endPosition>2015</gml:endPosition>
    </gml:TimePeriod>
  </gmd:extent>
</gmd:EX_TemporalExtent>

YYYY dates are already accepted into OneStop when encountered in the expected xpaths.

We can, however, add support for YYYY-MM dates -- however, in order to support these dates in Elasticsearch, we do add assumed values for missing fields. That is, if a start/begin value is just a year, we assume YYYY-01-01T00:00:00Z and if the end value is just a year, we assume YYYY-12-31T23:59:59Z. This is exclusively for the search index, and we do not overwrite what is actually given to us in the Inventory Manager side of OSIM.

In order to continue facilitating search, we will need to fill in the blanks for YYYY-MM dates, too, and will follow the same convention: a start value is at the first second of the month, an end value is at the final second of the month.

If that all sounds like what you're looking for, I can go ahead and put an issue on our backlog to parse YYYY-MM dates, too.

chris-noaa-affiliate commented 4 years ago

Yes, that's what I'm looking for. Please add it to your backlog.

ajakz commented 4 years ago

This was added into OSIM!