Cannot handle StationXML documents with undefined (open) end dates

[chad-earthscope]

From @ivor52:

so my next test was on an XML format file (previous was old RESP format as test #1) and it seems that the new XML formats are not providing and endDate when the epoch in question is the current one (and so there is no endDate).

RESP formtat just made a fake end date far into the future, but it seems XML format files are not following this convention.

except that evalresp is now throwing the error:

Sun Sep 9 10:07:35 2018 [ERROR] Could not find endDate in Channel

and i didn't find anything in the manual that discusses this, so i really don't know what the expected / required convention is: are the XML file generators doing this wrong? or should evalresp be made more intelligent to understand that the current epoch doesn't really need an endDate?

attached is the XML file i'm testing with

TITA1.HGZ.gz

[ivor52]

i had a look at the code. the line that ends up throwing the error is:

libsrc/evalresp/stationxml2resp/xml_to_dom.c::1141

i assume, however, that an open endDate (i.e., none specified) is only valid for the last and current epoch. but i don't know the program and data structure to know how to distinguish between the last epoch and all other epochs coming before (where an endDate is required).

so i won't be able to solve this myself, i'm afraid... :(

[andrewcooke-isti]

so it seems to me this is an inconsistency between the two formats? given the above, what i intend to do is:

• allow missing end dates when reading XML, leaving NULL in the appropriate data structures
• add a post-read step which scans the read structure for these missing values and:
  • if it is the last epoch, insert an arbitrary date in the future, as with SEED (eg 2999-01-01).
  • otherwise, raise an error

after this the two uses made of this data should work as expected:

• output to SEED format will include the arbitrary date, in line with SEED customs
• evaluating in evalresp will give results identical to SEED with the arbitrary date, in line with current behaviour,

please say if this seems wrong in some way.

[ivor52]

emulating XML behaviour to match current convention seems fine to me.

richard

On Tue, Oct 16, 2018 at 2:22 PM Andrew Cooke (ISTI projects) < notifications@github.com> wrote:

so it seems to me this is an inconsistency between the two formats? given the above, what i intende to do is:

-

allow missing end dates when reading XML, leaving NULL in the appropriate data structures

add a post-read step which scans the read structure for these missing values and:

  if it is the last epoch, insert an arbitrary date in the future, as
  with SEED (eg 2999-01-01).
  -

  otherwise, raise an error

after this the two uses made of this data should work as expected:

-

output to SEED format will include the arbitrary date, in line with SEED customs

evaluating in evalresp will give results identical to SEED with the arbitrary date, in line with current behaviour,

please say if this seems wrong in some way.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/iris-edu/evalresp/issues/69#issuecomment-430366017, or mute the thread https://github.com/notifications/unsubscribe-auth/AY-P6cZuX2-SVM35UmAsEM4KB3xAVJSeks5uljH-gaJpZM4Who_G .

[chad-earthscope]

@andrewcooke-isti I believe this approach is good.

[chad-earthscope]

@andrewcooke-isti Actually, let me back-track. I'm not comfortable inserting made-up data, if it's something that may be used elsewhere. Inside of a program, as an internal kludge, it would be OK.

Let me dig into precedent for how this should be done in SEED.

Note: the DMC has been "filling in" open end times for a long time due to some early database limitation, this was not the right approach in my opinion but has led others to follow this example.

[chad-earthscope]

so it seems to me this is an inconsistency between the two formats?

I do not believe so, both formats allow an open end time simply by omitting an end time and such usage is quite common. In SEED metadata headers times are represented as variable length strings (terminated with a ~), an open end time is a 0-length string.

This is usually expressed in RESP format as:

B052F22     Start date:  2008,123,00:00:00.0000
B052F23     End date:    No Ending Time

An example RESP file with just such an open end time is attached. evalresp works with these files just fine (3.x and 5.x versions), so an open end time is being represented internally somehow. Even if it is a far-future date it is not being exposed to the user, e.g. the verbose output still says "No Ending Time".

I would like reading of StationXML to do likewise, handle open end times and report them to the user accordingly.

RESP.GE.APE..BHZ.gz

[andrewcooke-isti]

ok, sure (i was just going by the copy+paste you posted at the top of this issue).

[ivor52]

does this mod update the version number? it seems it has not been upped in public.h

[andrewcooke-isti]

no, no i haven't done anything like that. version numbers would be associated with releases i imagine.

if you want to know if you have the latest code use the commit hash from git.

[chad-earthscope]

I'm going to do a few more tests and then I'll cut another release candidate.

@ivor52 If you are willing to pull from the current master branch and test from that it'd be very helpful. In commit d506182ee080dd5b03605abaa95b7a8df1b0b028 I've added the ability to auto-detect StationXML when options.station_xml == -1, but I'm not sure if it works in all calling paths of the library.

[ivor52]

@chad-iris following tests all completed successfully:

• auto-detect of XML format
• auto-detect of RESP format
• XML format last epoch not containing an endDate

from my perspective this bug is fixed.

[ivor52]

forgot to add that my path is the library, calling routine evalresp_channel_to_response()

[chad-earthscope]

@ivor52 Awesome, thanks!