search

SeisComP3 / seiscomp3

SeisComP is a seismological software for data acquisition, processing, distribution and interactive analysis.
Other
111 stars 87 forks source link

fdsnxml2inv - restricted value not correctly reported #81

Closed vlauciani closed 7 years ago

vlauciani commented 7 years ago

Hi all

We are using fdsnxml2inv script to convert StationXML to Inventory.

I noticed that converting a StationXML with Station and Channels restricted, the output Inventory is not correctly generated.

For example, the StationXML for our TV_SF01 station ( http://webservices.ingv.it/fdsnws/station/1/query?network=TV&station=SF01&level=response&format=xml&formatted=true&nodata=204&visibility=true&includerestricted=true ) has:

 <Network code="TV" startDate="2010-01-01T00:00:00"
restrictedStatus="partial">
   <Station code="SF01" startDate="2016-03-12T00:00:00"
restrictedStatus="closed">
     <Channel code="EHE" startDate="2009-07-07T00:00:00"
restrictedStatus="closed" locationCode="">
     <Channel code="EHN" startDate="2009-07-07T00:00:00"
restrictedStatus="closed" locationCode="">
     <Channel code="EHZ" startDate="2009-07-07T00:00:00"
restrictedStatus="closed" locationCode=“">

converting it in the Inventory, the output is: network) true station) false channel) true channel) true channel) true

but we expect: network) false station) true channel) true channel) true channel) true

Another example is for Z3_A280A station; the StationXML ( http://webservices.ingv.it/fdsnws/station/1/query?network=Z3&station=A280A&level=response&format=xml&formatted=true&nodata=204&visibility=true&includerestricted=true ) return:

 <Network code="Z3" startDate="2015-01-01T00:00:00"
restrictedStatus="closed">
   <Station code="A280A" startDate="2015-09-15T16:50:00"
restrictedStatus="closed">
     <Channel code="HHE" startDate="2015-09-15T16:50:00"
restrictedStatus="closed" locationCode="">
     <Channel code="HHN" startDate="2015-09-15T16:50:00"
restrictedStatus="closed" locationCode="">
     <Channel code="HHZ" startDate="2015-09-15T16:50:00"
restrictedStatus="closed" locationCode=“">

converting it in the Inventory, the output is: network) true station) false channel) true channel) true network) true

but we expect: network) true station) true channel) true channel) true channel) true

Is there a bug or is this correct?

gempa-jabe commented 7 years ago

I think your observation is correct and it is a bug. The station restricted flag is always set to false. In general we are setting restricted to true if the restricted status (FDSNXML) is anything else than open. As I understand you, you would map partial to false, right? The conversion is ambiguous anyway: 3 states vs 2 states. I committed a change regarding the station level bug.

vlauciani commented 7 years ago

Hi Jan

Yes, I think that partial should be false. The StationXML approach, if I understand correctly, is that partial is used to a not omogeneous situation. For exampe to identify a Network that has one or more station restricted and one or more station not restricted.

partial is also used when, for a station, one channel is different then the others (HHZ=restricted and HH[EN]=not restricted).

Thank you for your support, Valentino

gempa-jabe commented 7 years ago

Yes, partial means that an entity is partially open or partially closed. In SC3 XML we only have a flag for that, either open or closed. I have to check with current implementations (fdsnws, Arclink) how this flag is evaluated. When restricted=true means that the tree traversal is stopped at this node then the current approach is wrong and you are right. @andres-h, @gempa-stephan: can you comment?

andres-h commented 7 years ago

Tree traversal is not stopped, it's only channel restriction that has an effect on access in Arclink. I just double-checked fdsnws and this is also the case in fdsnws after the latest patch (should be mentioned in the changelog).

gempa-jabe commented 7 years ago

Thanks. What do you say to the proposed change? Would it be valid? I personally don't mind and since the subject is rather philosophical I am also open for a change. @vlauciani how would a change improve your situation?

petrrr commented 7 years ago

The purpose of these values is not completely clear. To me this looks that the inventory XML actually defines higher level values for restrictedStatus, which however are actually not used in the services and the aggregation is done on he fly.

If this is true, that this attributes can have arbitrary values. But then it would be clearer if these values were not present at all, to avoid confusion.

If the higher level definition is intended to be a shortcut to be avoid to have to specify restrictions on the lower level, but this short cut is not used (because you are using a explicit granular definition) these values have to be set to "false", independently of the aggregated values.

If these have in some way represent the aggregated values as in station XML, you inevitable have to deal with the mixed case, i.e. represent three states.

gempa-jabe commented 7 years ago

I think the purpose is actually to define what you call a shortcut. If a station is restricted but a channel does not set this attribute then it is inherited from the station. But that is not the point here. Actually those values have nothing to do with inventory at all, neither in Station XML nor SC3 XML. They are dependent on specific needs and implementations, e.g. EIDA node synchronization. But that is another topic to discuss.

Here we talk about how to map 3 states to 2 states. In my opinion there is no right or wrong, just a matter of personal preference. If there are good reasons to change it, I am fine with it.

petrrr commented 7 years ago

@gempa-jabe Agreed, you just cannot squeeze 3 states into 2 ;-) That is what I actually tried to transmit.

What I am saying is that here the SC3 XML is probably used to setup/config some services notably FDSN services and Arclink (or as you say to exchange this config). To avoid ambiguities the definition should be do only on one single level to avoid ambiguities.

Let's look at the examples above, that should be translated

TV_SF01 compacted version:

or the more verbose version, always on lowest level without any logic

Z3_A280A, compact:

or the explicit version:

andres-h commented 7 years ago

Currently only channel restriction is used for access control. Network and station restrictions are used in WebDC for visualization. There are no "shortcuts". I tend to think that partial restriction is still a restriction. I suppose for security it would be better to leave items restricted when in doubt.

andres-h commented 7 years ago

Tree traversal is not stopped, it's only channel restriction that has an effect on access in Arclink. I just double-checked fdsnws and this is also the case in fdsnws after the latest patch [1](should be mentioned in the changelog).

[1] https://github.com/SeisComP3/seiscomp3/commit/9641a3ee5d013c541439019a9a8c0ec6eced052e#diff-3ebbc7b7c593064b39b664a8066ec9f0L335

vlauciani commented 7 years ago

@andres-h on which SC3 release is it implemented? We are running the 2016.062.

gempa-jabe commented 7 years ago

This was added with PR #79. So it is not yet part of a release.

gempa-jabe commented 7 years ago

Is that issue resolved with the fixes of fdsnxml2inv or do we have to discuss about the restricted flag in more detail?

vlauciani commented 7 years ago

Could we apply the patch [1] mentioned from Andres on our 2016.161 release? If yes, how?

gempa-jabe commented 7 years ago

You can create your own diff with all the latest fdsnws changes. Just checkout master and run:

git diff origin/build/jakarta/2016.161 -- src/trunk/apps/fdsnws > fdsnws.patch

Then apply it to your release. To show all commits between the last release and master, run:

git log origin/build/jakarta/2016.161..master -- src/trunk/apps/fdsnws

Cherry pick the commits you are interested in.

petrrr commented 7 years ago

It seems we hit this bug, or what ever it is again.

Can we reopen this issue?

gempa-jabe commented 7 years ago

Can you please be more specific on what you expected and what you got? Since there was a fix mentioned we are not aware of an issue and need more information.

petrrr commented 7 years ago

I think what actually happened here can be summarized as follow:

Now however (apparently after the update 2016.333) the behavior of SC3 seem to have changed wrt. to this flag. If the restriction is set to true, the network as a whole is now marked as restricted in EIDA/Arclink. But this sort of behavior then not consistent with how fdsnxml2inv converts partial.

Concretely look at our network TV, where only few station are set restricted:

Most channels (and stations) have are set restrictedStatus="open", few set to restrictedStatus="closed". As a consequence the restriction status of the network restrictedStatus="partial".

Now if fdsnxml2inv is used this translates to the following translation:

network, restricted=true station, restricted=false stream, restricted=false (for most stations)

    <network publicID="NET/TV/20170329115310.761566.13561" code="TV">
      <start>2008-01-01T00:00:00.0000Z</start>
      <description>INGV Experiments Network</description>
      <netClass>t</netClass>
      <archive>INGV</archive>
      <restricted>true</restricted>
      <shared>true</shared>
      <station publicID="STA/TV/A001/20170329115310.761834.13562" code="A001">
[...]
        <restricted>false</restricted>
        <shared>true</shared>
        <sensorLocation publicID="LOC/TV/A001//20170329115310.762004.13563" code="">
[...]
          <stream code="EHE" datalogger="Datalogger#20170329115310.76235.5" sensor="Sensor#20170329115310.764295.15">
[...]
            <restricted>false</restricted>
          </stream>
[...]

Only few stations like AG11:

network, restricted=true station, restricted=true stream, restricted=true (for most stations)

However, the in Arclink the whole network is restricted and this is incorrect. See for example here: http://eida.gfz-potsdam.de/eida/status/inventory_comparison2.php

Instead the network should not be restricted, only some of the stations should.

gempa-jabe commented 7 years ago

As Arclink is the reference and I don't use Arclink nor am I involved in EIDA deployments, I suggest that you propose a fix to the issue together with your EIDA partners, e.g. @andres-h. Thank you.

petrrr commented 7 years ago

@andres-h: Could you have a look at this?

I believe in the current situation the use/intention/semantics of the restriction flags in not well-defined and as a consequence tools/components of SC3 behave inconsistently.