search

EIDA / userfeedback

This repository is meant to collect feedback from EIDA users by means of its Issue Tracker
11 stars 5 forks source link

[No Data] Z3.A020B - access denied with valid token/password #5

Closed flofux closed 5 years ago

flofux commented 5 years ago

Client: ObsPy 1.1.1 Python 2.7.12 Station: Z3.A020B Valid EIDA token? Yes, works with all (?) other Z3 data Valid Arclink password? Yes, works with all other Z3 data Last time tested: April 25, 2019, 16:30 CEST

Waveform data from station Z3.A020B (hosted at ODC node) is not accessible with my token or my Arclink password, both of which work fine for any other Z3 data. The WFCatalog also gives no metrics for this station. This code shows the issue. The behaviour is the same for any day. The data should be available at ODC:

from obspy.clients.fdsn import Client as fdsnClient
from obspy.clients.fdsn import RoutingClient
from obspy.clients.arclink import Client as ArcClient
from obspy import UTCDateTime

arclink = ArcClient('thepassword')
directfdsn = fdsnClient('ODC', eida_token='/home/fuchs/.eidatoken')
routedfdsn = RoutingClient('eida-routing', credentials={'EIDA_TOKEN': '/home/fuchs/.eidatoken'})

Using direct Client:

directfdsn.get_waveforms(network='Z3', station='A020B', channel='HH*', location='*', starttime=UTCDateTime('2018-03-01'), endtime=UTCDateTime('2018-03-02'))

results in

FDSNException: Authentication failed

The debug=True output states Error 403: Forbidden, access denied

Using RoutingClient

routedfdsn.get_waveforms(network='Z3', station='A020B', channel='HH*', location='*', starttime=UTCDateTime('2018-03-01'), endtime=UTCDateTime('2018-03-02'))

results in

FDSNNoDataException: No data available for request. Detailed response of server: No Content --

Notably, the debug=True output does not indicate any authentication issues for the routed client.

Using Arclink:

arclink.get_waveforms(network='Z3', station='A020B', channel='HH*', location='*', starttime=UTCDateTime('2018-03-01'), endtime=UTCDateTime('2018-03-02'))

results in

ArcLinkException: DENIED access for user THEPASSWORD

A maybe connected issue is, that A020B metadata is also not accessible via the ObsPy RoutingClient (direct fdsn client works): 

routedfdsn.get_stations(network='Z3', station='A020B', channel='HH*', location='*', starttime=UTCDateTime('2017-01-01'), endtime=UTCDateTime('2019-02-01'))

results in

FDSNNoDataException: No data available for request. Detailed response of server: No Content --

javiquinte commented 5 years ago

The problem with the Routing Client is that the routes at ODC are missing the A020B station. Could you fix it in your local configuration, @Jollyfant ? http://www.orfeus-eu.org/eidaws/routing/1/localconfig

The problem with the direct FDSN access is that at that level the token is not usable. The token can be used in the context of EIDA (like in the previous example). At the FDSN level you should use a username/password for each data centre and do individual requests to all of them.

Regarding the Arclink client I don't see where you pass the username to the server. I understand that the password is important to decrypt the data, but it is not used at the authorization stage.

Jollyfant commented 5 years ago

A020B was added to the routing configuration: http://www.orfeus-eu.org/eidaws/routing/1/query?station=A020B. Metrics should be available once this change has propagated.

flofux commented 5 years ago

The problem with the Routing Client is that the routes at ODC are missing the A020B station. Could you fix it in your local configuration, @Jollyfant ? http://www.orfeus-eu.org/eidaws/routing/1/localconfig

The problem with the direct FDSN access is that at that level the token is not usable. The token can be used in the context of EIDA (like in the previous example). At the FDSN level you should use a username/password for each data centre and do individual requests to all of them.

Regarding the Arclink client I don't see where you pass the username to the server. I understand that the password is important to decrypt the data, but it is not used at the authorization stage.

For the ObsPy Arclink client, the username (which is the password) is set when the client is initiated arclink = ArcClient('thepassword') and then automatically transmitted during the request. It works fine for any station but A020B.

Being naive, I also tested the direct FDSNWS access with the token, as in ObsPy it seems possible. It does actually work perfectly fine (not for A020B, though):

from obspy import UTCDateTime
from obspy.clients.fdsn import Client
client = Client('ODC', eida_token='/path/to/token', debug=True)
client.get_waveforms(network='Z3', station='A001A', channel='HHZ', location='*', starttime=UTCDateTime('2018-03-04'), endtime=UTCDateTime('2018-03-05'))

Out[4]: 1 Trace(s) in Stream: Z3.A001A.00.HHZ | 2018-03-03T23:59:58.985000Z - 2018-03-04T23:59:59.995000Z | 100.0 Hz, 8640102 samples

If I understand the debug=True output correctly, it does create a temporary user/pw based on the token:

Downloaded https://www.orfeus-eu.org/fdsnws/dataselect/1/auth with HTTP code: 200 Got temporary user/pw: MUn7hP2O5wII9s64ErXxRgVk/nXOdtga-HSVOsjSz Installed new opener with handlers: [<urllib2.HTTPDigestAuthHandler instance at 0x7f7a23da5518>, <obspy.clients.fdsn.client.NoRedirectionHandler instance at 0x7f7a23da52d8>]

Let's see if things get fixed now that A020B was added to the routing configuration (routed and direct still not working at April 26, 2019, 10:32 CEST, but that was maybe too early after the change).

Jollyfant commented 5 years ago

For it to work for A020B we will have to restart FDSNWS. The station wasn't only missing from routing configuration but also from the FDSNWS configuration. So after the next maintenance window it should work..

javiquinte commented 5 years ago

I see your change already in your local configuration (Routing Service) and test it with the synchronisation at our GFZ Routing Service and works perfectly. Let's wait then for the propagation of the changes. As usual, thanks @Jollyfant !

flofux commented 5 years ago

@Jollyfant Please let me know when the fix should be in place, so I can test again. Thanks!

flofux commented 5 years ago

Still getting the access denied message with both Arclink and FDSNWS :-/

Jollyfant commented 5 years ago

I will check once I get back to KNMI.. possibly a configuration problem.

javiquinte commented 5 years ago

Hi @Jollyfant Before you leave us, is this still an issue? I took a random day (so that is easy to see for you in the SDS structure if there is data) and got this as an output. See the last line of this output ("did not receive data from...")

$ fdsnws_fetch -vvv -N Z3 -S "A02" -C "HHZ" -s "2018-03-01" -e "2018-03-02" -o data.mseed using token in /home/javier/.eidatoken: {"valid_until": "2019-06-22T12:09:29.611215Z", "cn": "Javier Quinteros", "memberof": "/epos/alparray;/epos;/", "sn": "Quinteros", "issued": "2019-05-23T12:09:29.611219Z", "mail": "javier@gfz-potsdam.de", "givenName": "Javier", "expiration": "1m"} getting routes from http://geofon.gfz-potsdam.de/eidaws/routing/1/query?network=Z3&station=A02%2A&channel=HHZ&starttime=2018-03-01&endtime=2018-03-02&format=post ... got 21933568 bytes (application/vnd.fdsn.mseed) from http://www.orfeus-eu.org/fdsnws/dataselect/1/queryauth got 66665472 bytes (application/vnd.fdsn.mseed) from http://erde.geophysik.uni-muenchen.de/fdsnws/dataselect/1/queryauth did not receive data from Z3.A020A..HHZ, Z3.A020B..HHZ, Z3.A024A..HHZ, Z3.A028A..HHZ, Z3.A028B..HHZ

Jollyfant commented 5 years ago

Hi, I don't have data for stations A020A, A024A, A028A, A028B in 2019. Data for A020B stopped at day 052.

flofux commented 5 years ago

Hi, I don't have data for stations A020A, A024A, A028A, A028B in 2019. Data for A020B stopped at day 052.

A020A and A024A were closed long time ago, so no data in 2019 is OK. A020B was temporarily removed on day 052 (and not reinstalled, yet), so that's also OK. A028A/B is operated by LMU ... and should be anyways hosted at their node ;)

javiquinte commented 5 years ago

Hi @jbienkowski ! Could you tell us which data should we see for station A020B? I have another user asking for these data and I want to be sure what should I tell him.

jbienkowski commented 5 years ago

Hi @javiquinte,

I did not get any notification about your mention, sorry for the late reply. Below is our WFCatalog info visualized for A020B on channel HHE, last write in the SDS took place on June 15 at 22:23 (Z3.A020B.00.HHE.D.2019.166), LH channels stopped in February.

A020B-ODC-Data-Availability

javiquinte commented 5 years ago

Thanks @jbienkowski ! It seems to be solved finally.

$ fdsnws_fetch -vvv -N Z3 -S "A020B" -C "HHZ" -s "2018-03-01" -e "2018-03-01T01:00:00" -o data.mseed using token in /Users/javier/.eidatoken: {"valid_until": "2019-06-23T08:08:26.591281Z", "cn": "Javier Quinteros", "memberof": "/epos/alparray;/epos;/", "sn": "Quinteros", "issued": "2019-05-24T08:08:26.591287Z", "mail": "javier@gfz-potsdam.de", "givenName": "Javier", "expiration": "1m"} getting routes from http://geofon.gfz-potsdam.de/eidaws/routing/1/query?station=A020B&endtime=2018-03-01T01%3A00%3A00&network=Z3&starttime=2018-03-01&channel=HHZ&format=post authenticating at https://www.orfeus-eu.org/fdsnws/dataselect/1/auth authentication at https://www.orfeus-eu.org/fdsnws/dataselect/1/auth successful getting data from http://www.orfeus-eu.org/fdsnws/dataselect/1/queryauth got 540672 bytes (application/vnd.fdsn.mseed) from http://www.orfeus-eu.org/fdsnws/dataselect/1/queryauth