Different wildcard handling for station and waveform queries

[calum-chamberlain]

Square-bracket wildcards are handled in the fdsnws/dataselect query, but appear not to be in the fdsnws/station service. Example using ObsPy 1.1.0:

from obspy.clients.fdsn import Client
from obspy import UTCDateTime

client = Client("GEONET", debug=True)
stations = ['OGWZ', 'TUWZ', 'BSWZ', 'MSWZ', 'CAW', 'DUWZ', 'MRZ', 'KIW', 'CMWZ', 'WEL']
t1 = UTCDateTime("2016-11-14T00:00:00.000000Z")
t2 = t1 + 86400

# This square brackets wildcard expansion works fine for dataselect
bulk = []
for station in stations:
    bulk.append(('NZ', station, '10', '[E,H]H?', t1, t2))
data = client.get_waveforms_bulk(bulk=bulk)

# But fails for get_stations_bulk when setting level to "channel" or "response"
inventory = client.get_stations_bulk(bulk=bulk, level='channel')

Output of final failed request:

Downloading http://service.geonet.org.nz/fdsnws/station/1/query with requesting gzip compression
Sending along the following payload:
----------------------------------------------------------------------
level=channel
NZ OGWZ 10 [E,H]H? 2016-11-14T00:00:00.000000 2016-11-15T00:00:00.000000
NZ TUWZ 10 [E,H]H? 2016-11-14T00:00:00.000000 2016-11-15T00:00:00.000000
NZ BSWZ 10 [E,H]H? 2016-11-14T00:00:00.000000 2016-11-15T00:00:00.000000
NZ MSWZ 10 [E,H]H? 2016-11-14T00:00:00.000000 2016-11-15T00:00:00.000000
NZ CAW 10 [E,H]H? 2016-11-14T00:00:00.000000 2016-11-15T00:00:00.000000
NZ DUWZ 10 [E,H]H? 2016-11-14T00:00:00.000000 2016-11-15T00:00:00.000000
NZ MRZ 10 [E,H]H? 2016-11-14T00:00:00.000000 2016-11-15T00:00:00.000000
NZ KIW 10 [E,H]H? 2016-11-14T00:00:00.000000 2016-11-15T00:00:00.000000
NZ CMWZ 10 [E,H]H? 2016-11-14T00:00:00.000000 2016-11-15T00:00:00.000000
NZ WEL 10 [E,H]H? 2016-11-14T00:00:00.000000 2016-11-15T00:00:00.000000
----------------------------------------------------------------------
Downloaded http://service.geonet.org.nz/fdsnws/station/1/query with HTTP code: 204
---------------------------------------------------------------------------
FDSNNoDataException                       Traceback (most recent call last)
<ipython-input-63-d33e26c5ce1c> in <module>()
----> 1 inventory = client.get_stations_bulk(bulk=bulk, level='channel')

/home/sw/anaconda3/envs/conda_35/lib/python3.5/site-packages/obspy/clients/fdsn/client.py in get_stations_bulk(self, bulk, level, includerestricted, includeavailability, filename, **kwargs)
   1153 
   1154         data_stream = self._download(url,
-> 1155                                      data=bulk)
   1156         data_stream.seek(0, 0)
   1157         if filename:

/home/sw/anaconda3/envs/conda_35/lib/python3.5/site-packages/obspy/clients/fdsn/client.py in _download(self, url, return_string, data, use_gzip)
   1380             debug=self.debug, return_string=return_string, data=data,
   1381             timeout=self.timeout, use_gzip=use_gzip)
-> 1382         raise_on_error(code, data)
   1383         return data
   1384 

/home/sw/anaconda3/envs/conda_35/lib/python3.5/site-packages/obspy/clients/fdsn/client.py in raise_on_error(code, data)
   1704     if code == 204:
   1705         raise FDSNNoDataException("No data available for request.",
-> 1706                                   server_info)
   1707     elif code == 400:
   1708         msg = ("Bad request. If you think your request was valid "

FDSNNoDataException: No data available for request.
Detailed response of server:

My current workaround is to expand wildcards on my end:

from obspy.clients.fdsn import Client
from obspy import UTCDateTime

client = Client("GEONET", debug=True)
stations = ['OGWZ', 'TUWZ', 'BSWZ', 'MSWZ', 'CAW', 'DUWZ', 'MRZ', 'KIW', 'CMWZ', 'WEL']
t1 = UTCDateTime("2016-11-14T00:00:00.000000Z")
t2 = t1 + 86400

bulk = []
for station in stations:
    bulk.append(('NZ', station, '10', 'EH?', t1, t2))
    bulk.append(('NZ', station, '10', 'HH?', t1, t2))

inventory = client.get_stations_bulk(bulk=bulk, level='channel')

[nbalfour]

@calum-chamberlain I tried to recreate this error the same query but with the URL rather than obspy and it worked. I think this could this be an issue with obspy?

This is the URL query that worked: http://service.geonet.org.nz/fdsnws/station/1/query?station=OGW,TUWZ,BSWZ,MSWZ,CAW,DUWZ,MRZ,KIW,CMWZ,WEL&location=10&channel=[EH]H?&starttime=2016-11-14T00:00:00.0&endtime=2016-11-15T00:00:00.0&level=channel

This is the obspy script like yours but getting NZ station inventory from the IRIS FDSNWS. It returned the same error.

client = Client("IRIS", debug=True)
stations = ['BFZ', 'BKZ', 'CTZ', 'HIZ', 'KHZ', 'ODZ', 'OUZ', 'QRZ', 'RPZ', 'URZ']
bulk = []
for station in stations:
    bulk.append(('NZ', station, '10', '[E,H]H?', t1, t2))
inventory = client.get_stations_bulk(bulk=bulk, level='channel')
print(inventory)

[nbalfour]

@calum-chamberlain, this is a bit odd.

The following request works, which is the same as yours but without the comma [EH] instead of [E,H].

client = Client("GEONET", debug=True)
stations = ['OGWZ', 'TUWZ', 'BSWZ', 'MSWZ', 'CAW', 'DUWZ', 'MRZ', 'KIW', 'CMWZ', 'WEL']
t1 = UTCDateTime("2016-11-14T00:00:00.000000Z")
t2 = t1 + 86400
bulk = []
for station in stations:
    bulk.append(('NZ', station, '10', '[EH]H?', t1, t2))
inventory = client.get_stations_bulk(bulk=bulk, level='channel')

But it doesn't work with the IRIS request with and without the comma.

client = Client("IRIS")
stations = ['BFZ', 'BKZ', 'CTZ', 'HIZ', 'KHZ', 'ODZ', 'OUZ', 'QRZ', 'RPZ', 'URZ']
bulk = []
for station in stations:
    bulk.append(('NZ', station, '10', '[E,H]H?', t1, t2))
inventory = client.get_stations_bulk(bulk=bulk, level='channel')

Error is

FDSNException                             Traceback (most recent call last)
<ipython-input-9-09bd6561f3b4> in <module>()
      4 for station in stations:
      5     bulk.append(('NZ', station, '10', '[EH]H?', t1, t2))
----> 6 inventory = client.get_stations_bulk(bulk=bulk, level='channel')
      7 print(inventory)

~\AppData\Local\Continuum\anaconda3\envs\obspy\lib\site-packages\obspy\clients\fdsn\client.py in get_stations_bulk(self, bulk, level, includerestricted, includeavailability, filename, **kwargs)
   1153 
   1154         data_stream = self._download(url,
-> 1155                                      data=bulk)
   1156         data_stream.seek(0, 0)
   1157         if filename:

~\AppData\Local\Continuum\anaconda3\envs\obspy\lib\site-packages\obspy\clients\fdsn\client.py in _download(self, url, return_string, data, use_gzip)
   1380             debug=self.debug, return_string=return_string, data=data,
   1381             timeout=self.timeout, use_gzip=use_gzip)
-> 1382         raise_on_error(code, data)
   1383         return data
   1384 

~\AppData\Local\Continuum\anaconda3\envs\obspy\lib\site-packages\obspy\clients\fdsn\client.py in raise_on_error(code, data)
   1708         msg = ("Bad request. If you think your request was valid "
   1709                "please contact the developers.")
-> 1710         raise FDSNException(msg, server_info)
   1711     elif code == 401:
   1712         raise FDSNException("Unauthorized, authentication required.",

FDSNException: Bad request. If you think your request was valid please contact the developers.
Detailed response of server:

Error 400: Invalid input parameter: Invalid channel search value: [[EH]H?]
Request:
http://service.iris.edu/fdsnws/station/1/query
Request Submitted:
2017/12/06 21:55:49 UTC
Service version:
Service: fdsnws-station  version: 1.1.32

In the obspy documentation they use the wildcard without a comma. eg https://docs.obspy.org/packages/autogen/obspy.core.inventory.inventory.Inventory.plot_response.html

[calum-chamberlain]

Hmm, odd - I will raise this over at ObsPy - do you know if, for the other services (e.g. dataselect), commas are removed from the urls? Thanks for looking into this!

[nbalfour]

@calum-chamberlain we think it was a mistake that the commas were included as an option. Dataselect and station were done by different people and they might have interpreted the spec differently. It seems like the appropriate use from the obspy documentation is without commas, which works for both dataselect and station.

Since the service works according to spec and obspy documentation I am going to close this ticket.