Include Restricted FDSNWS stations and some advises for the FDSNWS url downlaod

[YacineBoussoufa]

I've noticed that during the download of some station sources, the restricted stations are automaticly rejected. I advise to allow the download of them removing the "includerestricted=false" statement from the generation because some services don't properly recognize the GET request and sometimes some restricted station have in fact a Seedlink server available.

In addition I would like to advise to create a control for the FDSN url generation, especially for the endafter parameter because as previously said, some fdsnws providers don't use it creating some issue with them:

[xspanger3770]

Thanks for the notice! Will definitely have a look into it. In the end, GlobalQuake worked originally with just 5 of those FDSNWS station sources and this issue probably occurred with the new ones. I hoped that the FDSNWS services would all behave the same but apparently they don't :D

[xspanger3770]

I tried testing the download with includerestricted and endafter parameters removed, but it lead to different problems: when removing the endafter parameters, a lot of stations will get downloaded that are no longer running - mostly some temporary stations from IRIS. Also, the amount of stations that needs to get downloaded can trigger HTTP 413 response: The result set of your request exceeds the configured maximum number of objects (250000). Refine your request parameters.

Maybe I will include another column with optional parameters for each fdsnws source or idk what is the best solution

[YacineBoussoufa]

Maybe if an error code is returned by the download url, it generates another url with only the most important parameters

[xspanger3770]

Yeah thinking about something like that... if 413 is returned, then generate new requests but with some limitation..maybe split by longitude into 2,4,8... requests

[xspanger3770]

I hope that all of the fdsnws services support at least geographic constraints for this to work

[xspanger3770]

@YacineBoussoufa Can you please show me which fdsnws provider had the issue with parsing endafter parameter? Can't find any.

[YacineBoussoufa]

https://esm-db.eu/fdsnws/station/1/query?level=station&format=text&endafter=2023-08-17T00:00:00&includerestricted=false (doesn't support both includerestricted and endafter) https://service.geonet.org.nz/fdsnws/station/1/query?level=station&format=text&endafter=2023-08-17T00:00:00&includerestricted=false (doesn't support includerestricted)

[xspanger3770]

Thanks! Trying to resolve this issues now...

[YacineBoussoufa]

I've also noted that https://ayiti.unice.fr/ayiti-seismes/fdsnws/station/1 and https://sismoazur.oca.eu/fdsnws/station/1 Work only when the last "/" is excluded from the url on Globalquake, while on the web and other fdsnws on the software the last / is necessary

[xspanger3770]

Oh... that's really weird, but fortunately you can exclude or include the "/" from the url within the edit dialog so it shouldn't be an issue i guess

[xspanger3770]

This is really a nightmare... just found out that some fdsnws providers can't even parse the channel filter

This is ok and will only download HHZ channels https://sismoazur.oca.eu/fdsnws/station/1/query?minlongitude=-180.0&maxlongitude=180.0&level=channel&endafter=2023-08-17T19:14:34&format=xml&channel=HHZ

But this will return empty page: (other providers return HHZ and BHZ channels) https://sismoazur.oca.eu/fdsnws/station/1/query?minlongitude=-180.0&maxlongitude=180.0&level=channel&endafter=2023-08-17T19:14:34&format=xml&channel=HHZ,BHZ

[YacineBoussoufa]

This can be fixed with using double parameters "&channel=HHZ&channel=BHZ"

[xspanger3770]

Oh great! hope it's universal

[YacineBoussoufa]

Never mind, just tested again with oca and doesn't work... it only shows one channel... I think the easies way it to use "&channel=?HZ"

[xspanger3770]

Yeah I see only HHZ channels are returned...

[YacineBoussoufa]

As I said i think "&channel=?HZ" is the most appropriate, because some station also only use EHZ for example.

[xspanger3770]

?HZ seems like the solution

[xspanger3770]

Also I discovered that different fdsnws providers run different versions of the protocol... for example https://esm-db.eu/fdsnws/station/1/version shows that esm runs on 0.0.1 and this version doesn't support the endafter parameter as you can see here: https://esm-db.eu/fdsnws/station/1/application.wadl

[xspanger3770]

I was able to put it all together by doing the following:

• I removed the includerestricted parameter
• I made a parser for the application.wadl file and for the providers that don't support endafter it will be ignored
• if HTTP 413 is returned, then additional geographic constraints are recursively applied until the number of elements is within limits

After adding the providers that were failing before the total number of available channels went from ~4200 to 5800!!! Most importantly, the entire New Zealand is now covered with available stations from Geonet!

Huge thanks for the suggestions!

[YacineBoussoufa]

Good job 😍

[xspanger3770]

Closed - those new cool features will be ready in 0.9.3.