search

EIDA / userfeedback

This repository is meant to collect feedback from EIDA users by means of its Issue Tracker
10 stars 4 forks source link

One connection restriction when downloading data from eida.ethz.ch with an Obspy Client #127

Closed savardge closed 1 year ago

savardge commented 1 year ago

Describe the problem I am attempting to download ambient noise data from eida.ethz.ch in parallel on my University's HPC cluster (login1.yggdrasil.hpc.unige.ch). In particular, I am using the S0A script part of NoisePy. The advantage of using this script as opposed to separately downloading the data with e.g. wget or Obspy Mass Downloader is that since the ambient noise data has to be reformated and pre-processed in the script and saved to HDF5 for the next steps, on-the-fly data requests using Obspy allows one to not eat up precious disk space on the university's HPC cluster by not saving intermediate mseed files locally. I tried to request data from ETH from the CH network, unfortunately I was blocked and the HPC staff were contacted by EIDA/ETH web service administrators to ask to limit the number of connections to 1. This effectively prevents me from using NoisePy's MPI script to speed up data download and pre-processing. Note that I did not use an EIDA token, but I just requested one now. Does using the token allows more than 1 connection to the ETH server ? Is there any way to allow the use of MPI scripts that download data with an Obspy Client ?

To Reproduce NoisePy see script S0A in src. I'm attempting to request data from CH network, for all stations within 8 km of Basel in parallel using MPI.

Screenshots A "No FDSN data available" / connection timeout error is raised.

Client eida.ethz.ch requesting data using Obspy regular Client

Additional context July 28 2022

damb commented 1 year ago

Thanks for reporting.

eida.ethz.ch is not eidaws-federator related. Instead, this is ETHZ/SED's EIDA endpoint URL. Generally, waveform data from network CH should be publicly available (i.e. the access shouldn't be restricted and therefore no token is required).

I'm not familiar with NoisePy. However, it seems to use obspy under the hood to download the waveform data: https://github.com/mdenolle/NoisePy/blob/cf93544c80904ab2b0166f1479e61049a4e54d23/src/S0A_dowload_ASDF_MPI.py#L273-L278

@savardge, how many concurrent connections are you using? AFAIK, the service allows downloading waveform data with using multiple connections. Though, I don't know what is the maximum number of concurrent connections / client.

@sheimers, could you please look into this. Thanks.

jschaeff commented 1 year ago

No news from OP, I close this issue. If needed, feel free to open again.