Selective error when loading data with erddap

andrewfagerheim commented 1 year ago

I'm selectively having trouble loading data from erddap, where about half the time I'm able to run the code below without a problem, but other times the error below is returned. This has happened when choosing the same region, running the code in the same notebook, within the same hour even. Do you have thoughts on how to more reliably load data and avoid the following error?

MCVE Code Sample

import argopy
from argopy import DataFetcher as ArgoDataFetcher
erddap_loader  = ArgoDataFetcher(src='erddap',parallel=True,progress=True)
ds = erddap_loader.region([-30,-29,-70,70,0,2000])
ds = ds.to_xarray()

Error Returned

100%|██████████| 28/28 [01:00<00:00,  2.15s/it]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [2], in <cell line: 5>()
      3 erddap_loader  = ArgoDataFetcher(src='erddap',parallel=True,progress=True)
      4 ds = erddap_loader.region([-30,-29,-70,70,0,2000])
----> 5 ds = ds.to_xarray()

File ~/.conda/envs/argo/lib/python3.9/site-packages/argopy/fetchers.py:426, in ArgoDataFetcher.to_xarray(self, **kwargs)
    421 if not self.fetcher:
    422     raise InvalidFetcher(
    423         " Initialize an access point (%s) first."
    424         % ",".join(self.Fetchers.keys())
    425     )
--> 426 xds = self.fetcher.to_xarray(**kwargs)
    427 xds = self.postproccessor(xds)
    429 # data_path = self.fetcher.cname() + self._mode + ".zarr"
    430 # log.debug(data_path)
    431 # if self.cache and self.fs.exists(data_path):
   (...)
    435 #     xds = self.postproccessor(xds)
    436 #     xds = self._write(data_path, xds)._read(data_path)

File ~/.conda/envs/argo/lib/python3.9/site-packages/argopy/data_fetchers/erddap_data.py:470, in ErddapArgoDataFetcher.to_xarray(self, errors)
    468 else:
    469     try:
--> 470         ds = self.fs.open_mfdataset(
    471             self.uri, method=self.parallel_method, progress=self.progress, errors=errors
    472         )
    473     except ClientResponseError as e:
    474         raise ErddapServerError(e.message)

File ~/.conda/envs/argo/lib/python3.9/site-packages/argopy/stores/filesystems.py:634, in httpstore.open_mfdataset(self, urls, max_workers, method, progress, concat, concat_dim, preprocess, preprocess_opts, errors, *args, **kwargs)
    632         return results
    633 elif len(failed) == len(urls):
--> 634     raise ValueError("Errors happened with all URLs, this could be due to an internal impossibility to read returned content.")
    635 else:
    636     raise DataNotFound(urls)

ValueError: Errors happened with all URLs, this could be due to an internal impossibility to read returned content.

Versions

Output of `argopy.show_versions()`

SYSTEM ------ commit: None python: 3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:56:21) [GCC 10.3.0] python-bits: 64 OS: Linux OS-release: 3.10.0-1160.49.1.el7.centos.plus.x86_64 machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8 libhdf5: 1.10.6 libnetcdf: 4.7.4 INSTALLED VERSIONS: MIN ----------------------- aiohttp : 3.8.3 argopy : 0.1.12 erddapy : 1.2.1 fsspec : 2022.5.0 netCDF4 : 1.5.4 packaging : 21.3 scipy : 1.9.3 sklearn : 1.1.3 toolz : 0.11.2 xarray : 2022.11.0 INSTALLED VERSIONS: EXT.EXTRA ----------------------------- dask : 2022.05.2 distributed : 2022.5.2 gsw : 3.4.0 pyarrow : - tqdm : 4.64.1 INSTALLED VERSIONS: EXT.PLOTTERS -------------------------------- IPython : 8.4.0 cartopy : 0.20.2 ipykernel : 5.5.5 ipywidgets : 7.7.2 matplotlib : 3.4.3 seaborn : 0.12.1 INSTALLED VERSIONS: DEV ----------------------- bottleneck : - cfgrib : - cftime : 1.6.2 conda : - nc_time_axis: - numpy : 1.23.4 pandas : 1.5.1 pip : 22.1.2 pytest : - setuptools : 65.5.1 sphinx : - zarr : -

gmaze commented 1 year ago

hi @andrewfagerheim This is indeed not satisfactory but not due to argopy machinery. This happens because of the erddap server configuration. I communicated the issue to the server maintainers and will keep you posted

In the mean time, you could update to the last release candidate pip install argopy==0.1.14rc2

gmaze commented 11 months ago

Closing this issue because no solution is possible on argopy side

euroargodev / argopy