search

OpenDrift / opendrift

Open source framework for ocean trajectory modelling
https://opendrift.github.io
GNU General Public License v2.0
245 stars 120 forks source link

NetCDF > 1.6.1 does not use .netrc/.dodsrc: Trouble accessing CMEMS sources #1083

Open Elcook opened 1 year ago

Elcook commented 1 year ago

Hi,

I installed opendrift folowing instructions you provide. When using a conda environment I am not able to reach CMEMS source, no mather where I add my .dodsrc and .netrc file.

I have no problems using ncdump -h 'https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i' outside of conda environment but after doing:

conda activate opendrift and (opendrift) [root@opendrift-poc ~]# ncdump -h 'https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i'

I get the following error message:

syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR context: <!DOCTYPE^ html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <head> <title>Login</title> <link href="http://marine.copernicus.eu/favicon.ico" rel="shortcut icon" type="image/x-icon" /> <!-- link rel="icon" type="image/png" href="http://myo-wp.techworks.ie/devel/templates/myocean/favicon.png" /-->> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <link rel="stylesheet" type="text/css" href="/cas/newCss/css/myocean/screen/content.css" media="all" /> <link rel="stylesheet" type="text/css" href="/cas/newCss/css/myocean/screen/content_cc.css" media="all" /> <link rel="stylesheet" type="text/css" href="/cas/newCss/css/myocean/screen/patch_content_cc.css" media="all" /> <link rel="stylesheet" type="text/css" href="/cas/newCss/css/cssmanager.php?version=4.0.2-1288885946&amp;files=/cas/newCss/css/myocean/print/print_003.css" media="print" /> <!--[if gte IE 6]><style type="text/css" media="screen">@import 'css/ie_cas.css';</style><![endif]--> <script type="text/javascript" src="js/common_rosters.js"></script> <style type="text/css">input.submit{cursor:pointer}</style> </head> <body id="cas" class="claro" onload="init();" > <div id="white_background_inner_page" style="height:105px;"></div> <div id="col3_content" class="clearfix"> <img src="/css/myocean/print/images/logo_myocean_print.png" width="1" height="1" align="left" id="logoPrint" alt="" /> <div id="breadcrumb"> <a href="http://marine.copernicus.eu/">Home</a> &gt; Login </div> <h1>Welcome to the CMEMS Data Access Portal</h1> <br/> <p>Fill your login/password and click on <strong>Login</strong> to download data. If you are not registred yet click on <a href="http://marine.copernicus.eu/web/56-user-registration-form.php" target="_parent">Register</a></p> <p>If you have trouble logging in, make sure your browser is set to accept cookies.</p> <br/> <form id="authentification" name="com-login" class="fm-v clearfix" action="/cas/login;jsessionid=BA9435DA67876D405E6490F3DFC023A9?service=http://localhost:8080/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i.dds" method="post"> <fieldset class="input"> <label for="username">Username</label> <input id="username" name="username" class="required" tabindex="1" accesskey="i" type="text" value="" size="25" autocomplete="false"/><br /> <label for="password">Password</label> <input id="password" name="password" class="required" tabindex="2" accesskey="m" type="password" value="" size="25" autocomplete="off"/><br /> <input type="hidden" name="lt" value="_c4793F86B-451F-2DD7-ECFA-D395B0B99CF4_k3D0FA0E2-B51A-ACF8-EF1A-B307A76D684E" /> <input type="hidden" name="_eventId" value="submit" /> <div class="authentification_retrait_right"> <a href="http://marine.copernicus.eu/web/56-user-registration-form.php" target="_parent"><img src="/cas/newCss/css/myocean/screen/images/bt_register.gif" alt="Register" /></a> <input class="submit" name="submit" accesskey="l" value=" " type="submit" alt="Login" /> </div> </fieldset> <p> <br/><br/> <i>- You are registered but have forgotten your login/password?</i><br/> <i>- Any question?</i><br/> <b>&rArr; CMEMS Service Desk : </b><A HREF="mailto:servicedesk.cmems@mercator-ocean.eu"><b>servicedesk.cmems@mercator-ocean.eu</b></A> </p> </form> <h3>Thank you for using CMEMS products</h3> <br/> <div id="sidebar"> <i>For security reasons, please Exit your web browser when you quit services requiring authentication!</i> </div> </div> </body></html> ncdump: https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i: NetCDF: Access failure

Of course, using python's modules inside the env yields similar results

from opendrift.readers import reader_netCDF_CF_generic reader_netCDF_CF_generic.Reader('https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i')

results in

Traceback (most recent call last):
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 210, in _acquire_with_cache_info
    file = self._cache[self._key]
           ~~~~~~~~~~~^^^^^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
    value = self._cache[key]
            ~~~~~~~~~~~^^^^^
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '297eccf1-3b01-443c-a779-283fa9800280']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/opendrift/readers/reader_netCDF_CF_generic.py", line 146, in __init__
    self.Dataset = xr.open_dataset(filename, decode_times=False)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/api.py", line 526, in open_dataset
    backend_ds = backend.open_dataset(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 577, in open_dataset
    store = NetCDF4DataStore.open(
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 382, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 329, in __init__
    self.format = self.ds.data_model
                  ^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 391, in ds
    return self._acquire()
           ^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 385, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/root/miniconda3/envs/opendrift/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 198, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 216, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 2449, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2012, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -77] NetCDF: Access failure: 'https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/opendrift/readers/reader_netCDF_CF_generic.py", line 148, in __init__
    raise ValueError(e)
ValueError: [Errno -77] NetCDF: Access failure: 'https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i'

The only instalation that worked is the one via a docker file, but it doesn't suit my needs.

my conda environments is as follows:

     active environment : opendrift
    active env location : /root/miniconda3/envs/opendrift
            shell level : 2
       user config file : /root/.condarc
 populated config files : /root/.condarc
          conda version : 23.1.0
    conda-build version : not installed
         python version : 3.10.9.final.0
       virtual packages : __archspec=1=x86_64
                          __glibc=2.28=0
                          __linux=4.18.0=0
                          __unix=0=0
       base environment : /root/miniconda3  (writable)
      conda av data dir : /root/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /root/miniconda3/pkgs
                          /root/.conda/pkgs
       envs directories : /root/miniconda3/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/23.1.0 requests/2.28.1 CPython/3.10.9 Linux/4.18.0-425.19.2.el8_7.x86_64 rocky/8.7 glibc/2.28
                UID:GID : 0:0
             netrc file : /root/.netrc
           offline mode : False

# conda environments:
#
base                     /root/miniconda3
opendrift             *  /root/miniconda3/envs/opendrift

sys.version: 3.10.9 (main, Jan 11 2023, 15:21:40) [GC...
sys.prefix: /root/miniconda3
sys.executable: /root/miniconda3/bin/python
conda location: /root/miniconda3/lib/python3.10/site-packages/conda
conda-build: None
conda-content-trust: /root/miniconda3/bin/conda-content-trust
conda-env: /root/miniconda3/bin/conda-env
user site dirs:

CIO_TEST: <not set>
CONDA_DEFAULT_ENV: opendrift
CONDA_EXE: /root/miniconda3/bin/conda
CONDA_PREFIX: /root/miniconda3/envs/opendrift
CONDA_PREFIX_1: /root/miniconda3
CONDA_PROMPT_MODIFIER: (opendrift)
CONDA_PYTHON_EXE: /root/miniconda3/bin/python
CONDA_ROOT: /root/miniconda3
CONDA_SHLVL: 2
CURL_CA_BUNDLE: <not set>
GDAL_DRIVER_PATH: /root/miniconda3/envs/opendrift/lib/gdalplugins
LD_PRELOAD: <not set>
PATH: /root/miniconda3/envs/opendrift/bin:/root/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
REQUESTS_CA_BUNDLE: <not set>
SSL_CERT_FILE: <not set>
http_proxy: <set>
https_proxy: <set>

Any help would be greatly appreciated, I have run out of ideas already.

Thank you very much

gauteh commented 1 year ago

It looks like a recent bug in netcdf, maybe you can downgrade that package.

knutfrode commented 1 year ago

I also get the same problem when updating my environment, and so that netCDF version is 1.6.3 I tried then to downgrade to netCDF 1.6.2 (working on another machine), but same problem. Also downgrading Xarray to 2022.12.0 (working on another machine) did not work. Then not sure how to fix this?

Elcook commented 1 year ago

Would it be possible that the problem comes from the netCDF library, and not from the netCDF4 python module?

I'm assuming that because ncdump works outside the conda environment ( netcdf library version 4.7.0 ) but fails inside it (netcdf library version 4.9.2).

Anyway, I will try to test this and report here

vegardb commented 1 year ago

I can make this work for me with conda and netcdf4=1.6.1

Elcook commented 1 year ago

I can make this work for me with conda and netcdf4=1.6.1

You are correct, it worked for me too, even with netcdf library version 4.9.2.

knutfrode commented 1 year ago

Ok, I have now temporarily locked netcdf4 to <=1.6.1 in opendrift environment.yml, until bug is hopefully solved and backported by someone. https://github.com/OpenDrift/opendrift/commit/4626d20ba1fc07b21c71257624f5dd0beaf5f396

AndresSepulveda commented 1 year ago

How do you downgrade to netcd4 1.6.1

knutfrode commented 1 year ago

If you update to latest, environment.yml contains netcdf4<=1.6.1 You can then do conda env create -f environment.yml --force followed by pip install -e . to reinstall dependencies

miguelcorreia19 commented 1 year ago

Is it possible to define the directory where the .dodsrc is present? I want to change the directory where these files are present but the opendrift always will try on HOME dir. We can change this behavior?

knutfrode commented 1 year ago

.dodsrc and .netrc authentication are general, and not a part of OpenDrift. You can see specifications here: https://docs.unidata.ucar.edu/nug/current/dap2.html