search

conda-forge / esgf-pyclient-feedstock

A conda-smithy repository for esgf-pyclient.
BSD 3-Clause "New" or "Revised" License
0 stars 7 forks source link

`esgf-pyclient` not working well for Python=3.8 and 3.9 #18

Closed valeriupredoi closed 1 year ago

valeriupredoi commented 1 year ago

Hey guys, we have just started using the conda-forge version (as opposed to the PyPI one that we have been using no problemo for ages now), and noticed that the package misbehaves in Python=3.8 and 3.9 (working first rate in 3.10). Here are the full fails in our GA test environment, but in a nutshell, here are the versions:

(esmvalcore-complete-py38) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValCore$ python -V
Python 3.8.13
(esmvalcore-complete-py38) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValCore$ conda list esgf
# packages in environment at /home/valeriu/miniconda3/envs/esmvalcore-complete-py38:
#
# Name                    Version                   Build  Channel
esgf-pyclient             0.3.1              pyh6c4a22f_1    conda-forge
(esmvalcore-complete-py38) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValCore$ conda list requests
# packages in environment at /home/valeriu/miniconda3/envs/esmvalcore-complete-py38:
#
# Name                    Version                   Build  Channel
requests                  2.28.1             pyhd8ed1ab_1    conda-forge
requests_cache            0.4.13                     py_0    conda-forge
types-requests            2.28.11.4          pyhd8ed1ab_0    conda-forge

and a minimal reproducible test fail can be run as:

import pyesgf.search
import requests.exceptions
from pathlib import Path

def esgf_search_files(facets):
    urls = ['https://esgf-index1.ceda.ac.uk/esg-search', 'https://esgf-node.llnl.gov/esg-search', 'https://esgf-data.dkrz.de/esg-search', 'https://esgf-node.ipsl.upmc.fr/esg-search', 'https://esg-dn1.nsc.liu.se/esg-search', 'https://esgf.nci.org.au/esg-search', 'https://esgf.nccs.nasa.gov/esg-search', 'https://esgdata.gfdl.noaa.gov/esg-search']
    search_args = {'distrib': True, 'timeout': 120, 'cache': Path('/home/valeriu/.esmvaltool/cache/pyesgf-search-results'), 'expire_after': 86400}
    for url in urls:
        connection = pyesgf.search.SearchConnection(url=url, **search_args)
        context = connection.new_context(
            pyesgf.search.context.FileSearchContext,
            **facets,
            latest=True,
        )
        results = context.search(
            batch_size=500,
            ignore_facet_check=True,
        )
    results = _search_index_nodes(facets)
    return results

facets = {'project': 'CMIP6', 'source_id': 'EC-Earth3', 'variant_label': 'r1i1p1f1', 'experiment_id': 'historical', 'grid_label': 'gr', 'table_id': 'Amon', 'variable': 'tas'}
esgf_search_files(facets)

which barfs the following trace stack:

Traceback (most recent call last):
  File "min_esgf_fail.py", line 25, in <module>
    esgf_search_files(facets)
  File "min_esgf_fail.py", line 16, in esgf_search_files
    results = context.search(
  File "/home/valeriu/miniconda3/envs/esmvalcore-complete-py38/lib/python3.8/site-packages/pyesgf/search/context.py", line 141, in search
    return ResultSet(sc, batch_size=batch_size)
  File "/home/valeriu/miniconda3/envs/esmvalcore-complete-py38/lib/python3.8/site-packages/pyesgf/search/results.py", line 42, in __init__
    self.__get_batch(0)
  File "/home/valeriu/miniconda3/envs/esmvalcore-complete-py38/lib/python3.8/site-packages/pyesgf/search/results.py", line 82, in __get_batch
    response = (self.context.connection
  File "/home/valeriu/miniconda3/envs/esmvalcore-complete-py38/lib/python3.8/site-packages/pyesgf/search/connection.py", line 158, in send_search
    self.open()
  File "/home/valeriu/miniconda3/envs/esmvalcore-complete-py38/lib/python3.8/site-packages/pyesgf/search/connection.py", line 103, in open
    self.session = create_single_session(
  File "/home/valeriu/miniconda3/envs/esmvalcore-complete-py38/lib/python3.8/site-packages/pyesgf/search/connection.py", line 366, in create_single_session
    session = (requests_cache
  File "/home/valeriu/miniconda3/envs/esmvalcore-complete-py38/lib/python3.8/site-packages/requests_cache/core.py", line 68, in __init__
    self.cache = backends.create_backend(backend, cache_name, backend_options)
  File "/home/valeriu/miniconda3/envs/esmvalcore-complete-py38/lib/python3.8/site-packages/requests_cache/backends/__init__.py", line 50, in create_backend
    return registry[backend_name](cache_name, **options)
  File "/home/valeriu/miniconda3/envs/esmvalcore-complete-py38/lib/python3.8/site-packages/requests_cache/backends/sqlite.py", line 28, in __init__
    self.responses = DbPickleDict(location + extension, 'responses', fast_save=fast_save)
TypeError: unsupported operand type(s) for +: 'PosixPath' and 'str'

the problem is the PosixPath from search_args - it seems that for older Pythonns Path objects are not supported by the conda-forge esgf pyclient package, perhaps you built the package against a too new requests_cache? Cheers :beer:

valeriupredoi commented 1 year ago

CC @bouweandela here too

valeriupredoi commented 1 year ago

OK I found the problem - upgrading requests_cache to 0.9.6 from conda-forge for noarch does the trick, if you OK with that I can open a PR pin the pkg in meta here :+1:

bouweandela commented 1 year ago

I noticed there are two packages for requests-cache on conda forge:

# Name                       Version           Build  Channel             
requests_cache                0.4.13            py_0  conda-forge

and

# Name                       Version           Build  Channel             
requests-cache                0.4.12          py27_0  conda-forge         
requests-cache                0.4.12          py34_0  conda-forge         
requests-cache                0.4.12          py35_0  conda-forge         
requests-cache                0.4.12          py36_0  conda-forge         
requests-cache                0.4.13          py27_0  conda-forge         
requests-cache                0.4.13          py35_0  conda-forge         
requests-cache                0.4.13          py36_0  conda-forge         
requests-cache                0.4.13            py_1  conda-forge         
requests-cache                 0.5.0            py_0  conda-forge         
requests-cache                 0.5.0    pyhd8ed1ab_1  conda-forge         
requests-cache                 0.5.2    pyhd8ed1ab_0  conda-forge         
requests-cache                 0.6.3    pyhd8ed1ab_0  conda-forge         
requests-cache                 0.6.3    pyhd8ed1ab_1  conda-forge         
requests-cache                 0.6.4    pyhd8ed1ab_0  conda-forge         
requests-cache                 0.7.0    pyhd8ed1ab_0  conda-forge         
requests-cache                 0.7.1    pyhd8ed1ab_0  conda-forge         
requests-cache                 0.7.2    pyhd8ed1ab_0  conda-forge         
requests-cache                 0.7.3    pyhd8ed1ab_0  conda-forge         
requests-cache                 0.7.4    pyhd8ed1ab_0  conda-forge         
requests-cache                 0.8.0    pyhd8ed1ab_0  conda-forge         
requests-cache                 0.8.1    pyhd8ed1ab_0  conda-forge         
requests-cache                 0.9.0    pyhd8ed1ab_0  conda-forge         
requests-cache                 0.9.1    pyhd8ed1ab_0  conda-forge         
requests-cache                 0.9.2    pyhd8ed1ab_0  conda-forge         
requests-cache                 0.9.3    pyhd8ed1ab_0  conda-forge         
requests-cache                 0.9.4    pyhd8ed1ab_0  conda-forge         
requests-cache                 0.9.5    pyhd8ed1ab_0  conda-forge         
requests-cache                 0.9.6    pyhd8ed1ab_0  conda-forge

which is regularly updated. Changing the dependency here from requests_cache to requests-cache seems to make sense, but why do you specifically need version 0.9.6?

valeriupredoi commented 1 year ago

I think that the very old ones ie 0.4.x are PyPI builds - I don't need 0.9.6 specifically, I just need something newer that has the sqlite.py updated to do a + between two Paths :grin: