search

astropy / astroquery

Functions and classes to access online data resources. Maintainers: @keflavich and @bsipocz and @ceb8
http://astroquery.readthedocs.org/en/latest/
BSD 3-Clause "New" or "Revised" License
706 stars 399 forks source link

Horizons session management #1807

Closed nabobalis closed 3 years ago

nabobalis commented 4 years ago

In sunpy we have a function that runs the following (cutdown example):

from astroquery.jplhorizons import Horizons
query = Horizons(*args,**kwargs)
result = query.vectors()

and we see the following warnings in our unit tests:

Exception ignored in: <ssl.SSLSocket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.21.156.203', 55482), raddr=('137.78.251.144', 443)>
ResourceWarning: unclosed <ssl.SSLSocket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.21.156.203', 55482), raddr=('137.78.251.144', 443)>

Adding query._session.close() removes these warnings but it feels like a hack to do it this way. Is there a better way to manage the session and have it autoclose?

bsipocz commented 4 years ago

cc @mommermi

mommermi commented 4 years ago

Weird. I cannot reproduce this behavior using something like

from astroquery.jplhorizons import Horizons
query = Horizons('Ceres')
result = query.vectors()

under Python 3.8.5. Could you provide a more complete example based on your unit tests (or can you point me somewhere)?

bsipocz commented 4 years ago

@nabobalis - These are wild guesses, but what is the version of astroquery for the tests where you see these? Is that latest deployed or latest tagged? What about astropy? is this with the dev, or a released one from the 4.0.x series?

nabobalis commented 4 years ago

All good questions, I will get back with an answer over the next few days. Thanks

nabobalis commented 4 years ago

Doing this again I can not create a simple example that errors. It only errors within the sunpy test suite:

pytest --remote-data=any sunpy/coordinates/tests/test_ephemeris.py::test_get_horizons_coord_array_time

======================================================== test session starts =========================================================
platform linux -- Python 3.8.5, pytest-6.0.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/nabil/GitHub/sunpy, configfile: setup.cfg
plugins: hypothesis-5.36.1, openfiles-0.5.0, forked-1.3.0, filter-subpackage-0.1.1, doctestplus-0.8.0, xdist-2.1.0, cov-2.10.1, astropy-header-0.1.2, arraydiff-0.3, remotedata-0.3.2, mock-3.3.1, asdf-2.7.1
collected 1 item                                                                                                                     

sunpy/coordinates/tests/test_ephemeris.py .                                                                                    [100%]

========================================================= 1 passed in 1.15s ==========================================================
Exception ignored in: <ssl.SSLSocket fd=12, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.17.208.239', 59116), raddr=('137.78.251.144', 443)>
ResourceWarning: unclosed <ssl.SSLSocket fd=12, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.17.208.239', 59116), raddr=('137.78.251.144', 443)>

The code this function calls: https://github.com/sunpy/sunpy/blob/master/sunpy/coordinates/ephemeris.py#L200 Which I have a cut down version here:

import numpy as np
from astropy.time import Time
from astroquery.jplhorizons import Horizons
from sunpy.time import parse_time
import logging

def get_horizons_coord(body, time='now', id_type='majorbody', *, include_velocity=False):
    obstime = parse_time(time)
    # Convert to an array, even if scalar
    array_time = np.reshape(obstime, (-1,))
    query = Horizons(id=body, id_type=id_type,
                     location='500@10',  # Heliocentric (mean ecliptic)
                     epochs=array_time.tdb.jd.tolist())  # Time must be provided in JD TDB
    try:
        result = query.vectors()
    except Exception as e:  # Catch and re-raise all exceptions, and also provide query URL if generated
        if query.uri is not None:
            logging.error(
                f"See the raw output from the JPL HORIZONS query at {query.uri}")
        raise e
    return result

def test_get_horizons_coord_array_time():
    array_time = Time(['2013-05-01', '2013-06-01', '2013-04-01', '2013-03-01'])
    e = get_horizons_coord('Geocenter', array_time)

This does not raise the error that I get in the test suite.

This suggests something is amiss with the sunpy test suite and not the library here.

@nabobalis - These are wild guesses, but what is the version of astroquery for the tests where you see these? Is that latest deployed or latest tagged? What about astropy? is this with the dev, or a released one from the 4.0.x series?

Our test suite will put the latest version it can find on Pypi for astropy and astroquery.

Sorry I can't be more helpful.

bsipocz commented 4 years ago

For astroquery, is it picking up the deployed dev version, or the tagged version? If you have a test header with the version, it would be extremely useful to see that. Do you have a link to the failing build?

nabobalis commented 4 years ago

I will see if I can dig up a build. But I can replicate it locally when I remove the session close line in sunpy.

I am using astroquery==0.4.1 locally and in the CI.

What do you mean by test header?

bsipocz commented 4 years ago

E.g what https://github.com/astropy/pytest-astropy-header returns :)

As you can reproduce it locally, would you mind trying with the versions pip install --pre gives you? It should be https://pypi.org/project/astroquery/0.4.2.dev6366/ for astroquery and 4.1rc2 for astropy.

nabobalis commented 4 years ago

Stable packages

pytest --remote-data=any sunpy/coordinates/tests/test_ephemeris.py::test_get_horizons_coord_array_time                                                                                                                    sunpy  

============================================================================================================================= test session starts =============================================================================================================================
platform linux -- Python 3.8.5, pytest-6.0.2, py-1.9.0, pluggy-0.13.1

Running tests with Astropy version 4.0.1.post1.
Running tests in sunpy/coordinates/tests/test_ephemeris.py::test_get_horizons_coord_array_time.

Date: 2020-09-28T22:01:24

Platform: Linux-4.19.128-microsoft-standard-x86_64-with-glibc2.10

Executable: /opt/miniconda/envs/sunpy/bin/python

Full Python Version: 
3.8.5 (default, Sep  4 2020, 07:30:14) 
[GCC 7.3.0]

encodings: sys: utf-8, locale: UTF-8, filesystem: utf-8
byteorder: little
float info: dig: 15, mant_dig: 15

Package versions: 
Numpy: 1.19.2
Scipy: 1.5.2
Matplotlib: 3.3.2
h5py: not available
Pandas: 1.1.2
astropy-helpers: 4.0.1

Using Astropy options: remote_data: any.

rootdir: /home/nabil/GitHub/sunpy, configfile: setup.cfg
plugins: hypothesis-5.36.1, openfiles-0.5.0, forked-1.3.0, filter-subpackage-0.1.1, doctestplus-0.8.0, xdist-2.1.0, cov-2.10.1, astropy-header-0.1.2, arraydiff-0.3, remotedata-0.3.2, mock-3.3.1, asdf-2.7.1
collected 1 item                                                                                                                                                                                                                                                              

sunpy/coordinates/tests/test_ephemeris.py .                                                                                                                                                                                                                             [100%]

============================================================================================================================== 1 passed in 1.11s ==============================================================================================================================
Exception ignored in: <ssl.SSLSocket fd=12, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.17.221.75', 45710), raddr=('137.78.251.144', 443)>
ResourceWarning: unclosed <ssl.SSLSocket fd=12, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.17.221.75', 45710), raddr=('137.78.251.144', 443)>

Pre packages: Successfully installed astropy-4.1rc2 astroquery-0.4.2.dev0 mimeparse-0.1.3 pyvo-1.1

pytest --remote-data=any sunpy/coordinates/tests/test_ephemeris.py::test_get_horizons_coord_array_time                                                                                                                    sunpy  

============================================================================================================================= test session starts =============================================================================================================================
platform linux -- Python 3.8.5, pytest-6.0.2, py-1.9.0, pluggy-0.13.1

Running tests with Astropy version 4.1rc2.
Running tests in sunpy/coordinates/tests/test_ephemeris.py::test_get_horizons_coord_array_time.

Date: 2020-09-28T22:03:47

Platform: Linux-4.19.128-microsoft-standard-x86_64-with-glibc2.10

Executable: /opt/miniconda/envs/sunpy/bin/python

Full Python Version: 
3.8.5 (default, Sep  4 2020, 07:30:14) 
[GCC 7.3.0]

encodings: sys: utf-8, locale: UTF-8, filesystem: utf-8
byteorder: little
float info: dig: 15, mant_dig: 15

Package versions: 
Numpy: 1.19.2
Scipy: 1.5.2
Matplotlib: 3.3.2
h5py: not available
Pandas: 1.1.2

Using Astropy options: remote_data: any.

rootdir: /home/nabil/GitHub/sunpy, configfile: setup.cfg
plugins: hypothesis-5.36.1, openfiles-0.5.0, forked-1.3.0, filter-subpackage-0.1.1, doctestplus-0.8.0, xdist-2.1.0, cov-2.10.1, astropy-header-0.1.2, arraydiff-0.3, remotedata-0.3.2, mock-3.3.1, asdf-2.7.1
collected 1 item                                                                                                                                                                                                                                                              

sunpy/coordinates/tests/test_ephemeris.py .                                                                                                                                                                                                                             [100%]

============================================================================================================================== 1 passed in 1.15s ==============================================================================================================================
Exception ignored in: <ssl.SSLSocket fd=12, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.17.221.75', 45742), raddr=('137.78.251.144', 443)>
ResourceWarning: unclosed <ssl.SSLSocket fd=12, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.17.221.75', 45742), raddr=('137.78.251.144', 443)>