Problem with MPC.get_observations

[Yeqzids]

I am getting "Ran out of input" or "_pickle.PicklingError: Can't pickle <class 'astropy.utils.masked.core.MaskedQuantity'>: attribute lookup MaskedQuantity on astropy.utils.masked.core failed" on certain objects. Querying on the MPC website successfully return observations of these objects, and there is no obvious pattern to me on the objects that fail.

For example, when trying to run the example here:

>>> from astroquery.mpc import MPC
>>> MPC.get_observations(12893)

I am getting "Ran out of input" error on astroquery=0.4.5, but it runs without issue on 0.4.1. However, if I try MPC.get_observations(12894), I am getting the PicklingError on both versions. I see that #2280 refers to the same PicklingError during initialization, not sure if it is relevant.

Any thoughts/pointers are appreciated. Thanks!

[mkelley]

This is definitely the same problem I'm having and #2280 was not the right fix for it. I'm trying to reproduce this in the testing environment.

[mkelley]

A quick fix is probably to downgrade astropy to <5. I'll need to investigate the mpc module to see if we can avoid using MaskedQuantity at all. According to astropy's docs, this is an experimental feature so, pickling error aside, we should avoid using it.

@migueldvb This is relevant to the sbpy fix you were looking into.

[Yeqzids]

@mkelley What's the environment that worked for you? I have tried astropy=3.2.3 and 4.3.1 and I am still getting the "Ran out of input" error with the example (MPC.get_observations(12893)).

[mkelley]

I think there are two issues occurring here. First is the pickling error with MaskedQuantity, which causes a bad cache file to be written. Then next time the query is run, the bad cache file is read and the "Ran out of input" error occurs. At least, that's my understanding at the moment.

I generally clear my cache by deleting ~/.astropy/cache/astroquery/ or, to be more specific to this problem, ~/.astropy/cache/astroquery/MPC.

[mkelley]

But I do not understand why the caching is even trying to pickle a masked quantity in the first place. Running the debugger on the crash within to_cache():

(Pdb) pickle
<module 'pickle' from '/usr/local/lib/python3.8/pickle.py'>
(Pdb) requests
<module 'requests' from '/tmp/astroquery/.tox/py38-test/lib/python3.8/site-packages/requests/__init__.py'>
(Pdb) response       
<Response [200]>
(Pdb) response.__class__
<class 'requests.models.Response'>
(Pdb) pickle.dumps(response)
*** _pickle.PicklingError: Can't pickle <class 'astropy.utils.masked.core.MaskedQuantity'>: attribute lookup MaskedQuantity on astropy.utils.masked.core failed

Running the query and pickling outside of astroquery is fine:

>>> import requests
>>> import pickle
>>> response = requests.get('https://minorplanetcenter.net/search_db/observations?number=12893&object_type=M&table=observations')
>>> pickle.dumps(response)[:100]
b'\x80\x04\x950\x00\x00\x00\x00\x00\x00\x00\x8c\x0frequests.models\x94\x8c\x08Response\x94\x93\x94)\x81\x94}\x94(\x8c\x08_content\x94B\x1a!\x08\x00[{"designation":"1998 QS55","discove'

I'm trying to search the response object for MaskedQuantity...

[mkelley]

I'm not finding the problem. The only thing I can think to propose is to catch errors at the time the cache file is written and avoid the crash, but that's not the best solution.

[keflavich]

I'm quite perplexed by the debugger stuff you showed. Could we try something like str(response) (or, str(response).encode('ascii')) to force it to something genuinely picklable?

But why does Response contain quantities? That still seems weird.

[Yeqzids]

I think there are two issues occurring here. First is the pickling error with MaskedQuantity, which causes a bad cache file to be written. Then next time the query is run, the bad cache file is read and the "Ran out of input" error occurs. At least, that's my understanding at the moment.

I generally clear my cache by deleting ~/.astropy/cache/astroquery/ or, to be more specific to this problem, ~/.astropy/cache/astroquery/MPC.

Thanks, @mkelley . In case anyone goes down this route: using astropy=4.3.4 and clearing the cache works nicely. However, if you want to use astroquery.jplhorizons to retrieve SB ephemerides in the same script/environment, you may encounter "Max retries exceeded... unsafe legacy renegotiation disabled" error, which does not occur under astropy=5.0. My workaround is to run MPC.get_observations under astropy=4.3.4 to cache the observations, and then manipulate them under astropy=5.0. I am not filing a separate issue for now since astropy=4.3.4 has been superceded, but I am writing it down here in case anybody runs into the same issue...

My sample code to reproduce this, under 4.3.4:

>>> import astroquery.jplhorizons as jpl
>>> j = jpl.Horizons(id='12893', location='500', epochs=2451545).ephemerides()

[keflavich]

using dill.detect, it becomes clear that somewhere in the stack, the response object is pointing back to the Query that created it.

ipdb> dill.detect.errors(response)
T4: <class 'requests.models.Response'>
# T4
D2: <dict object at 0x7fd186fbc9c0>
T4: <class 'requests.structures.CaseInsensitiveDict'>
# T4
D2: <dict object at 0x7fd1873b8e80>
T4: <class 'collections.OrderedDict'>
# T4
# D2
D2: <dict object at 0x7fd187273080>
D2: <dict object at 0x7fd1ac3f0e00>
# D2
T4: <class 'requests.cookies.RequestsCookieJar'>
# T4
D2: <dict object at 0x7fd1871c1380>
T4: <class 'http.cookiejar.DefaultCookiePolicy'>
# T4
D2: <dict object at 0x7fd186fba4c0>
# D2
D2: <dict object at 0x7fd186ecbc40>
# D2
# D2
T4: <class 'datetime.timedelta'>
# T4
T4: <class 'requests.models.PreparedRequest'>
# T4
D2: <dict object at 0x7fd18702aac0>
D2: <dict object at 0x7fd187024d40>
# D2
D2: <dict object at 0x7fd186ec36c0>
D2: <dict object at 0x7fd1872d6a80>
# D2
D2: <dict object at 0x7fd187478d80>
# D2
# D2
D2: <dict object at 0x7fd186ec8a00>
Me: <bound method BaseQuery._response_hook of <astroquery.mpc.core.MPCClass object at 0x7fd18de15280>>
T1: <class 'method'>
F2: <function _load_type at 0x7fd18747a700>
# F2
# T1
F1: <function BaseQuery._response_hook at 0x7fd18d77eaf0>
F2: <function _create_function at 0x7fd1861600d0>
# F2
Co: <code object _response_hook at 0x7fd18de1cc90, file "/home/adam/repos/astroquery/astroquery/query.py", line 183>
F2: <function _create_code at 0x7fd186160160>
# F2
Co: <code object <genexpr> at 0x7fd18de1c9d0, file "/home/adam/repos/astroquery/astroquery/query.py", line 188>
# Co
Co: <code object <genexpr> at 0x7fd18de1ca80, file "/home/adam/repos/astroquery/astroquery/query.py", line 199>
# Co
# Co
D4: <dict object at 0x7fd18df2ea80>
# D4
D2: <dict object at 0x7fd18702eb00>
# D2
# F1
T4: <class 'astroquery.mpc.core.MPCClass'>
# T4
D2: <dict object at 0x7fd18d775ac0>
T4: <class 'requests.sessions.Session'>
# T4
D2: <dict object at 0x7fd187240080>
D2: <dict object at 0x7fd18d775680>
# D2
D2: <dict object at 0x7fd18d6b2800>
D2: <dict object at 0x7fd18d775700>
# D2
D2: <dict object at 0x7fd18d7757c0>
# D2
# D2
D2: <dict object at 0x7fd18d775640>
# D2
D2: <dict object at 0x7fd18d775440>
Me: <bound method BaseQuery._response_hook of <astroquery.mpc.core.MPCClass object at 0x7fd18de15280>>
# Me
# D2
D2: <dict object at 0x7fd18d7756c0>
# D2
T4: <class 'requests.adapters.HTTPAdapter'>
# T4
D2: <dict object at 0x7fd187238c00>
T4: <class 'urllib3.util.retry.Retry'>
# T4
D2: <dict object at 0x7fd18d7753c0>
# D2
D2: <dict object at 0x7fd18d775780>
# D2
# D2
D2: <dict object at 0x7fd1871d8980>
D2: <dict object at 0x7fd18d775880>
# D2
D2: <dict object at 0x7fd18d775a40>
# D2
# D2
# D2
T4: <class 'astroquery.query.AstroQuery'>
# T4
D2: <dict object at 0x7fd1873c6080>
D2: <dict object at 0x7fd1874056c0>
# D2
# D2
T4: <class 'astropy.table.table.QTable'>
# T4
B3: <built-in function _reconstruct>
F2: <function _get_attr at 0x7fd186160b80>
# F2
M2: <module 'numpy.core._multiarray_umath' from '/home/adam/anaconda3/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so'>
F2: <function _import_module at 0x7fd186160ca0>
# F2
# M2
# B3
T4: <class 'astropy.table.column.Column'>
# T4
T4: <class 'numpy.dtype'>
# T4
F2: <function _mareconstruct at 0x7fd1ad7d1430>
# F2
T4: <class 'astropy.table.column.MaskedColumn'>
# T4
T4: <class 'astropy.table.column.BaseColumn'>
# T4
T4: <class 'numpy.ndarray'>
# T4
T4: <class 'astropy.units.quantity.Quantity'>
# T4
D2: <dict object at 0x7fd18701a2c0>
T4: <class 'astropy.units.core.Unit'>
# T4
D2: <dict object at 0x7fd18ccf5740>
T4: <class 'astropy.units.core.CompositeUnit'>
# T4
D2: <dict object at 0x7fd1871c4cc0>
D2: <dict object at 0x7fd1871e53c0>
D2: <dict object at 0x7fd1871dc340>
F2: <function _recreate_irreducible_unit at 0x7fd194d23e50>
# F2
T4: <class 'astropy.units.core.IrreducibleUnit'>
# T4
D2: <dict object at 0x7fd187211d40>
D2: <dict object at 0x7fd186fc0d00>
# D2
# D2
# D2
D2: <dict object at 0x7fd194f15540>
# D2
# D2
D2: <dict object at 0x7fd1871d8580>
# D2
# D2
D2: <dict object at 0x7fd194f1e4c0>
# D2
# D2
T4: <class 'astropy.units.quantity.QuantityInfo'>
# T4
D2: <dict object at 0x7fd1871f2c40>
# D2
# D2
D2: <dict object at 0x7fd1871cd580>
D2: <dict object at 0x7fd1873cb900>
D2: <dict object at 0x7fd1872e2e40>
D2: <dict object at 0x7fd1871f93c0>
D2: <dict object at 0x7fd194e65a40>
# D2
# D2
# D2
D2: <dict object at 0x7fd194e68140>
# D2
# D2
D2: <dict object at 0x7fd1871de180>
# D2
# D2
D2: <dict object at 0x7fd1871e3880>
D2: <dict object at 0x7fd1871f5c00>
# D2
# D2
T4: <class 'astropy.utils.masked.core.MaskedQuantity'>
PicklingError("Can't pickle <class 'astropy.utils.masked.core.MaskedQuantity'>: it's not found as astropy.utils.masked.core.MaskedQuantity")

The key clue is here: Me: <bound method BaseQuery._response_hook of <astroquery.mpc.core.MPCClass object at 0x7fd18de15280>> which is the only place the response object is modified.

I think the solution here is to remove the response hook before pickling - and this is something we should do generically.