astroquery.gaia. exiting with TimeoutError [60] when using `launch_job_async()`

[jorgemarpa]

I am doing an asynchronous query to Gaia using launch_job_async(), but when the search radius gets larger than ~ 0.7 deg I get the time out error. For queries with smaller searches radius works ok. When the query is launch, I can see the job on the ESA Gaia archive website, but the astroquery job finishes with the error before the job is finished successfully on the server side.

Here's an example of the query I'm running.

from astroquery.gaia import Gaia, GaiaClass, TapPlus

Gaia.login(user="dummy_user", password="dummy_pass")

ras = [281.97159801]
decs = [44.29987168]
rads = [0.7]
epoch = 2009.3120406806293

wheres = [
        f"""1=CONTAINS(
                  POINT('ICRS',ra,dec),
                  CIRCLE('ICRS',{ra},{dec},{rad}))"""
        for ra, dec, rad in zip(ras, decs, rads)
    ]

where = """\n\tOR """.join(wheres)
query_txt = f"""SELECT designation,
                    coord1(prop) AS ra, ra_error, coord2(prop) AS dec, dec_error, parallax,
                    parallax_error, pmra, pmra_error, pmdec, pmdec_error, dr2_radial_velocity, dr2_radial_velocity_error,
                    ruwe, phot_g_n_obs, phot_g_mean_flux,
                    phot_g_mean_flux_error, phot_g_mean_mag,
                    phot_bp_n_obs, phot_bp_mean_flux, phot_bp_mean_flux_error, phot_bp_mean_mag, phot_rp_n_obs,
                    phot_rp_mean_flux, phot_rp_mean_flux_error,
                    phot_rp_mean_mag FROM (
             SELECT *,
             EPOCH_PROP_POS(ra, dec, parallax, pmra, pmdec, 0, ref_epoch, {epoch}) AS prop
             FROM gaiaedr3.gaia_source
             WHERE {where}
            )  AS subquery
            WHERE phot_g_mean_mag<=20
            """

job = Gaia.launch_job_async(query_txt, verbose=True)
r = job.get_results()

Output and error traceback:

Launched query: 'SELECT designation,
                    coord1(prop) AS ra, ra_error, coord2(prop) AS dec, dec_error, parallax,
                    parallax_error, pmra, pmra_error, pmdec, pmdec_error, dr2_radial_velocity, dr2_radial_velocity_error,
                    ruwe, phot_g_n_obs, phot_g_mean_flux,
                    phot_g_mean_flux_error, phot_g_mean_mag,
                    phot_bp_n_obs, phot_bp_mean_flux, phot_bp_mean_flux_error, phot_bp_mean_mag, phot_rp_n_obs,
                    phot_rp_mean_flux, phot_rp_mean_flux_error,
                    phot_rp_mean_mag FROM (
             SELECT *,
             EPOCH_PROP_POS(ra, dec, parallax, pmra, pmdec, 0, ref_epoch, 2009.3120406806293) AS prop
             FROM gaiaedr3.gaia_source
             WHERE 1=CONTAINS(
                  POINT('ICRS',ra,dec),
                  CIRCLE('ICRS',281.97159801,44.29987168,0.7))
            )  AS subquery
            WHERE phot_g_mean_mag<=20
            '
------>https
host = gea.esac.esa.int:443
context = /tap-server/tap/async
Content-type = application/x-www-form-urlencoded
303 303
[('Date', 'Thu, 13 May 2021 23:20:42 GMT'), ('Server', 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.4.16 mod_jk/1.2.43'), ('Cache-Control', 'no-cache, no-store, max-age=0, must-revalidate'), ('Pragma', 'no-cache'), ('Expires', '0'), ('X-XSS-Protection', '1; mode=block'), ('X-Frame-Options', 'SAMEORIGIN'), ('X-Content-Type-Options', 'nosniff'), ('Location', 'https://gea.esac.esa.int/tap-server/tap/async/1620948042470O'), ('Transfer-Encoding', 'chunked'), ('Content-Type', 'text/plain;charset=ISO-8859-1')]
job 1620948042470O, at: https://gea.esac.esa.int/tap-server/tap/async/1620948042470O
Retrieving async. results...

---------------------------------------------------------------------------
TimeoutError                              Traceback (most recent call last)
<ipython-input-79-97c30fb3390f> in <module>
----> 1 job = Gaia.launch_job_async(query_txt, verbose=True)
      2 r = job.get_results()

~/.pyenv/versions/adap/lib/python3.8/site-packages/astroquery/utils/tap/core.py in launch_job_async(self, query, name, output_file, output_format, verbose, dump_to_file, background, upload_resource, upload_table_name, autorun)
    441                         job.save_results(verbose)
    442                     else:
--> 443                         job.get_results()
    444                         log.info("Query finished.")
    445         return job

~/.pyenv/versions/adap/lib/python3.8/site-packages/astroquery/utils/tap/model/job.py in get_results(self)
    245         else:
    246             # async: result is in the server once the job is finished
--> 247             self.__load_async_job_results()
    248             return self.results
    249 

~/.pyenv/versions/adap/lib/python3.8/site-packages/astroquery/utils/tap/model/job.py in __load_async_job_results(self, debug)
    329 
    330     def __load_async_job_results(self, debug=False):
--> 331         wjResponse, phase = self.wait_for_job_end()
    332         subContext = "async/" + str(self.jobid) + "/results/result"
    333         resultsResponse = self.connHandler.execute_tapget(subContext)

~/.pyenv/versions/adap/lib/python3.8/site-packages/astroquery/utils/tap/model/job.py in wait_for_job_end(self, verbose)
    314                     print("Exception when trying to start job", ex)
    315         while True:
--> 316             responseData = self.get_phase(update=True)
    317             currentResponse = self.__last_phase_response_status
    318 

~/.pyenv/versions/adap/lib/python3.8/site-packages/astroquery/utils/tap/model/job.py in get_phase(self, update)
    184         if update:
    185             phase_request = "async/"+str(self.jobid)+"/phase"
--> 186             response = self.connHandler.execute_tapget(phase_request)
    187 
    188             self.__last_phase_response_status = response.status

~/.pyenv/versions/adap/lib/python3.8/site-packages/astroquery/utils/tap/conn/tapconn.py in execute_tapget(self, subcontext, verbose)
    194         else:
    195             context = self.__get_tap_context(subcontext)
--> 196             return self.__execute_get(context, verbose)
    197 
    198     def execute_dataget(self, query, verbose=False):

~/.pyenv/versions/adap/lib/python3.8/site-packages/astroquery/utils/tap/conn/tapconn.py in __execute_get(self, context, verbose)
    241             print("host = " + str(conn.host) + ":" + str(conn.port))
    242             print("context = " + context)
--> 243         conn.request("GET", context, None, self.__getHeaders)
    244         response = conn.getresponse()
    245         self.__currentReason = response.reason

~/.pyenv/versions/3.8.6/lib/python3.8/http/client.py in request(self, method, url, body, headers, encode_chunked)
   1253                 encode_chunked=False):
   1254         """Send a complete request to the server."""
-> 1255         self._send_request(method, url, body, headers, encode_chunked)
   1256 
   1257     def _send_request(self, method, url, body, headers, encode_chunked):

~/.pyenv/versions/3.8.6/lib/python3.8/http/client.py in _send_request(self, method, url, body, headers, encode_chunked)
   1299             # default charset of iso-8859-1.
   1300             body = _encode(body, 'body')
-> 1301         self.endheaders(body, encode_chunked=encode_chunked)
   1302 
   1303     def getresponse(self):

~/.pyenv/versions/3.8.6/lib/python3.8/http/client.py in endheaders(self, message_body, encode_chunked)
   1248         else:
   1249             raise CannotSendHeader()
-> 1250         self._send_output(message_body, encode_chunked=encode_chunked)
   1251 
   1252     def request(self, method, url, body=None, headers={}, *,

~/.pyenv/versions/3.8.6/lib/python3.8/http/client.py in _send_output(self, message_body, encode_chunked)
   1008         msg = b"\r\n".join(self._buffer)
   1009         del self._buffer[:]
-> 1010         self.send(msg)
   1011 
   1012         if message_body is not None:

~/.pyenv/versions/3.8.6/lib/python3.8/http/client.py in send(self, data)
    948         if self.sock is None:
    949             if self.auto_open:
--> 950                 self.connect()
    951             else:
    952                 raise NotConnected()

~/.pyenv/versions/3.8.6/lib/python3.8/http/client.py in connect(self)
   1422                 server_hostname = self.host
   1423 
-> 1424             self.sock = self._context.wrap_socket(self.sock,
   1425                                                   server_hostname=server_hostname)
   1426 

~/.pyenv/versions/3.8.6/lib/python3.8/ssl.py in wrap_socket(self, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname, session)
    498         # SSLSocket class handles server_hostname encoding before it calls
    499         # ctx._wrap_socket()
--> 500         return self.sslsocket_class._create(
    501             sock=sock,
    502             server_side=server_side,

~/.pyenv/versions/3.8.6/lib/python3.8/ssl.py in _create(cls, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname, context, session)
   1038                         # non-blocking
   1039                         raise ValueError("do_handshake_on_connect should not be specified for non-blocking sockets")
-> 1040                     self.do_handshake()
   1041             except (OSError, ValueError):
   1042                 self.close()

~/.pyenv/versions/3.8.6/lib/python3.8/ssl.py in do_handshake(self, block)
   1307             if timeout == 0.0 and block:
   1308                 self.settimeout(None)
-> 1309             self._sslobj.do_handshake()
   1310         finally:
   1311             self.settimeout(timeout)

TimeoutError: [Errno 60] Operation timed out

Thanks for the help!

[bsipocz]

@jorgemarpa - Thanks for the report. The gaia module is a bit different than the rest so there isn't (yet) an easy way to set the timeout on the Python end.

cc @jcsegovia as he may know a workaround

[barentsen]

This is a bit of a long shot because it may be overridden somewhere in the call chain, but you could try setting Python's default socket timeout as follows:

import socket
socket.setdefaulttimeout(600)

[jorgemarpa]

thanks for the suggestion @barentsen. I tried that before posting the issue, but it did not work.