Vizier remote disconnect without response

[awe2]

Unknown if related to issue with units; currently this service is disconnecting without a response. If my two cents help, sounds like the URL form submitted might be formatted incorrectly (perhaps due to my own hack of casting series to np.ndarray) i.e., this could be fixed with resolution of units issue.

OR this service is just down?

---------------------------------------------------------------------------
RemoteDisconnected                        Traceback (most recent call last)
File ~/.conda/envs/torch2.0.0/lib/python3.9/site-packages/urllib3/connectionpool.py:714, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    713 # Make the request on the httplib connection object.
--> 714 httplib_response = self._make_request(
    715     conn,
    716     method,
    717     url,
    718     timeout=timeout_obj,
    719     body=body,
    720     headers=headers,
    721     chunked=chunked,
    722 )
    724 # If we're going to release the connection in ``finally:``, then
    725 # the response doesn't need to know about the connection. Otherwise
    726 # it will also try to release it and we'll have a double-release
    727 # mess.

File ~/.conda/envs/torch2.0.0/lib/python3.9/site-packages/urllib3/connectionpool.py:466, in HTTPConnectionPool._make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    462         except BaseException as e:
    463             # Remove the TypeError from the exception chain in
    464             # Python 3 (including for exceptions like SystemExit).
    465             # Otherwise it looks like a bug in the code.
--> 466             six.raise_from(e, None)
    467 except (SocketTimeout, BaseSSLError, SocketError) as e:

File <string>:3, in raise_from(value, from_value)

File ~/.conda/envs/torch2.0.0/lib/python3.9/site-packages/urllib3/connectionpool.py:461, in HTTPConnectionPool._make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    460 try:
--> 461     httplib_response = conn.getresponse()
    462 except BaseException as e:
    463     # Remove the TypeError from the exception chain in
    464     # Python 3 (including for exceptions like SystemExit).
    465     # Otherwise it looks like a bug in the code.

File ~/.conda/envs/torch2.0.0/lib/python3.9/http/client.py:1377, in HTTPConnection.getresponse(self)
   1376 try:
-> 1377     response.begin()
   1378 except ConnectionError:

File ~/.conda/envs/torch2.0.0/lib/python3.9/http/client.py:320, in HTTPResponse.begin(self)
    319 while True:
--> 320     version, status, reason = self._read_status()
    321     if status != CONTINUE:

File ~/.conda/envs/torch2.0.0/lib/python3.9/http/client.py:289, in HTTPResponse._read_status(self)
    286 if not line:
    287     # Presumably, the server closed the connection before
    288     # sending a valid response.
--> 289     raise RemoteDisconnected("Remote end closed connection without"
    290                              " response")
    291 try:

RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

ProtocolError                             Traceback (most recent call last)
File ~/.conda/envs/torch2.0.0/lib/python3.9/site-packages/requests/adapters.py:489, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    488 if not chunked:
--> 489     resp = conn.urlopen(
    490         method=request.method,
    491         url=url,
    492         body=request.body,
    493         headers=request.headers,
    494         redirect=False,
    495         assert_same_host=False,
    496         preload_content=False,
    497         decode_content=False,
    498         retries=self.max_retries,
    499         timeout=timeout,
    500     )
    502 # Send the request.
    503 else:

File ~/.conda/envs/torch2.0.0/lib/python3.9/site-packages/urllib3/connectionpool.py:798, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    796     e = ProtocolError("Connection aborted.", e)
--> 798 retries = retries.increment(
    799     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
    800 )
    801 retries.sleep()

File ~/.conda/envs/torch2.0.0/lib/python3.9/site-packages/urllib3/util/retry.py:550, in Retry.increment(self, method, url, response, error, _pool, _stacktrace)
    549 if read is False or not self._is_method_retryable(method):
--> 550     raise six.reraise(type(error), error, _stacktrace)
    551 elif read is not None:

File ~/.conda/envs/torch2.0.0/lib/python3.9/site-packages/urllib3/packages/six.py:769, in reraise(tp, value, tb)
    768 if value.__traceback__ is not tb:
--> 769     raise value.with_traceback(tb)
    770 raise value

File ~/.conda/envs/torch2.0.0/lib/python3.9/site-packages/urllib3/connectionpool.py:714, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    713 # Make the request on the httplib connection object.
--> 714 httplib_response = self._make_request(
    715     conn,
    716     method,
    717     url,
    718     timeout=timeout_obj,
    719     body=body,
    720     headers=headers,
    721     chunked=chunked,
    722 )
    724 # If we're going to release the connection in ``finally:``, then
    725 # the response doesn't need to know about the connection. Otherwise
    726 # it will also try to release it and we'll have a double-release
    727 # mess.

File ~/.conda/envs/torch2.0.0/lib/python3.9/site-packages/urllib3/connectionpool.py:466, in HTTPConnectionPool._make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    462         except BaseException as e:
    463             # Remove the TypeError from the exception chain in
    464             # Python 3 (including for exceptions like SystemExit).
    465             # Otherwise it looks like a bug in the code.
--> 466             six.raise_from(e, None)
    467 except (SocketTimeout, BaseSSLError, SocketError) as e:

File <string>:3, in raise_from(value, from_value)

File ~/.conda/envs/torch2.0.0/lib/python3.9/site-packages/urllib3/connectionpool.py:461, in HTTPConnectionPool._make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    460 try:
--> 461     httplib_response = conn.getresponse()
    462 except BaseException as e:
    463     # Remove the TypeError from the exception chain in
    464     # Python 3 (including for exceptions like SystemExit).
    465     # Otherwise it looks like a bug in the code.

File ~/.conda/envs/torch2.0.0/lib/python3.9/http/client.py:1377, in HTTPConnection.getresponse(self)
   1376 try:
-> 1377     response.begin()
   1378 except ConnectionError:

File ~/.conda/envs/torch2.0.0/lib/python3.9/http/client.py:320, in HTTPResponse.begin(self)
    319 while True:
--> 320     version, status, reason = self._read_status()
    321     if status != CONTINUE:

File ~/.conda/envs/torch2.0.0/lib/python3.9/http/client.py:289, in HTTPResponse._read_status(self)
    286 if not line:
    287     # Presumably, the server closed the connection before
    288     # sending a valid response.
--> 289     raise RemoteDisconnected("Remote end closed connection without"
    290                              " response")
    291 try:

ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

ConnectionError                           Traceback (most recent call last)
Cell In[14], line 4
      1 # run the association algorithm!
      2 # this first checks the GHOST database for a SN by name, then by coordinates, and
      3 # if we have no match then it manually associates them.
----> 4 hosts = getTransientHosts(snName, snCoord, snClass, verbose=True, starcut='gentle', ascentMatch=True)

File ~/.conda/envs/torch2.0.0/lib/python3.9/site-packages/astro_ghost/ghostHelperFunctions.py:633, in getTransientHosts(transientName, snCoord, snClass, verbose, starcut, ascentMatch, px, savepath, GHOSTpath, redo_search)
    630 snCoord_remaining = df_transients_remaining['snCoord'].values
    631 snClass_remaining = df_transients_remaining['snClass'].values
--> 633 tempHost3 = findNewHosts(transientName_remaining, snCoord_remaining, snClass_remaining, verbose, starcut, ascentMatch, px, savepath)
    635 if (len(transientName_remaining) > 0) and (len(tempHost3)==0) and (redo_search):
    636      #bump up the search radius to 150 arcsec for extremely low-redshift hosts...
    637      if verbose:

File ~/.conda/envs/torch2.0.0/lib/python3.9/site-packages/astro_ghost/ghostHelperFunctions.py:708, in findNewHosts(transientName, snCoord, snClass, verbose, starcut, ascentMatch, px, savepath, rad)
    706 #new low-z method (beta) - before we do anything else, find and associate with GLADE
    707 fn_glade = "gladeDLR.txt"
--> 708 foundGladeHosts, noGladeHosts = chooseByGladeDLR(path, fn_glade, snDF, todo="r")
    710 #open transients df and drop the transients already found in GLADE. We'll add these back in at the end
    711 snDF = snDF[snDF['Name'].isin(noGladeHosts)]

File ~/.conda/envs/torch2.0.0/lib/python3.9/site-packages/astro_ghost/DLR.py:417, in chooseByGladeDLR(path, fn, snDF, todo)
    414 class_SN = str(row['Obj. Type'])
    416 #query the glade catalog
--> 417 result = Vizier.query_region(SkyCoord(ra=ra_SN, dec=dec_SN,unit=(u.deg, u.deg),frame='icrs'),width="1d",catalog=["VII/275/glade1"])
    418 hosts = result[0].to_pandas()
    419 hosts.dropna(subset=['a_b', 'maj', 'min'], inplace=True)

File ~/.local/lib/python3.9/site-packages/astroquery/utils/class_or_instance.py:25, in class_or_instance.__get__.<locals>.f(*args, **kwds)
     23 def f(*args, **kwds):
     24     if obj is not None:
---> 25         return self.fn(obj, *args, **kwds)
     26     else:
     27         return self.fn(cls, *args, **kwds)

File ~/.local/lib/python3.9/site-packages/astroquery/utils/process_asyncs.py:26, in async_to_sync.<locals>.create_method.<locals>.newmethod(self, *args, **kwargs)
     22 @class_or_instance
     23 def newmethod(self, *args, **kwargs):
     24     verbose = kwargs.pop('verbose', False)
---> 26     response = getattr(self, async_method_name)(*args, **kwargs)
     27     if kwargs.get('get_query_payload') or kwargs.get('field_help'):
     28         return response

File ~/.local/lib/python3.9/site-packages/astroquery/vizier/core.py:473, in VizierClass.query_region_async(self, coordinates, radius, inner_radius, width, height, catalog, get_query_payload, cache, return_type, column_filters, frame)
    470 if get_query_payload:
    471     return data_payload
--> 473 response = self._request(
    474     method='POST', url=self._server_to_url(return_type=return_type),
    475     data=data_payload, timeout=self.TIMEOUT, cache=cache)
    476 return response

File ~/.local/lib/python3.9/site-packages/astroquery/query.py:317, in BaseQuery._request(self, method, url, params, data, headers, files, save, savedir, timeout, cache, stream, auth, continuation, verify, allow_redirects, json, return_response_on_save)
    315     response = query.from_cache(self.cache_location)
    316     if not response:
--> 317         response = query.request(self._session,
    318                                  self.cache_location,
    319                                  stream=stream,
    320                                  auth=auth,
    321                                  allow_redirects=allow_redirects,
    322                                  verify=verify,
    323                                  json=json)
    324         to_cache(response, query.request_file(self.cache_location))
    325 self._last_query = query

File ~/.local/lib/python3.9/site-packages/astroquery/query.py:71, in AstroQuery.request(self, session, cache_location, stream, auth, verify, allow_redirects, json)
     68 def request(self, session, cache_location=None, stream=False,
     69             auth=None, verify=True, allow_redirects=True,
     70             json=None):
---> 71     return session.request(self.method, self.url, params=self.params,
     72                            data=self.data, headers=self.headers,
     73                            files=self.files, timeout=self.timeout,
     74                            stream=stream, auth=auth, verify=verify,
     75                            allow_redirects=allow_redirects,
     76                            json=json)

File ~/.conda/envs/torch2.0.0/lib/python3.9/site-packages/requests/sessions.py:587, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    582 send_kwargs = {
    583     "timeout": timeout,
    584     "allow_redirects": allow_redirects,
    585 }
    586 send_kwargs.update(settings)
--> 587 resp = self.send(prep, **send_kwargs)
    589 return resp

File ~/.conda/envs/torch2.0.0/lib/python3.9/site-packages/requests/sessions.py:701, in Session.send(self, request, **kwargs)
    698 start = preferred_clock()
    700 # Send the request
--> 701 r = adapter.send(request, **kwargs)
    703 # Total elapsed time of the request (approximately)
    704 elapsed = preferred_clock() - start

File ~/.conda/envs/torch2.0.0/lib/python3.9/site-packages/requests/adapters.py:547, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    544             raise
    546 except (ProtocolError, OSError) as err:
--> 547     raise ConnectionError(err, request=request)
    549 except MaxRetryError as e:
    550     if isinstance(e.reason, ConnectTimeoutError):
    551         # TODO: Remove this in 3.0.0: see #2811

ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

[alexandergagliano]

@awe2 I've released a new version where I bump up the timeout from 60 to 500 seconds to account for spotty wifi. Please let me know if you're still running into the problem.

[alexandergagliano]

Switching to asynchronous queries to account for lockout after ~330 objects, let me know if this fixes things!

[alexandergagliano]

@awe2 Are you still having this problem?

[awe2]

I am still running into some issues like this, but from what I can gather it has to do with the frequency of pings being made by computer to these catalogue. If they occur too frequently the catalogue servers will block an IP. I used a band-aid solution of time.sleep(1/4) before each query to vizier and simbad, and that seems to be working now.

[alexandergagliano]

@awe2 I've now packaged up a small version of the GLADE catalog with the astro_ghost package so that Vizier is not queried. Could you try re-running for your set of transients with the latest release (github or pip) to verify that it runs without issue?

[awe2]

howdy Alex, i'll be away this weekend but I'll take a peek next week to confirm. Thanks!

On Wed, Oct 25, 2023 at 8:42 PM Alex Gagliano @.***> wrote:

@awe2 https://urldefense.com/v3/__https://github.com/awe2__;!!DZ3fjg!_gwUCkN41pBDSjLojFkFyak19hMiDEIZ96MloO7GvwjJYraOO-M10PvfgB8lO1UbJgPhAsZblEGoiEXlXQEF-s0Uow$ I've now packaged up a small version of the GLADE catalog with the astro_ghost package so that Vizier is not queried. Could you try re-running for your set of transients with the latest release (github or pip) to verify that it runs without issue?

— Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/uiucsn/astro_ghost/issues/17*issuecomment-1780366702__;Iw!!DZ3fjg!_gwUCkN41pBDSjLojFkFyak19hMiDEIZ96MloO7GvwjJYraOO-M10PvfgB8lO1UbJgPhAsZblEGoiEXlXQFkZ9gr5g$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AJL6A6KTOFF5QOVYM7JBZQTYBHLZFAVCNFSM6AAAAAAZJY4UI6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBQGM3DMNZQGI__;!!DZ3fjg!_gwUCkN41pBDSjLojFkFyak19hMiDEIZ96MloO7GvwjJYraOO-M10PvfgB8lO1UbJgPhAsZblEGoiEXlXQE1uubnYw$ . You are receiving this because you were mentioned.Message ID: @.***>

[alexandergagliano]

@awe2 Any luck?