search

sot / chandra_aca

Chandra Aspect Camera Tools
https://sot.github.io/chandra_aca
BSD 2-Clause "Simplified" License
0 stars 0 forks source link

Use a public copy of zero offset aimpoint file if local not available #132

Closed jeanconn closed 1 year ago

jeanconn commented 1 year ago

Description

Use public copy of file if local not available

Fixes #125 (doesn't add table to ska data but makes it accessible)

Interface impacts

Sets the code to use a public version of the file which is a benign change with no impact for the user except they will be able to run the code without access to ICXC.

Testing

Unit tests

Independent check of unit tests by [REVIEWER NAME]

Functional tests

On laptop without VPN and the code in master:

In [1]: from chandra_aca.drift import get_default_zero_offset_table

In [2]: get_default_zero_offset_table()
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
~/git/chandra_aca/chandra_aca/drift.py in get_default_zero_offset_table()
    196     try:
--> 197         CACHE['ZERO_OFFSET_TABLE'] = Table.read(
    198             '/data/mpcrit1/aimpoint_table/zero_offset_aimpoints.txt',

~/miniconda3/envs/ska3/lib/python3.8/site-packages/astropy/table/connect.py in __call__(self, *args, **kwargs)
     61 
---> 62         out = self.registry.read(cls, *args, **kwargs)
     63 

~/miniconda3/envs/ska3/lib/python3.8/site-packages/astropy/io/registry/core.py in read(self, cls, format, cache, *args, **kwargs)
    198             reader = self.get_reader(format, cls)
--> 199             data = reader(*args, **kwargs)
    200 

~/miniconda3/envs/ska3/lib/python3.8/site-packages/astropy/io/ascii/connect.py in io_read(format, filename, **kwargs)
     17         kwargs['format'] = format
---> 18     return read(filename, **kwargs)
     19 

~/miniconda3/envs/ska3/lib/python3.8/site-packages/astropy/io/ascii/ui.py in read(table, guess, **kwargs)
    340         # more useful traceback.
--> 341         dat = _guess(table, new_kwargs, format, fast_reader)
    342         if dat is None:

~/miniconda3/envs/ska3/lib/python3.8/site-packages/astropy/io/ascii/ui.py in _guess(table, read_kwargs, format, fast_reader)
    505             reader.guessing = True
--> 506             dat = reader.read(table)
    507             _read_trace.append({'kwargs': copy.deepcopy(guess_kwargs),

~/miniconda3/envs/ska3/lib/python3.8/site-packages/astropy/io/ascii/core.py in read(self, table)
   1333         # Get a list of the lines (rows) in the table
-> 1334         self.lines = self.inputter.get_lines(table, newline=newline)
   1335 

~/miniconda3/envs/ska3/lib/python3.8/site-packages/astropy/io/ascii/core.py in get_lines(self, table, newline)
    328                     or ('\n' not in table + '' and '\r' not in table + '')):
--> 329                 with get_readable_fileobj(table,
    330                                           encoding=self.encoding) as fileobj:

~/miniconda3/envs/ska3/lib/python3.8/contextlib.py in __enter__(self)
    112         try:
--> 113             return next(self.gen)
    114         except StopIteration:

~/miniconda3/envs/ska3/lib/python3.8/site-packages/astropy/utils/data.py in get_readable_fileobj(name_or_obj, encoding, cache, show_progress, remote_timeout, sources, http_headers)
    270                 http_headers=http_headers)
--> 271         fileobj = io.FileIO(name_or_obj, 'r')
    272         if is_url and not cache:

FileNotFoundError: [Errno 2] No such file or directory: '/data/mpcrit1/aimpoint_table/zero_offset_aimpoints.txt'

During handling of the above exception, another exception occurred:

URLError                                  Traceback (most recent call last)
<ipython-input-2-9fe1eab9661d> in <module>
----> 1 get_default_zero_offset_table()

~/git/chandra_aca/chandra_aca/drift.py in get_default_zero_offset_table()
    199             format='ascii')
    200     except FileNotFoundError:
--> 201         CACHE['ZERO_OFFSET_TABLE'] = Table.read(
    202             "https://icxc.harvard.edu/mp/html/aimpoint_table/zero_offset_aimpoints.txt",
    203             format='ascii')

~/miniconda3/envs/ska3/lib/python3.8/site-packages/astropy/table/connect.py in __call__(self, *args, **kwargs)
     60         descriptions = kwargs.pop('descriptions', None)
     61 
---> 62         out = self.registry.read(cls, *args, **kwargs)
     63 
     64         # For some readers (e.g., ascii.ecsv), the returned `out` class is not

~/miniconda3/envs/ska3/lib/python3.8/site-packages/astropy/io/registry/core.py in read(self, cls, format, cache, *args, **kwargs)
    197 
    198             reader = self.get_reader(format, cls)
--> 199             data = reader(*args, **kwargs)
    200 
    201             if not isinstance(data, cls):

~/miniconda3/envs/ska3/lib/python3.8/site-packages/astropy/io/ascii/connect.py in io_read(format, filename, **kwargs)
     16         format = re.sub(r'^ascii\.', '', format)
     17         kwargs['format'] = format
---> 18     return read(filename, **kwargs)
     19 
     20 

~/miniconda3/envs/ska3/lib/python3.8/site-packages/astropy/io/ascii/ui.py in read(table, guess, **kwargs)
    339         # through below to the non-guess way so that any problems result in a
    340         # more useful traceback.
--> 341         dat = _guess(table, new_kwargs, format, fast_reader)
    342         if dat is None:
    343             guess = False

~/miniconda3/envs/ska3/lib/python3.8/site-packages/astropy/io/ascii/ui.py in _guess(table, read_kwargs, format, fast_reader)
    504 
    505             reader.guessing = True
--> 506             dat = reader.read(table)
    507             _read_trace.append({'kwargs': copy.deepcopy(guess_kwargs),
    508                                 'Reader': reader.__class__,

~/miniconda3/envs/ska3/lib/python3.8/site-packages/astropy/io/ascii/core.py in read(self, table)
   1332 
   1333         # Get a list of the lines (rows) in the table
-> 1334         self.lines = self.inputter.get_lines(table, newline=newline)
   1335 
   1336         # Set self.data.data_lines to a slice of lines contain the data rows

~/miniconda3/envs/ska3/lib/python3.8/site-packages/astropy/io/ascii/core.py in get_lines(self, table, newline)
    327             if (hasattr(table, 'read')
    328                     or ('\n' not in table + '' and '\r' not in table + '')):
--> 329                 with get_readable_fileobj(table,
    330                                           encoding=self.encoding) as fileobj:
    331                     table = fileobj.read()

~/miniconda3/envs/ska3/lib/python3.8/contextlib.py in __enter__(self)
    111         del self.args, self.kwds, self.func
    112         try:
--> 113             return next(self.gen)
    114         except StopIteration:
    115             raise RuntimeError("generator didn't yield") from None

~/miniconda3/envs/ska3/lib/python3.8/site-packages/astropy/utils/data.py in get_readable_fileobj(name_or_obj, encoding, cache, show_progress, remote_timeout, sources, http_headers)
    265         is_url = _is_url(name_or_obj)
    266         if is_url:
--> 267             name_or_obj = download_file(
    268                 name_or_obj, cache=cache, show_progress=show_progress,
    269                 timeout=remote_timeout, sources=sources,

~/miniconda3/envs/ska3/lib/python3.8/site-packages/astropy/utils/data.py in download_file(remote_url, cache, show_progress, timeout, sources, pkgname, http_headers, ssl_context, allow_insecure)
   1394                 f"included as a valid source.")
   1395         elif len(sources) == 1:
-> 1396             raise errors[sources[0]]
   1397         else:
   1398             raise urllib.error.URLError(

~/miniconda3/envs/ska3/lib/python3.8/site-packages/astropy/utils/data.py in download_file(remote_url, cache, show_progress, timeout, sources, pkgname, http_headers, ssl_context, allow_insecure)
   1357     for source_url in sources:
   1358         try:
-> 1359             f_name = _download_file_from_source(
   1360                     source_url,
   1361                     timeout=timeout,

~/miniconda3/envs/ska3/lib/python3.8/site-packages/astropy/utils/data.py in _download_file_from_source(source_url, show_progress, timeout, remote_url, cache, pkgname, http_headers, ftp_tls, ssl_context, allow_insecure)
   1161                 raise
   1162 
-> 1163     with _try_url_open(source_url, timeout=timeout, http_headers=http_headers,
   1164                        ftp_tls=ftp_tls, ssl_context=ssl_context,
   1165                        allow_insecure=allow_insecure) as remote:

~/miniconda3/envs/ska3/lib/python3.8/site-packages/astropy/utils/data.py in _try_url_open(source_url, timeout, http_headers, ftp_tls, ssl_context, allow_insecure)
   1098 
   1099     try:
-> 1100         return urlopener.open(req, timeout=timeout)
   1101     except urllib.error.URLError as exc:
   1102         reason = exc.reason

~/miniconda3/envs/ska3/lib/python3.8/urllib/request.py in open(self, fullurl, data, timeout)
    523 
    524         sys.audit('urllib.Request', req.full_url, req.data, req.headers, req.get_method())
--> 525         response = self._open(req, data)
    526 
    527         # post-process response

~/miniconda3/envs/ska3/lib/python3.8/urllib/request.py in _open(self, req, data)
    540 
    541         protocol = req.type
--> 542         result = self._call_chain(self.handle_open, protocol, protocol +
    543                                   '_open', req)
    544         if result:

~/miniconda3/envs/ska3/lib/python3.8/urllib/request.py in _call_chain(self, chain, kind, meth_name, *args)
    500         for handler in handlers:
    501             func = getattr(handler, meth_name)
--> 502             result = func(*args)
    503             if result is not None:
    504                 return result

~/miniconda3/envs/ska3/lib/python3.8/urllib/request.py in https_open(self, req)
   1395 
   1396         def https_open(self, req):
-> 1397             return self.do_open(http.client.HTTPSConnection, req,
   1398                 context=self._context, check_hostname=self._check_hostname)
   1399 

~/miniconda3/envs/ska3/lib/python3.8/urllib/request.py in do_open(self, http_class, req, **http_conn_args)
   1355                           encode_chunked=req.has_header('Transfer-encoding'))
   1356             except OSError as err: # timeout error
-> 1357                 raise URLError(err)
   1358             r = h.getresponse()
   1359         except:

URLError: <urlopen error timed out>

vs run with code in this PR

In [1]: from chandra_aca.drift import get_default_zero_offset_table

In [2]: get_default_zero_offset_table()
Downloading https://cxc.harvard.edu/mta/ASPECT/drift/zero_offset_aimpoints.txt
|=============================================================================| 2.6k/2.6k (100.00%)         0s
Out[2]: 
<Table length=34>
date_effective cycle_effective detector  chipx   chipy  chip_id obsvis_cal
    str10           int64        str6   float64 float64  int64     str4   
-------------- --------------- -------- ------- ------- ------- ----------
    2012-12-15              15   ACIS-I   941.0   988.0       3        1.6
    2012-12-15              15   ACIS-S   224.0   490.0       7        1.6
    2012-12-15              15    HRC-I  7615.0  7862.0       0        1.6
    2012-12-15              15    HRC-S  2075.0  8984.0       2        1.6
    2013-12-15              16   ACIS-I   932.0  1009.0       3        1.7
    2013-12-15              16   ACIS-S   205.0   480.0       7        1.7
    2013-12-15              16    HRC-I  7606.0  7941.0       0        1.7
    2013-12-15              16    HRC-S  2050.0  9023.0       2        1.7
           ...             ...      ...     ...     ...     ...        ...
    2018-11-07              19   ACIS-S   191.6   511.2       7       1.10
    2018-11-07              20   ACIS-S   191.6   511.2       7       1.11
    2018-11-16              17   ACIS-S   200.7   476.9       7       1.8a
    2018-11-16              18   ACIS-S   210.0   520.0       7        1.9
    2018-11-16              19   ACIS-S   210.0   520.0       7       1.10
    2018-11-16              20   ACIS-S   210.0   520.0       7       1.11
    2022-09-19              24   ACIS-I  953.74  958.74       3       1.16
    2022-09-19              24   ACIS-S  193.74   520.0       7       1.16
taldcroft commented 1 year ago

Since we may actually use this now, the progress bar when downloading the web file is not desirable, so I fixed that. And once I got into the code I couldn't help but modernizing it.

jeanconn commented 1 year ago

OK. I removed the CACHE still hanging around (though I suppose that is also an API change but I don't think anything was going to be using chandra_aca.drift.CACHE). Looks like flake8 is failing on the new Path so I suppose that needs new exception to the checks?