search

TOMToolkit / tom_base

The base Django project for a Target and Observation Manager
https://tom-toolkit.readthedocs.io
GNU General Public License v3.0
26 stars 45 forks source link

Add catalog and image cutout query tool for the Rubin Science Platform #680

Open rachel3834 opened 1 year ago

rachel3834 commented 1 year ago

This epic will be broken into 2 main sections:

Using this thread to share my notes from the team's meeting with Leanne Guy at the Rubin Project and Community Workshop, 2023 in Tucson, AZ:

https://data.lsst.cloud/ - RSP URL Github login or ORCID most common CILogon used for OAuth workflow, handles data rights.

TOM permissions must have at least one person with data rights. They will need to be an admin user.

https://ds0-2.lsst.io/ - documentation for user login

https://dp0-2.lsst.io/data-access-analysis-tools/nb-intro.html

Documentation on configuring security tokens for external applications: https://nb.lsst.io/environment/tokens.html

RSP platform -> User profile -> create token -> Store token in TOM/Topcat etc.

Set an environment variable with the access token, and one with the external TAP URL.

Once configured, can use LSST software within application, e.g. Python

This requires pip install lsst-rsp package.

Documentation describing import and use of service from Python.

Service URL is https://data.lsst.cloud/api/tap Also one for image cutouts

Need to warn users not that high traffic calls may be throttled. Include this in documentation. Consider bulk-queries at the institute level.

rachel3834 commented 1 year ago

Following these instructions on setting up a token and using it through a Python 3.10 venv under Mac OS, I get the following:

` In [1]: from lsst.rsp import get_tap_service, retrieve_query

In [2]: service = get_tap_service()

:1: DeprecationWarning: get_tap_service() is deprecated, use get_tap_service("tap") service = get_tap_service() In [3]: query = "SELECT * FROM tap_schema.schemas" In [4]: results = service.search(query).to_table() --------------------------------------------------------------------------- E19 Traceback (most recent call last) File ~/software/lsst_venv/lib/python3.10/site-packages/pyvo/dal/query.py:241, in DALQuery.execute_votable(self, post) 240 try: --> 241 return votableparse(self.execute_stream(post=post).read) 242 except Exception as e: File ~/software/lsst_venv/lib/python3.10/site-packages/astropy/utils/decorators.py:604, in deprecated_renamed_argument..decorator..wrapper(*args, **kwargs) 602 warnings.warn(msg, warning_type, stacklevel=2) --> 604 return function(*args, **kwargs) File ~/software/lsst_venv/lib/python3.10/site-packages/astropy/io/votable/table.py:177, in parse(source, columns, invalid, verify, chunk_size, table_number, table_id, filename, unit_format, datatype_mapping, _debug_python_based_parser) 174 with iterparser.get_xml_iterator( 175 source, _debug_python_based_parser=_debug_python_based_parser 176 ) as iterator: --> 177 return tree.VOTableFile(config=config, pos=(1, 1)).parse(iterator, config) File ~/software/lsst_venv/lib/python3.10/site-packages/astropy/io/votable/tree.py:3884, in VOTableFile.parse(self, iterator, config) 3883 else: -> 3884 vo_raise(E19, (), config, pos) 3885 config.update(self._get_version_checks()) File ~/software/lsst_venv/lib/python3.10/site-packages/astropy/io/votable/exceptions.py:124, in vo_raise(exception_class, args, config, pos) 123 config = {} --> 124 raise exception_class(args, config, pos) E19: None:1:0: E19: File does not appear to be a VOTABLE During handling of the above exception, another exception occurred: DALServiceError Traceback (most recent call last) Cell In[4], line 1 ----> 1 results = service.search(query).to_table() File ~/software/lsst_venv/lib/python3.10/site-packages/pyvo/dal/tap.py:257, in TAPService.run_sync(self, query, language, maxrec, uploads, **keywords) 228 def run_sync( 229 self, query, language="ADQL", maxrec=None, uploads=None, 230 **keywords): 231 """ 232 runs sync query and returns its result 233 (...) 253 TAPResults 254 """ 255 return self.create_query( 256 query, language=language, maxrec=maxrec, uploads=uploads, --> 257 **keywords).execute() File ~/software/lsst_venv/lib/python3.10/site-packages/pyvo/dal/tap.py:1076, in TAPQuery.execute(self) 1062 def execute(self): 1063 """ 1064 submit the query and return the results as a TAPResults instance 1065 (...) 1074 for errors parsing the VOTable response 1075 """ -> 1076 return TAPResults(self.execute_votable(), url=self.queryurl, session=self._session) File ~/software/lsst_venv/lib/python3.10/site-packages/pyvo/dal/query.py:243, in DALQuery.execute_votable(self, post) 241 return votableparse(self.execute_stream(post=post).read) 242 except Exception as e: --> 243 self.raise_if_error() 244 raise DALFormatError(e, self.queryurl) File ~/software/lsst_venv/lib/python3.10/site-packages/pyvo/dal/query.py:252, in DALQuery.raise_if_error(self) 250 if self._ex: 251 e = self._ex --> 252 raise DALServiceError.from_except(e, self.queryurl) DALServiceError: 401 Client Error: Unauthorized for url: https://data.lsst.cloud/api/tap/sync ` I will raise this with Leanne. I created a new token in the RSP with access to the TAP catalogs and images for this.
rachel3834 commented 1 year ago

OK the authentication error was simply due to not having the environment variables named according to the convention required. The two parameters need to be called: EXTERNAL_TAP_URL and ACCESS_TOKEN

rachel3834 commented 4 months ago

Here's a wireframe of a v1.0 implementation of a tool to search the RSP source catalogs as an example.

Image

The documentation includes some notebooks which give good examples of the essential search capabilities that I would expect a TOM to provide, such as this one for searching the point source catalogs.

The workflow that I would expect for this would be:

rachel3834 commented 4 months ago

The LSST Data Products Definition Document gives a full description of the source catalog tables and terminology.

jchate6 commented 4 months ago

This epic will be broken into 2 main sections: