Refactor Data Services (Broker, Catalog, etc.)

[jchate6]

The TOM Toolkit offers many different ways to import data from external sources. For this issue, we want to re-evaluate how we handle any kind of data import that could be considered an external catalog search. Currently these are broken into several different categories, each of which serve different but overlapping purposes and use drastically different vocabulary.

Current Data Services

• Alert Brokers
  • Use "Broker" apps (some built in, some external)
  • Creates Queries that are saved to be run at a later date
  • intended to be run on updating Catalogs
  • ingests NEW targets into the TOM.
  • Can bring along data if available
  • Cannot update existing targets
• Catalogs
  • use "Harvesters" to ingest data
  • intended to get single target matching a name query.
  • Does not handle target data
  • Targets not updated (updated orbital elements, improved distances, etc.)
• Forced Photometry
  • intended to pull data from from a catalog's forced photometry service.
  • results in a csv of data to be imported as a Data Product for a single target.
• AlertStreams
  • uses tom-alertstreams
  • uses "Topic Handlers" to ingest data
  • intended to listen to a kafka stream and import data into TOM
  • Topic Handlers are flexible, and can import targets, individual Reduced Datums, or do anything else with an incoming alert.

The next effort is to add Data Searches that import data directly from a catalog for a given target or targets. At a fundamental level, these new data searches, "brokers", and "catalogs" all query an external catalog and ingest the resulting data into the TOM.

This issue is to explore the possibility of consolidating these different data services into a unified schema with related vocabulary and less redundant code.

[jchate6]

Write up for Refactor

[jchate6]

See Issue #894