New tools in utility/DP02 for converting DP02 schema (yaml) to AVSC, for pulling DP02 from the RSP, converting to AVRO, and pushing into kafka.
New schema in common/schema with downstream changes for schema browser, for CQL tables
pipeline/ingest.py modified to take new DP02 style data packets. In particular no longer diaSource+[prvSources], rather just [diaSourceList]. Also no diaNonDetections. Also diaForcedSources becomes forcedSourceOnDiaObjects.
Modified pipeline/filter/features to take new name for the forced sources.
Deploy is changed so the settings.SCHEMA is the dp02.avsc
New tools in
utility/DP02for converting DP02 schema (yaml) to AVSC, for pulling DP02 from the RSP, converting to AVRO, and pushing into kafka.New schema in
common/schemawith downstream changes for schema browser, for CQL tablespipeline/ingest.pymodified to take new DP02 style data packets. In particular no longerdiaSource+[prvSources], rather just[diaSourceList]. Also nodiaNonDetections. AlsodiaForcedSourcesbecomesforcedSourceOnDiaObjects.Modified
pipeline/filter/featuresto take new name for the forced sources.Deploy is changed so the
settings.SCHEMAis thedp02.avsc