Working group for ongoing development and iteration of the Singer Spec, the de-facto protocol for open source data connectors. Please use "Issues" to create discussion items - or use "Discussions" for general questions.
Apache License 2.0
13
stars
4
forks
source link
Taxonomy of optional capabilities for taps and targets #5
discover - the ability to print a catalog.json file with all known streams and their schemas (as described in existing Singer Spec)
catalog (newer) and properties (previous iteration) - the ability to pass in a catalog.json file with selection logic (as described in existing Singer Spec)
log-based the ability for database-type sources to read from logs (as described in existing Singer Spec)
For targets only:
soft-delete - The ability to flag records as deleted, based on a deleted_at property or some other signal.
hard-delete - The ability to permanently delete records, based on a deleted_at property or some other signal.
datatype-failsafe - The promise that records will be written to the target when the data type is not in an expected/handled set of cases and/or when the target cannot parse the JSON Schema type (as proposed in #20).
For both taps and targets:
about - prints out capabilities and settings (as proposed in #8)
activate-version support (currently unofficial but widely used, as described in #9)
batch support (future capability, as proposed in #2)
(Partially enumerated in: https://gitlab.com/meltano/meltano/-/issues/2829 and in related SDK MR: https://gitlab.com/meltano/sdk/-/merge_requests/141/diffs#1a46f5bc80ac2afe420e39f5eb5c82cb4d690483_0_112)
For taps only:
discover- the ability to print acatalog.jsonfile with all known streams and their schemas (as described in existing Singer Spec)catalog(newer) andproperties(previous iteration) - the ability to pass in acatalog.jsonfile with selection logic (as described in existing Singer Spec)log-basedthe ability for database-type sources to read from logs (as described in existing Singer Spec)For targets only:
soft-delete- The ability to flag records as deleted, based on adeleted_atproperty or some other signal.hard-delete- The ability to permanently delete records, based on adeleted_atproperty or some other signal.datatype-failsafe- The promise that records will be written to the target when the data type is not in an expected/handled set of cases and/or when the target cannot parse the JSON Schema type (as proposed in #20).For both taps and targets:
about- prints out capabilities and settings (as proposed in #8)activate-versionsupport (currently unofficial but widely used, as described in #9)batchsupport (future capability, as proposed in #2)Related to: https://github.com/MeltanoLabs/singer-spec-working-group/issues/4