Open MeltyBot opened 3 years ago
This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen
label, or request that it be added.
Migrated from GitLab: https://gitlab.com/meltano/sdk/-/issues/30
Originally created by @aaronsteers on 2021-03-02 18:49:07
Background
In my effort to support development on
tap-powerbi-metadata
, I'm realizing that I am overly reliant on the tester and co-developer because I don't have access to valid credentials for the source. It is not trivial to create test creds either, since this requires at minimum: creating a new Azure account with a valid credit card, creating an Active Directory Domain, creating a Service Principal, creating a new Power BI tenant, and finally granting my Service Principal access to the Power BI tenant. In fact, there are still more steps on top of the above, since my new Power BI environment probably doesn't have any reports, workspaces, log history, etc. In other words, even after gaining valid creds, I would not necessarily have valid test data to validate the data sync methods are working properly.Proposal:
As a sibling and complement of planned connection tests in #14, we would like to be able to run "as close as possible" to a full sync test without having any access to the upstream connection. This is difficult by nature to make generic, since there isn't a "dummy data set" which would make sense for every stream. We'd also like to create portable and replayable versions of synced data, so that we get better reproducibility - and to do so in a way that "just works", regardless of which tap or source system we are dealing with.
Use cases
SCHEMA
message generation is correct, based on catalog input and/or predefined tap logic.STATE
message emitting behaviors are tested against config inputs.jsonschema
config compliant to the JSON Schema spec and that data within the sample stream complies with the specifiedjsonschema
definition on the stream.singer-db
, we can more realistically scale to hundreds or thousands of taps, performing automated generic CI/CD testing without having to manage hundreds or thousands of corresponding source credentials. (Having sample data capabilities would likely result in a "badge" of some sort in the index, along with the latest CI/CD test results based on that sample data.)Proposed implementation
Internal changes proposed:
Write a new
Stream._replay_records()
into the SDK base classes as an alternative path toget_records()
. This function would never need to be overridden by developers since it would be implemented generically. In order to meet that design goal (i.e. not requiring dev effort), we would require a generic, predefined text file format. The easiest and most generalizable file format is our already-definedjsonl
output from the tap itself.Proposed CLI updates:
--replay=path/to/output.jsonl
capability which would then run in dry-run mode using the sample data. The process of creating a source connection would then be skipped.--catalog
would be required whenever--replay
is set.--replay
option should be--demo
capability which is automatically enabled if the tap developer specifies a path to a valid and replayable demo data set, including a catalog file and at least one jsonl file. When the capability is supported,tap-mysource --demo
is equivalent totap-mysource --replay=path/to/demo/replay-file.json --catalog=path/to/demo/catalog.json
.Why raw
jsonl
sync output as the standard "data replay" format:After considering several options, I landed on native jsonl output as the best storage mechanism I could think of for enabling this functionality across the wide ecosystem of existing taps.
target-csv
generically but experience has shown that CSV doesn't work well for complex and nested data sets. We could considertarget-jsonl
ortarget-parquet
but neither is simpler or offers any significant benefit over simply replaying the raw output data. (See "out of scope" section below for possible future extensibility options.)jsonl
is very easy to review for PII and other confidential information which could then be relatively easily be replaced with obfuscated/generic data. (See "out of scope" for thoughts around auto-obfuscation.)Other Notes:
Caveats:
For this to be valuable and effective for testing purposes, we should run through as much of the "real" data flow as possible:
SCHEMA
andRECORD
message types are properly generated, we would need to treat RECORD messages as "raw data" and not simply echo them.jsonl
output should either be ignored completely or used as test assertions. We would not simply echo them out, since one of the objectives of the test are to ensure that they are correctly generated by the developers implementation.Out-of-scope but worthy of discussion:
_replay_records()
method discussed here as generic.