Proposed Best Practice: Dry Run Mode

[mdelaurentis]

As a best practice, a Tap or Target should provide a Dry Run mode, where it just verifies that it can connect to its data source or destination using the configuration provided.

Motivation

It would be helpful if a Tap could give the user quick feedback as to whether it can connect to the data source using the configuration provided. Currently if you run a tap with invalid configuration, it will exit with a non-zero status rather quickly. But if you run it with valid credentials, it will start streaming data. Users may find it desireable to have a mode of operation where the Tap just makes a quick attempt to connect to the data source and then exits zero or non-zero in order to indicate success or failure.

Proposed Solution

A Tap or Target should support a -n and --dry-run option. This option indicates that the Tap or Target should just attempt to connect to the data source or destination with the configuration provided. If it can connect, exit 0. If it can't, exit non-zero with a useful error message.

[karstendick]

That should work. There are some cases where we might have a partial failure (e.g. a given credential can access some endpoints/tables but not all), but the error message could convey that information.

[mdelaurentis]

@karstendick and I were just talking about this and we had an interesting idea. Don't modify the Tap at all. Instead, change the program that calls the Tap so that if it wants to check the credentials, it just starts the Tap (without a Target), reads the stdout of the Tap, waits for the first RECORD to show up, then records success. if the Tap exits non-zero, the wrapper program (in our case the orchestrator) would interpret that as failure to connect.

Pros

• This would allow us to avoid putting any more complexity into the Tap, which would be nice.
• It might also provide a more accurate measure of whether the Tap can connect, since producing one record is certainly a sign of being able to connect.

Cons

• It might take a while for the first record to show up, depending on the data source

[mdelaurentis]

We've experimented with this a little internally and I think we feel pretty strongly that it's best not to add dry run mode, and instead have the program that runs the Tap wait for a record to show up.