use singer taps and targets programatically

[OriHoch]

Hi, I'm participating in development of dataflows which has similar goals to your projects, and we would like to be able to integrate between the libraries - use singer taps / targets inside a data flow, and use a data flow as a singer tap / target (https://github.com/datahq/dataflows/issues/16)

To enable this integration we need to be able to call singer taps / targets from Python code, this is easy to do using subprocess.Popen, see example here

I think it would really useful to have this in a more standard way as part of the singer-python library.

Example: singer.read_tap

Install the tap: pip install tap-exchangeratesapi

Read from the tap:

>>> tap = singer.read_tap('exchangeratesapi', {"base": "ILS", "start_date": "2018-10-01"})
>>> for message in tap:
>>>     print(message)  # SchemaMessage / RecordMessage / StateMessage

[timvisher]

dataflows looks neat. :)

This is an interesting proposal. We're going to discuss this for a bit internally and I'll get back to you sometime this week with our thoughts.

[timvisher]

Thanks again for submitting this proposal. We discussed it a bit internally and we think it might be worth exploring via a PR. You are more than welcome to submit one.

The biggest concern we had internally is how this interface would handle the subprocess being killed or sent other signals by the OS or what would happen in the case of subprocess death generally. One nice thing about the Popen is that it's searingly obvious that you need to deal with all that. This interface would be nice but it would still need to respond to similar things.

Are you interested in submitting a PR?

[timvisher]

I'm going to close this for now but please feel free to reopen it if you're interested in exploring it via a PR.

[timvisher]

Since we've got that PR now I'll go ahead and reopen.

[timvisher]

I'll close this out for now since we don't have much forward momentum on the PR for now.