Explore github tap / tap development best practices

[machow]

from pairing w/ @soryrawyer

• our tap repo: https://github.com/codeforphilly/tap-github
• I'll send our @codeforphilly-bot token
• local testing of tap
• note that sometimes taps succeed on initial run, but fail on re-run
• (bonus) github action to test tap. I'm happy to pair around using gh actions, etc..

This is a tangent, but we're also interested in ingesting google analytics:

• TODO: what questions do we want to answer using GA?

[machow]

@soryrawyer I did some exploration of our taps again, and wanted to send a quick walkthrough I made of everything I wished I knew about taps and meltano beforehand:

It seems like the main problems right now are:

problems

• a tap might run for 15 minutes before failing, leaving little to debug with.
  • it could fail at the tap or the target level.
  • for example, if you have tap-github start pulling commits, by changing this line of yml to be just commits.*, I think our github-to-snowflake will start failing.
• when we run the slack tap, it keeps snowflake running for 2 hours while we download slack data.

possible solution

• could we capture tap output, and then re-play it into targets?