Closed tayloramurphy closed 1 year ago
Thanks @tayloramurphy.
@sbalnojan thinking on the "wow" factor a bit more - I wonder if there's some simple CLI visualization tool that we can use along with DuckDB to create a simple chart that outputs to the command line.
I'm thinking we can have the user run 2 commands - a full meltano run
that does the extract load and transformation. And then a python script (also via meltano run) that queries duckdb and outputs a chart based on that data to the command line. (maybe with a tool like this https://github.com/mkaz/termgraph ? that's a random repo I found).
I'm worried that just having them run an end to end meltano run isn't a full enough wow, but that if we can show a cool image in the terminal and then walk them through the pipeline that would be a good how.
@tayloramurphy, I'm all for presenting it in a shiny light, but I also think we've should stick to the E&L focus. Completely. That what has to wow, no fancy graphic on top of it.
My suggestion: let me prepare something first, then we can all play it and see how it feels, and how we could enhance the "wow".
My preliminary idea: Wow on the E&L parts that we already have, just shine a light on them. Enhance the script I currently have to display the results from the database to:
I just tried it and it seems to be working without an CD'ing.
@sbalnojan You're totally right, no need to cd into anything, but then the meltano init
output is incorrect as it says "Next steps: cd new_project"
@sbalnojan yeah - I'm sold there - keep it focused on EL. I do wonder then if it'd be worth having something like:
meltano run <tap> <map> <target> <python script>
where the Python script is doing exactly what you say: output results, new tables, columns etc.
@DouweM good point, maybe I can make the output I have in there anyways bigger and fatter :-D (Like don't cd,go straight to step 2, we already got your covered).
@tayloramurphy, I think that would again distract. the magic is in "meltano run tap map target". That's it.
I think it's fine to have a second step to make it visible. With a line like: "Wait we're already done? That's it??? Yes! The beauty of meltano is that it is soooo simple that you need a second step to make it visible".
@sbalnojan I trust your judgement on this! I think I'm just trying to avoid having the magic be in reading a bunch of log lines.
Douwe shared some feedback given from Sid:
meltano run
and then we can break down what's happening and / or have them do their own project?FYI @DouweM based on the feedback you shared with me.