ELT should run automatically when user schedules their first pipeline

[MeltyBot]

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/795

Originally created by @dmor on 2019-07-15 20:53:03

---

I originally thought it was a bug, but @mbergeron has confirmed this has not been implemented yet. The problem is that our UI does not tell the user they need to install Airflow... it just sends them to Analyze, and then they get the error messages in the repro steps below.

---

Repro Steps:

1. Install most recent version of Meltano (0.32) and create a new Meltano instance
2. Install tap Carbon Emissions through the UI, and select "All" entities
3. Install target Snowflake through the UI and set fields as so (password is in 1Password):
4. Configure first pipeline like so:
5. Go to Analyze and install "model-carbon-intensity" via the UI
6. Click on "Analyze" next to Region
7. From http://localhost:5000/analyze/carbon/region select "ID" from Columns and observe you receive a red error message in the bottom right

Output from the Command Line:

[ Thread-99][INFO][2019-07-15 14:51:30,107] 127.0.0.1 - - [15/Jul/2019 14:51:30] "POST /api/v1/sql/get/carbon/region HTTP/1.1" 500 - Traceback (most recent call last): File "/Users/daniellemorrill/Virtual/Meltano/lib/python3.7/site-packages/flask/app.py", line 2328, in call return self.wsgi_app(environ, start_response) File "/Users/daniellemorrill/Virtual/Meltano/lib/python3.7/site-packages/flask/app.py", line 2314, in wsgi_app response = self.handle_exception(e) File "/Users/daniellemorrill/Virtual/Meltano/lib/python3.7/site-packages/flask_restful/init.py", line 269, in error_router return original_handler(e) File "/Users/daniellemorrill/Virtual/Meltano/lib/python3.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function return cors_after_request(app.make_response(f(*args, *kwargs))) File "/Users/daniellemorrill/Virtual/Meltano/lib/python3.7/site-packages/flask/app.py", line 1760, in handle_exception reraise(exc_type, exc_value, tb) File "/Users/daniellemorrill/Virtual/Meltano/lib/python3.7/site-packages/flask/_compat.py", line 36, in reraise raise value File "/Users/daniellemorrill/Virtual/Meltano/lib/python3.7/site-packages/flask/app.py", line 2311, in wsgi_app response = self.full_dispatch_request() File "/Users/daniellemorrill/Virtual/Meltano/lib/python3.7/site-packages/flask/app.py", line 1834, in full_dispatch_request rv = self.handle_user_exception(e) File "/Users/daniellemorrill/Virtual/Meltano/lib/python3.7/site-packages/flask_restful/init.py", line 269, in error_router return original_handler(e) File "/Users/daniellemorrill/Virtual/Meltano/lib/python3.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function return cors_after_request(app.make_response(f(args, kwargs))) File "/Users/daniellemorrill/Virtual/Meltano/lib/python3.7/site-packages/flask/app.py", line 1737, in handle_user_exception reraise(exc_type, exc_value, tb) File "/Users/daniellemorrill/Virtual/Meltano/lib/python3.7/site-packages/flask/_compat.py", line 36, in reraise raise value File "/Users/daniellemorrill/Virtual/Meltano/lib/python3.7/site-packages/flask/app.py", line 1832, in full_dispatch_request rv = self.dispatch_request() File "/Users/daniellemorrill/Virtual/Meltano/lib/python3.7/site-packages/flask/app.py", line 1818, in dispatch_request return self.view_functions[rule.endpoint](req.view_args) File "/Users/daniellemorrill/Virtual/Meltano/lib/python3.7/site-packages/meltano/api/controllers/sql.py", line 95, in get_sql sql_dict = sqlHelper.get_sql(design, incoming_json) File "/Users/daniellemorrill/Virtual/Meltano/lib/python3.7/site-packages/meltano/core/sql/sql_utils.py", line 8, in get_sql definition=incoming_json, design_helper=design, schema=schema File "/Users/daniellemorrill/Virtual/Meltano/lib/python3.7/site-packages/meltano/core/sql/base.py", line 596, in init self.parse_definition(definition) File "/Users/daniellemorrill/Virtual/Meltano/lib/python3.7/site-packages/meltano/core/sql/base.py", line 691, in parse_definition design_graph, primary_table["name"], table.name File "/Users/daniellemorrill/Virtual/Meltano/lib/python3.7/site-packages/meltano/core/sql/base.py", line 739, in joins_for_table return nx.shortest_path(design_graph, source=source_table, target=target_table) File "/Users/daniellemorrill/Virtual/Meltano/lib/python3.7/site-packages/networkx/algorithms/shortest_paths/generic.py", line 170, in shortest_path paths = nx.bidirectional_shortest_path(G, source, target) File "/Users/daniellemorrill/Virtual/Meltano/lib/python3.7/site-packages/networkx/algorithms/shortest_paths/unweighted.py", line 223, in bidirectional_shortest_path raise nx.NodeNotFound(msg.format(source, target)) networkx.exception.NodeNotFound: Either source region or target generationmix is not in G

Expected behaviors:

• Useful error message it the UI
• Some fields will work (got the same error in the CLI for everything I selected)
• If this combination of tap and target is not supported, don't allow users to set it up

Possible further debugging -- did the ELT actually run?

When I select "Region Name" in Analyze I don't get an error, I get this query:

When I select "Run Query" I get the following error message:

Looks like the table never was created. The UI made it look like I did everything needed in order to successfully start analyzing my data... if that is not the case, @derek-knox we should connect with @mbergeron on the flow.

Confirming Airflow is not installed:

I am not clear on whether the ELT should have run when I selected @once, without needing Airflow, but there was never any instructions in the UI to go through the Airflow install process.

[MeltyBot]

View 4 previous comments from the original issue on GitLab