search

kuwala-io / kuwala

Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data science models and products with a focus on geospatial data. Currently, the following data connectors are available worldwide: a) High-resolution demographics data b) Point of Interests from Open Street Map c) Google Popular Times
https://kuwala.io
Apache License 2.0
788 stars 52 forks source link

🧱 Export blocks and deleting blocks #144

Closed mattigrthr closed 2 years ago

mattigrthr commented 2 years ago

This PR includes a newly added export block catalog with the ability to export results as CSV files.

Also, now, blocks can be deleted from the canvas.

This PR also includes a significant state management refactoring which improved the application's overall stability.

Closes #89

mattigrthr commented 2 years ago

I checked out dev branch and were able to build the front end, backend and make a external database connection to florian_demo (hosted on AWS). I can see the data but the block is failed to create when configuring variables in the dataset on the canvas. I see in the cataloge section now also the export blocks however I was not able to create complete workflow because the dataset were not created on the canvas.

backend | ERROR: Exception in ASGI application backend | Traceback (most recent call last): backend | File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 372, in run_asgi backend | result = await app(self.scope, self.receive, self.send) backend | File "/usr/local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in call backend | return await self.app(scope, receive, send) backend | File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 259, in call backend | await super().call(scope, receive, send) backend | File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 112, in call backend | await self.middleware_stack(scope, receive, send) backend | File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 181, in call backend | raise exc backend | File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in call backend | await self.app(scope, receive, _send) backend | File "/usr/local/lib/python3.9/site-packages/starlette/middleware/cors.py", line 92, in call backend | await self.simple_response(scope, receive, send, request_headers=headers) backend | File "/usr/local/lib/python3.9/site-packages/starlette/middleware/cors.py", line 147, in simple_response backend | await self.app(scope, receive, send) backend | File "/usr/local/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in call backend | raise exc backend | File "/usr/local/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in call backend | await self.app(scope, receive, sender) backend | File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in call backend | raise e backend | File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call backend | await self.app(scope, receive, send) backend | File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 656, in call backend | await route.handle(scope, receive, send) backend | File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 259, in handle backend | await self.app(scope, receive, send) backend | File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 61, in app backend | response = await func(request) backend | File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 227, in app backend | raw_response = await run_endpoint_function( backend | File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 162, in run_endpoint_function backend | return await run_in_threadpool(dependant.call, *values) backend | File "/usr/local/lib/python3.9/site-packages/starlette/concurrency.py", line 39, in run_in_threadpool backend | return await anyio.to_thread.run_sync(func, args) backend | File "/usr/local/lib/python3.9/site-packages/anyio/to_thread.py", line 31, in run_sync backend | return await get_asynclib().run_sync_in_worker_thread( backend | File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread backend | return await future backend | File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 867, in run backend | result = context.run(func, *args) backend | File "/opt/core/backend/app/routers/data_block.py", line 19, in create_data_block backend | data_block_id, model_name = data_block_controller.create_data_block( backend | File "/opt/core/backend/app/controller/data_block_controller.py", line 207, in create_data_block backend | run_dbt_models(dbt_dir=dbt_dir, dbt_model_names=dbt_model_names) backend | File "/opt/core/backend/app/controller/dbt_controller.py", line 67, in run_dbt_models backend | output.stdout.decode("utf8").split("ERROR=")[1].split("SKIP=")[0][:-1] backend | IndexError: list index out of range

That seems to be related to the demo database again. I deleted the dbt_kuwala schema in the database. Last time that fixed the issue. You can also delete that and look at the tables created there through tools like Datagrip or another SQL viewer.

floriankuwala commented 2 years ago

Ready to Merge