databricks / dbt-databricks

A dbt adapter for Databricks.
https://databricks.com
Apache License 2.0
224 stars 119 forks source link

Show command with nested json yields "Object of type ndarray is not JSON serializable" #489

Open racheldaniel opened 1 year ago

racheldaniel commented 1 year ago

Describe the bug

dbt-databricks throws Object of type ndarray is not JSON serializable when previewing nested json (dbt show)

Steps To Reproduce

), cast_types as (

select

), explode_series as (

select * , explode(series) from cast_types

) select * from explode_series

### Expected behavior
I expect this to result in previewed code as with other adapters:
filename series obiscode interval rolloverlimit values ...
first json [{"obisCode":"1.8... 1.8.0 00:05:00 999,999.999 [{"value": 0, "re... ...

### Screenshots and log output
Traceback in logs:

{"data": {"exc_info": "Traceback (most recent call last):\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/task/base.py\", line 372, in safe_run\n result = self.compile_and_execute(manifest, ctx)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/task/base.py\", line 323, in compile_and_execute\n result = self.run(ctx.node, manifest)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/task/base.py\", line 419, in run\n return self.execute(compiled_node, manifest)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/task/show.py\", line 38, in execute\n adapter_response, execute_result = self.adapter.execute(\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/databricks/utils.py\", line 72, in wrapper\n return func(*new_args, *new_kwargs)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/databricks/impl.py\", line 150, in execute\n return super().execute(sql=sql, auto_begin=auto_begin, fetch=fetch, limit=limit)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/base/impl.py\", line 290, in execute\n return self.connections.execute(sql=sql, auto_begin=auto_begin, fetch=fetch, limit=limit)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/databricks/connections.py\", line 798, in execute\n table = self.get_result_from_cursor(cursor, limit)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/sql/connections.py\", line 132, in get_result_from_cursor\n return dbt.clients.agate_helper.table_from_data_flat(data, column_names)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/clients/agate_helper.py\", line 127, in table_from_data_flat\n value = json.dumps(value, cls=dbt.utils.JSONEncoder)\n File \"/usr/lib/python3.8/json/init.py\", line 234, in dumps\n return cls(\n File \"/usr/lib/python3.8/json/encoder.py\", line 199, in encode\n chunks = self.iterencode(o, _one_shot=True)\n File \"/usr/lib/python3.8/json/encoder.py\", line 257, in iterencode\n return _iterencode(o, 0)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/utils.py\", line 351, in default\n return super().default(obj)\n File \"/usr/lib/python3.8/json/encoder.py\", line 179, in default\n raise TypeError(f'Object of type {o.class.name} '\nTypeError: Object of type ndarray is not JSON serializable\n"}, "info": {"category": "", "code": "Z011", "extra": {}, "invocation_id": "4d4b3d45-15e5-4f27-8045-3ae4fb04c5b0", "level": "debug", "msg": "Traceback (most recent call last):\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/task/base.py\", line 372, in safe_run\n result = self.compile_and_execute(manifest, ctx)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/task/base.py\", line 323, in compile_and_execute\n result = self.run(ctx.node, manifest)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/task/base.py\", line 419, in run\n return self.execute(compiled_node, manifest)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/task/show.py\", line 38, in execute\n adapter_response, execute_result = self.adapter.execute(\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/databricks/utils.py\", line 72, in wrapper\n return func(new_args, **new_kwargs)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/databricks/impl.py\", line 150, in execute\n return super().execute(sql=sql, auto_begin=auto_begin, fetch=fetch, limit=limit)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/base/impl.py\", line 290, in execute\n return self.connections.execute(sql=sql, auto_begin=auto_begin, fetch=fetch, limit=limit)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/databricks/connections.py\", line 798, in execute\n table = self.get_result_from_cursor(cursor, limit)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/sql/connections.py\", line 132, in get_result_from_cursor\n return dbt.clients.agate_helper.table_from_data_flat(data, column_names)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/clients/agate_helper.py\", line 127, in table_from_data_flat\n value = json.dumps(value, cls=dbt.utils.JSONEncoder)\n File \"/usr/lib/python3.8/json/init.py\", line 234, in dumps\n return cls(\n File \"/usr/lib/python3.8/json/encoder.py\", line 199, in encode\n chunks = self.iterencode(o, _one_shot=True)\n File \"/usr/lib/python3.8/json/encoder.py\", line 257, in iterencode\n return _iterencode(o, 0)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/utils.py\", line 351, in default\n return super().default(obj)\n File \"/usr/lib/python3.8/json/encoder.py\", line 179, in default\n raise TypeError(f'Object of type {o.class.name} '\nTypeError: Object of type ndarray is not JSON serializable\n", "name": "LogDebugStackTrace", "pid": 50, "thread": "Thread-8", "ts": "2023-10-26T19:13:37.437631Z"}}

### System information
**The output of `dbt --version`:**

The operating system you're using: This has been confirmed on MacOS Ventura 13.5.2 and Ubuntu 20.04

The output of python --version: python 3.8.10

Additional context

Add any other context about the problem here.

benc-db commented 1 year ago

Thanks for reporting, will investigate.

github-actions[bot] commented 6 months ago

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue.