Open racheldaniel opened 1 year ago
Thanks for reporting, will investigate.
This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue.
Describe the bug
dbt-databricks throws Object of type ndarray is not JSON serializable when previewing nested json (
dbt show
)Steps To Reproduce
This will throw Object of type ndarray is not JSON serializable
), cast_types as (
select
), explode_series as (
select * , explode(series) from cast_types
) select * from explode_series
{"data": {"exc_info": "Traceback (most recent call last):\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/task/base.py\", line 372, in safe_run\n result = self.compile_and_execute(manifest, ctx)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/task/base.py\", line 323, in compile_and_execute\n result = self.run(ctx.node, manifest)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/task/base.py\", line 419, in run\n return self.execute(compiled_node, manifest)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/task/show.py\", line 38, in execute\n adapter_response, execute_result = self.adapter.execute(\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/databricks/utils.py\", line 72, in wrapper\n return func(*new_args, *new_kwargs)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/databricks/impl.py\", line 150, in execute\n return super().execute(sql=sql, auto_begin=auto_begin, fetch=fetch, limit=limit)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/base/impl.py\", line 290, in execute\n return self.connections.execute(sql=sql, auto_begin=auto_begin, fetch=fetch, limit=limit)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/databricks/connections.py\", line 798, in execute\n table = self.get_result_from_cursor(cursor, limit)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/sql/connections.py\", line 132, in get_result_from_cursor\n return dbt.clients.agate_helper.table_from_data_flat(data, column_names)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/clients/agate_helper.py\", line 127, in table_from_data_flat\n value = json.dumps(value, cls=dbt.utils.JSONEncoder)\n File \"/usr/lib/python3.8/json/init.py\", line 234, in dumps\n return cls(\n File \"/usr/lib/python3.8/json/encoder.py\", line 199, in encode\n chunks = self.iterencode(o, _one_shot=True)\n File \"/usr/lib/python3.8/json/encoder.py\", line 257, in iterencode\n return _iterencode(o, 0)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/utils.py\", line 351, in default\n return super().default(obj)\n File \"/usr/lib/python3.8/json/encoder.py\", line 179, in default\n raise TypeError(f'Object of type {o.class.name} '\nTypeError: Object of type ndarray is not JSON serializable\n"}, "info": {"category": "", "code": "Z011", "extra": {}, "invocation_id": "4d4b3d45-15e5-4f27-8045-3ae4fb04c5b0", "level": "debug", "msg": "Traceback (most recent call last):\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/task/base.py\", line 372, in safe_run\n result = self.compile_and_execute(manifest, ctx)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/task/base.py\", line 323, in compile_and_execute\n result = self.run(ctx.node, manifest)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/task/base.py\", line 419, in run\n return self.execute(compiled_node, manifest)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/task/show.py\", line 38, in execute\n adapter_response, execute_result = self.adapter.execute(\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/databricks/utils.py\", line 72, in wrapper\n return func(new_args, **new_kwargs)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/databricks/impl.py\", line 150, in execute\n return super().execute(sql=sql, auto_begin=auto_begin, fetch=fetch, limit=limit)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/base/impl.py\", line 290, in execute\n return self.connections.execute(sql=sql, auto_begin=auto_begin, fetch=fetch, limit=limit)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/databricks/connections.py\", line 798, in execute\n table = self.get_result_from_cursor(cursor, limit)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/sql/connections.py\", line 132, in get_result_from_cursor\n return dbt.clients.agate_helper.table_from_data_flat(data, column_names)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/clients/agate_helper.py\", line 127, in table_from_data_flat\n value = json.dumps(value, cls=dbt.utils.JSONEncoder)\n File \"/usr/lib/python3.8/json/init.py\", line 234, in dumps\n return cls(\n File \"/usr/lib/python3.8/json/encoder.py\", line 199, in encode\n chunks = self.iterencode(o, _one_shot=True)\n File \"/usr/lib/python3.8/json/encoder.py\", line 257, in iterencode\n return _iterencode(o, 0)\n File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/utils.py\", line 351, in default\n return super().default(obj)\n File \"/usr/lib/python3.8/json/encoder.py\", line 179, in default\n raise TypeError(f'Object of type {o.class.name} '\nTypeError: Object of type ndarray is not JSON serializable\n", "name": "LogDebugStackTrace", "pid": 50, "thread": "Thread-8", "ts": "2023-10-26T19:13:37.437631Z"}}
The operating system you're using: This has been confirmed on MacOS Ventura 13.5.2 and Ubuntu 20.04
The output of
python --version
: python 3.8.10Additional context
Add any other context about the problem here.