-
### Description
The issue use to add support to read/select file metadata cloumns for parquet scan if spark user explicitly reference them with (for example) `select _metadata.file_path` statement.…
-
**Task Description:**
Training a simple classifier using keras + horovod spark and getting below error
**Error:**
```
[3]:Error in sys.excepthook:
[3]:
[3]:Original exception was:
[3]:#
…
-
Often in spark ML pipelines, we work with vector columns (Sparse and Dense vectors) through the dataframe API. One operation we recently tried is writing these columns into a snowflake table where the…
-
### Steps to reproduce the behavior (Required)
1. create table :
CREATE TABLE bi_realtime_sr.`da_brn_down_detail_test_01` (
`insert_hash_key` varchar(30) NOT NULL,
`id` bigint(20) NOT NULL…
-
CDM run fails with the below error for complex tables when trying to preserve TTL and Writetime using properties. Performed the below steps :
1. Update below property in cdm.properties-
spark.cdm.…
-
I understand this might be by design, but https://docs.delta.io/latest/delta-batch.html#add-columns could be made more clear. It currently says,
> By default, nullability is true.
![image](https…
-
Hi,
I am having problems with the Automatic Schema Evolution for merges with delta tables.
I have a certain Delta table in my data lake with around 330 columns (the target table) and I want to u…
-
Dear community,
I found an error in using Hudi. If we use the array type in the table.Afterwards, every time I add columns and then write , it throws an exception, The error message is as follows:
…
-
# Context
Spark / Spark SQL supportes _nested_ datatypes. So called [`StructType`](https://docs.databricks.com/sql/language-manual/data-types/struct-type.html) are basically a _container_ for other a…
-
Define a function and associated unit tests to compute the *[spark](https://en.wikipedia.org/wiki/Spark_(mathematics))* of a matrix.
Note that the spark is NP-hard to compute in general. Here is a …