-
## Billing Concerns:
I'm a user of the Spark BigQuery Connector, and I've been using it in two different scenarios:
1. Directly executing a SQL query using option("query", sql).
2. Loading a tabl…
-
Hi.
I am developing a process to ingest data from my hdfs using Hudi. I want to partition the data using a custom keygenerator class where the partition key will be a tuple columnName@NumPartitions. …
ghost updated
4 months ago
-
Hi,
I am trying to save a Spark dataframe using the connector. The dataframe schema is:
root
|-- CustomerNumber: string (nullable = true)
|-- ProductId: string (nullable = true)
|-- Multiplie…
-
While Running the Api in Spark Shell
this command -> val sodasDf = spark.read.format("org.apache.dsext.spark.datasource.rest.RestDataSource").options(parmg).load()
is resulting in error , the e…
-
With Spark 2.0.0, the Spark session / SQL context / hive context have all been amalgamated into a single object. Right now, `hive_context()` returns that:
```
> hive_context(sc)
class org.apache.s…
-
### Description
We are working on [Gluten ](https://github.com/oap-project/gluten) and find ANSI Compliance of Spark functions is weak. When we enable ANSI mode and run Spark UTs with Velox backend, …
-
I am running my code locally in intellij idea with sreaming lens maven dependancy .
I am getting below error . No output , let me know what i am doing wrong here
```
package com.manu.sstreaming…
-
We use the settings about batch size like below confs:
```
spark.sql.inMemoryColumnarStorage.batchSize 32768
spark.sql.execution.arrow.maxRecordsPerBatch 32768
spark.sql.parquet.columna…
-
when I use assembly-0.5.0.jar to access acid table , it seems that dataframe API works well, reading or writing table , either partitioned or not.
But I can not use spark.sql() to access acid tabl…
-
## Feature request
### Overview
The Spark community is adding a new interface to Spark 3.4 that is called [Spark Connect](https://docs.google.com/document/d/1Mnl6jmGszixLW4KcJU5j9IgpG9-UabS0dcM6…