-
I am using VS code with databricks-connect v2.4.8, and I wrote some pytests. In `conftest.py` I created a fixture:
```
@pytest.fixture(scope="session")
def spark():
spark = DatabricksSession.b…
-
**Describe the problem you faced**
Record lookup in a table with record level index results in `None.get` exception.
**To Reproduce**
Steps to reproduce the behavior:
1. Create a table wi…
-
## Description
Develop a comprehensive set of test cases for the Open Spark UI to ensure functionality, usability, and responsiveness across different screens and features. The Open Spark UI includes …
-
## Problem
### How much memory does a spark-dependencies job take while handling about 12Gb data index?
I am totally new to the spark project and I have tried serval times to run a spark-depend…
-
**Details:**
Vulnerability Details (PRISMA-2021-0055)
IBM X-Force ID: 177835
DESCRIPTION: Apache Commons Codec could allow a remote attacker to obtain sensitive information, caused by the imp…
-
### Background
#6 - 사용자의 behavior log에 대해서 Continuous Sessionization 까지 PoC를 완료했습니다.
데이터가 생성 및 수집 1시간 안에 session id를 부과하는 것을 대응하기 위해 Workflow 도구인, Airflow를 도입하고자 합니다.
현재 Spark 및 Airflow 환경이 갖추어지지 않…
-
### What feature you would like to be added?
Currently if the driver pod still reports not found for too long after successfully running `spark-submit` then the spark application fails with `"applica…
-
Hello! Does this adapter work with only the Fabric Livy endpoints or would it also work with [Spark Livy endpoints](https://learn.microsoft.com/en-us/rest/api/synapse/data-plane/spark-batch/create-spa…
-
### Describe the bug
This was found during debugging CI failures of #1050.
One example of failed test is `date_add with int scalars` in `CometExpressionSuite`. The query is `"SELECT _20 + CAST(2 a…
-
There hasn't been any records to the stream after previous job run. Job failed to restart with "Assertion failed exception" the current batch id is -1 at restart which is less than previous batchId.
…