soumilshah1995
commented
10 months ago previous test i used spark parquet and had launched thrift server using similar technique worked fine
Thrift sever
spark-submit \
--master 'local[*]' \
--conf spark.executor.extraJavaOptions=-Duser.timezone=Etc/UTC \
--conf spark.eventLog.enabled=false \
--conf spark.sql.warehouse.dir=file:///Users/soumilshah/Desktop/soumil/sparkwarehouse \
--packages 'org.apache.spark:spark-sql_2.12:3.4.0,org.apache.spark:spark-hive_2.12:3.4.0' \
--class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 \
--name "Thrift JDBC/ODBC Server" \
--executor-memory 512mthis was just using spark and parquet
{{ config(
materialized='incremental',
file_format='parquet',
incremental_strategy='insert_overwrite',
) }}
SELECT
date(d) AS id,
d AS full_date,
EXTRACT (YEAR FROM d) AS YEAR,
EXTRACT (WEEK FROM d) AS year_week,
EXTRACT (DAY FROM d) AS year_day,
EXTRACT (YEAR FROM d) AS fiscal_year,
EXTRACT (QUARTER FROM d) AS fiscal_qtr,
EXTRACT (MONTH FROM d) AS MONTH,
date_format(d, 'MMMM') AS month_name,
EXTRACT (DOW FROM d) AS week_day,
date_format(d, 'EEEE') AS day_name,
(CASE WHEN date_format(d, 'EEEE') NOT IN ('Sunday', 'Saturday') THEN 0 ELSE 1 END) AS day_is_weekday
FROM (SELECT EXPLODE(months) AS d FROM (SELECT SEQUENCE (TO_DATE('2000-01-01'), TO_DATE('2023-01-01'), INTERVAL 1 DAY) AS months))DBT project.yml
name: 'sparkdbt'
version: '1.0.0'
config-version: 2
profile: 'sparkdbt'
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]
clean-targets:
- "target"
- "dbt_packages"
models:
sparkdbt:
core:
+enabled: true
+materialized: table
dim_date:
+materialized: tableworks fine i was able to see table and query it using beeline and dbeaver as well i am seeing issue with Hudi tables not sure what CONF i am missing here
I also have read examples provided https://github.com/apache/hudi/blob/master/hudi-examples/hudi-examples-dbt/README.md
could not get it to work for some reason
ad1happy2go
Hey community,
I hope you're doing well. I recently launched a Thrift server using Spark, incorporating the Hudi library. The server runs smoothly, and I can interact with it using Beeline to query data successfully.
BEELINE
Works fine
INSerted data
DBT debug
Directory
schema.yml
hudi_insert_overwrite_table.sql
dbt_project.yml
DBT run
Any insights or guidance on resolving this issue would be greatly appreciated! If you have any experience with integrating Hudi into Spark Thrift Server and overcoming similar challenges, your expertise would be invaluable.
Thanks in advance for your help!
Regards Soumil