How we can read and write the data to Hive?

YotpoLtd / metorikku

A simplified, lightweight ETL Framework based on Apache Spark

https://yotpoltd.github.io/metorikku/

MIT License

583 stars 155 forks source link

Open sunilpandey-de opened 3 years ago

sunilpandey-de commented 3 years ago

Can someone help me understand using metorikku how we can read and write the data to Hive tables.

Thank You!

lyogev commented 3 years ago

Can you share your environment details? (yarn/k8s/spark-shell?) which hive you are using? what version of spark?

Thanks

sunilpandey-de commented 3 years ago

We are running this on our development yarn cluster, Hive version 2.3 and Spark 3.0.

lyogev commented 3 years ago

So hive is configured correctly in the spark-defaults etc? If so all you need to save in hive is set tableName in your output https://github.com/YotpoLtd/metorikku/blob/7fdd0838480e89285e5c7b77cc3e45d54d6bf5a0/config/metric_config_sample.yaml#L50