search

databricks / Spark-The-Definitive-Guide

Spark: The Definitive Guide's Code Repository
http://shop.oreilly.com/product/0636920034957.do
Other
2.85k stars 2.76k forks source link

what does "in SQL" mean in book #26

Closed yuxh closed 6 years ago

yuxh commented 6 years ago

there are many statements in book like this: // in Scala df.select("ORIGIN_COUNTRY_NAME", "DEST_COUNTRY_NAME").distinct().count() -- in SQL SELECT COUNT(DISTINCT(ORIGIN_COUNTRY_NAME, DEST_COUNTRY_NAME)) FROM dfTable I understand "in Scala " means execution in spark-shell. Does "in SQL" mean short hand for spark.sql("xx")? similar question is how can I execute commands in Spark-The-Definitive-Guide/code/xx.sql file?

bllchmbrs commented 6 years ago

Yes, that’s what in Scala and in Sql means. You’ll have to execute them 1 by 1

On Wed, Sep 5, 2018 at 01:34 yuxh notifications@github.com wrote:

there are many statements in book like this: // in Scala df.select("ORIGIN_COUNTRY_NAME", "DEST_COUNTRY_NAME").distinct().count() -- in SQL SELECT COUNT(DISTINCT(ORIGIN_COUNTRY_NAME, DEST_COUNTRY_NAME)) FROM dfTable I understand "in Scala " means execution in spark-shell. Does "in SQL" mean short hand for spark.sql("xx")? similar question is how can I execute commands in Spark-The-Definitive-Guide/code/xx.sql file?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/databricks/Spark-The-Definitive-Guide/issues/26, or mute the thread https://github.com/notifications/unsubscribe-auth/ABkQB0KkxqmANi4v1ts7e8RVW91C5vgjks5uX4x4gaJpZM4WabcL .