RevolutionAnalytics / dplyr-spark

spark backend for dplyr
48 stars 18 forks source link

copy to from files, urls and what not #30

Open piccolbo opened 9 years ago

piccolbo commented 9 years ago

using https://spark.apache.org/docs/latest/sql-programming-guide.html#loading-data-programmatically or LOAD INPATH, see db insert into

piccolbo commented 9 years ago

CREATE TEMPORARY TABLE impressions USING org.apache.spark.sql.jdbc OPTIONS ( url "jdbc:postgresql:dbserver", dbtable "impressions" )

Oops this doesn't work in hivql, it's load inpath or bust

piccolbo commented 9 years ago

After some research dplyr doesn't have a feature to load data in a db that is not in memory. Propose the following interface: