zio / zio-quill

Compile-time Language Integrated Queries for Scala
https://zio.dev/zio-quill
Apache License 2.0
2.15k stars 346 forks source link

Document Quill Athena/Presto #1415

Open Dr-Nikson opened 5 years ago

Dr-Nikson commented 5 years ago

Can I use Quill with Athena/Presto JDBC driver? I would appreciate any examples.

deusaquilus commented 5 years ago

Okay... I haven't actually tested any of these instructions so please tell me what works and what doesn't.

Easiest way would be to use JDBC with both presto as well as athena together with the Postgres since they're basically the same. You can get the athena JDBC driver directly from amazon here, I do not see it on maven-central. You will need to manually deploy it to the maven/ivy repo you are using or find some other way to get it into your classpath. I'll walk you through the former.

Annoying Build Stuff

I just installed it into my local maven repo using install-file. If you have a private maven repo you can use deploy-file in a very similar way.

mvn org.apache.maven.plugins:maven-deploy-plugin:2.8.2:install-file -Dfile=AthenaJDBC42_2.0.7.jar -DpomFile=pom.xml

The pom I used defines basic coordinates of how to reference this thing:

<?xml version='1.0' encoding='UTF-8'?>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0">
    <modelVersion>4.0.0</modelVersion>
    <groupId>annoying.amazon</groupId>
    <artifactId>athena-simba-42</artifactId>
    <packaging>jar</packaging>
    <description>Annoying Upload of AWS Simba Driver because they didn't put it into Maven-Central</description>
    <version>2.0.7</version>
</project>

Then you can pull it into your build.sbt like so:

libraryDependencies ++= Seq(
  "annoying.amazon" % "athena-simba-42" % "2.0.7"
)

Athena

Once that's done, you can use the data source directly like this:

val ds = new com.simba.athena.jdbc.DataSource()
ds.setURL("jdbc:awsathena://AwsRegion=us-east-2;UID...")
ds.setUserID("joe")
ds.setPassword("foo123")

val ctx = new PostgresJdbcContext[Literal](Literal, ds.asInstanceOf[DataSource with Closeable])
import ctx._

case class MyAthenaTable(id:Int, value:String)
val q = quote {
  query[MyAthenaTable].filter(_.id == 1)
}

Athena with Hikari

In Quill we typically don't use data sources directly so I would probably wrap the athena driver into a hikari connection pool like this:

// application.conf
myAthenaDB.dataSourceClassName=com.simba.athena.jdbc.Driver
myAthenaDB.dataSource.url="jdbc:awsathena://AwsRegion=us-east-2;UID..."
myAthenaDB.dataSource.user=joe
myAthenaDB.dataSource.password=foo123

Then your code looks like this:

val ctx = new PostgresJdbcContext[Literal](Literal, "myAthenaDB")
import ctx._

case class MyAthenaTable(id:Int, value:String)

val q = quote {
  query[MyAthenaTable].filter(_.id == 1)
}

Presto with Hikari

For presto it's very similar but they've done you the kindness of puting their driver into maven-central. Add this to your build.sbt:

libraryDependencies ++= Seq(
  "com.facebook.presto" % "presto-jdbc" % "0.218"
)

I'm not sure what the DataSource class with Presto is I'll just show you the hikari approach:

// application.conf
myPrestoDB.dataSourceClassName=com.facebook.presto.jdbc.PrestoDriver
myPrestoDB.dataSource.url="jdbc:presto://example.net:8080/hive/sales"
myPrestoDB.dataSource.user=joe
myPrestoDB.dataSource.password=foo123

Then your code looks like this:

val ctx = new PostgresJdbcContext[Literal](Literal, "myPrestoDB")
import ctx._

case class MyPrestoTable(id:Int, value:String)

val q = quote {
  query[MyPrestoTable].filter(_.id == 1)
}