How to use multiple quill contexts in the same file

[benwbooth]

I'm writing ammonite scripts that query a Postgres database, then do analysis on them using Spark DataFrames. I would like to use Quill queries for both databases, but I'm finding it difficult to figure out how to use both contexts within the same script file. When I try to load the PostgresJdbcContext after importing the Quill SparkContext, I get error messages about missing Decoders, or complaints that the lift, quote, or run macros are already defined in the SparkContext. Is there any documentation on how to achieve this?

Thanks! @getquill/maintainers

[deusaquilus]

Yeah, this is probably the single most difficult thing about query contexts. They're really not into sharing functionality. Basically, instead being a sane person and plugging the object/trait with your queries into some imported context where your spark/database sits around, you'll need to use a sort of inversion-of-control pattern where you'll have a trait with a bunch of common queries, then tack this trait onto the different contexts you are using... and yes, encoders and decoders will be a mess. Also, when you start doing this, you can't always directly use JdbcContext or SparkContext when doing self types (deep groan) since they are classes and not traits. With out further ado, Let the mind bending begin!

For starters, create a common trait with a bunch of encoders that you can assume exist in both Postgres and Spark. These are the encoders that will be available to your common queries:

trait EncodersProvider {
  implicit val stringEncoder: Encoder[String]
  implicit val bigDecimalEncoder: Encoder[BigDecimal]
  implicit val booleanEncoder: Encoder[Boolean]
  ...
}

Then create the class where you are going to have queries common to Spark and Postegres, whose self type is a Quill Context as well the encoders provider you created earlier.

trait QuillExtensions {
  this: Context[_, _] with EncodersProvider =>

    def personGreaterThenAge(age:Int) =
      (query:Query[Person]) => query.filter(_.age > lift(age))
}

Once you've done that, throw together a bunch of traits in a haphazard way to get contexts. Since these things don't commute (oh Dotty, how we long for thee!) i.e. the order you do 'A with B with C' actually matters, you have to try a couple of ways to make it work properly. The below recipe should do the trick:

import io.getquill.context.spark.{Decoders, Encoders}

// Have to use a 2-step process with Spark, otherwise it didn't work for me:
trait SparkEncodersProvider extends EncodersProvider with QuillSparkContext with Encoders with Decoders
object SparkQuillContext extends QuillSparkContext with QuillExtensions with SparkEncodersProvider with Encoders with Decoders {}

// Postgres was a bit simpler:
class PostgresQuillContext(dataSource: DataSource with Closeable) extends PostgresJdbcContext[Literal](Literal, dataSource)
  with QuillExtensions with EncodersProvider with io.getquill.context.jdbc.Encoders with io.getquill.context.jdbc.Decoders {}

... and that's all folks! Anything you put inside the QuillExtensions will be available in both the SparkQuillContext and the PostgresJdbcContext, then you pass these contexts down your stack and query stuff from them. Beware that working your way around date logic is tricky but with this pattern you can do it by placing MappedEncodings into your PostgresJdbcContext and SparkQuillContext. Let me know if you want more help with that and I can put up a gist.

@maintainers. It really is a giant pain in the rear to share functionality across different kinds of Quill contexts. This would be greatly simplified by having some minimal trait containing Encoders/Decoders common to all contexts as well as turning things like JdbcContext into traits as opposed to classes. Perhaps even top level context like PostgresJdbcContext or QuillSparkContext should be traits as opposed to classes (that could easily have implementations DefaultPostgresJdbcContext etc...) because that would give us much nicer composition properties when writing custom contexts.

[benwbooth]

@deusaquilus Thanks for the comprehensive write-up! This seems like a real pain, and I'm not sure how well it would translate to my use-case of writing ad-hoc ammonite REPL scripts. So I'll probably put off trying this out until the maintainers can come up with a simpler way to achieve this. Thanks again!

[benwbooth]

Why was this issue closed? This is a serious usability issue and should be addressed.

[deusaquilus]

Hi @benwbooth. I know it's been quite a long time but if this issue still interests you, please have a look at my example here. It shows an example of exactly how to do this. The CrossCompatibleExtensions is where you put the queries that will be available to both Spark and JDBC contexts (I used H2 as an example but Postgres will work just as well).

[pedrohidalgo]

FYI, I implemented a simple solution, I was able to use multiple Quill contexts within the same file like this:

// In any place other than UserRepo.scala
sealed trait DBContext
case object MySqlContext extends DBContext
case object H2Context extends DBContext

// UserRepo.scala
class UserRepo(dbContext: DBContext, config: Config) {

  private lazy val ctx = dbContext match {
    case MySqlContext => new MysqlJdbcContext(SnakeCase, config)
    case H2Context => new H2JdbcContext(SnakeCase, config)
  }

  import ctx._

  def cleanUpTable(): Unit = {
    ctx.run(query[User].delete)
  }

  def findAllUsers(): List[User] = {
    ctx.run(query[User])
  }

}