Implement `zio-jdbc` backend

guizmaii commented 1 year ago

Adapted from the doobie backend

Needs:

oyvindberg commented 1 year ago

current todo:

[x] Get two PRs merged in zio-jdbc. To test newest code you can merge them both to main and run publishLocal
[x] Try to use JdbcDecoder.apply instead of new JdbcDecoder for unary types (less byte code, faster)
[x] Remove JdbcEncoder for row types, I'm quite sure we don't need it (like we don't need Write for doobie)
[x] zio-json: Implement DeriveJsonDecoder.gen and DeriveJsoEncoder.gen in typo so we don't need to run all the macros at compile-time. See support for other json libraries
[x] Port ParameterMetadata from anorm. I think it's alright if we just generate it from typo actually, we can upstream it to a zio-jdbc-postgres module later if wanted. This is what we need to get SqlExpr.Const working and fix the last failing test

Iterate more on arrays:
[ ] implement for string enums
[ ] Ensure we have all instances we need. Extend ArrayTest to cover all columns types we support. I'll work on this
[ ] confirm that primitiveArrayDecoder works

guizmaii commented 1 year ago

zio-json: Implement DeriveJsonDecoder.gen and DeriveJsoEncoder.gen in typo so we don't need to run all the macros at compile-time. See support for other json libraries

I'm not sure about this one as zio-json is not like the other JSON libs. It doesn't use a JSON AST. To write the decoders manually you have to manually write the parser. To write the encoders manually you have to manually write the JSON string.

Some DeriveJsonDecoder.gen and DeriveJsoEncoder.gen can probably be replaced by manually written instances in simple cases but in more complex ones I'm not sure it's worth it. WDYT?

oyvindberg commented 1 year ago

To write the encoders manually you have to manually write the JSON string.

ok, maybe we should call zio-json a bad match and skip it? you can use zio-jdbc with play-json or circe without any problems

guizmaii commented 1 year ago

ok, maybe we should call zio-json a bad match and skip it? you can use zio-jdbc with play-json or circe without any problems

ZIO users tend to prefer to stay in the ZIO ecosystem. zio-json has, for example, the particularity to be way faster than Circe or Play (because it's not using an AST amongst other things) I think it'd be better for ZIO users to offer them the "ZIO experience". That's what they will ask for/look for.

What's the issue with using the derived schema?

guizmaii commented 1 year ago

Try to use JdbcDecoder.apply instead of new JdbcDecoder for unary types (less byte code, faster)

I take this one.

Edit: Done

guizmaii commented 1 year ago

Port ParameterMetadata from anorm.

I'm giving a try to this one

Edit: Done. All the tests are passing now 🎉

guizmaii commented 1 year ago

we can upstream it to a zio-jdbc-postgres module later if wanted

How do you see this? What would you add to zio-jdbc?

oyvindberg commented 1 year ago

What's the issue with using the derived schema?

Schema. The produced json structure is different from that of the other json libraries, I'd very much want them to be in sync. For the circe and play-json codecs column/field names are those from the database, not the scala names.
Maintenance. The zio test project currently compiles some 30 seconds slower than the others, I guess most of the reason is these heavy-duty macro calls, and I'll compile them many, many times.

If we can solve the schema issue I can fix the maintenance issue by just not compiling them as part of the huge chunk of tests.

oyvindberg commented 1 year ago

Edit: Done. All the tests are passing now 🎉

Fantastic! We're getting somewhere now!

Thanks again for pushing through with this, I wasn't all that optimistic that somebody would be able to take those 700 odd of lines of macro-like code and port them to a new library even if it was possible in theory

guizmaii commented 1 year ago

I wasn't all that optimistic that somebody would be able to take those 700 odd of lines of macro-like code and port them to a new library even if it was possible in theory

Without your help, it wouldn't have been possible 🙇‍♂️

guizmaii commented 1 year ago

Let me check what we can do with the zio-json issue

guizmaii commented 1 year ago

Get two PRs merged in zio-jdbc.

Merged 🎉

guizmaii commented 1 year ago

Did some research and it seems possible to write JsonDecoder and JsonEncoder instances the way you want them.

Here's an example for TypoLine:

  implicit lazy val jsonDecoder: JsonDecoder[TypoLine] = {
    val aCursor = JsonCursor.field("a").isNumber
    val bCursor = JsonCursor.field("b").isNumber
    val cCursor = JsonCursor.field("c").isNumber

    JsonDecoder[Json].mapOrFail { json =>
      for {
        a <- json.get(aCursor).map(_.value.doubleValue())
        b <- json.get(bCursor).map(_.value.doubleValue())
        c <- json.get(cCursor).map(_.value.doubleValue())
      } yield TypoLine(a, b, c)
    }
  }
  implicit lazy val jsonEncoder: JsonEncoder[TypoLine] =
    new JsonEncoder[TypoLine] {
      override def unsafeEncode(a: TypoLine, indent: Option[Int], out: Write): Unit = {
        out.write("{")
        out.write("\"a\":")
        JsonEncoder[Double].unsafeEncode(a.a, indent, out)
        out.write(",")
        out.write("\"b\":")
        JsonEncoder[Double].unsafeEncode(a.b, indent, out)
        out.write(",")
        out.write("\"c\":")
        JsonEncoder[Double].unsafeEncode(a.c, indent, out)
        out.write("}")
      }
    }

The JsonEncoder seems easy to generate. The JsonDecoder is more tricky I think. I'll have a look if we can write the JsonDecoder differently, avoiding the JsonCursor API which probably makes the generation of the instances more complex 🤔

Edit: Found how to write the JsonDecoder 😏🥳

guizmaii commented 1 year ago

@oyvindberg Done! We're not using macros for zio-json instances anymore 🎉

guizmaii commented 1 year ago

can we drop UpdateResult here and just return the row? We know there will 1 changed row

Done

guizmaii commented 1 year ago

@oyvindberg From your comment here: https://github.com/oyvindberg/typo/pull/57#issuecomment-1784257027, there's only these things left to be done:

[x] implement for string enums
[x] Ensure we have all instances we need. Extend ArrayTest to cover all columns types we support. I'll work on this
[x] confirm that primitiveArrayDecoder works

guizmaii commented 1 year ago

Remove JdbcEncoder for row types, I'm quite sure we don't need it (like we don't need Write for doobie)

Done. The tests pass without these instances

guizmaii commented 1 year ago

There's a bug in zio-jdbc that makes the tests fail in Scala 2.12. See https://github.com/zio/zio-jdbc/pull/185

guizmaii commented 1 year ago

Test are now passing on all Scala versions 🎉

oyvindberg commented 1 year ago

Test are now passing on all Scala versions 🎉

We're quite close then! 🚀

I hope to be able to dig up some time tonight after Halloween celebrations :)

oyvindberg commented 1 year ago

now it compiles almost as fast as the other combinations

oyvindberg commented 1 year ago

discovered https://github.com/zio/zio-jdbc/issues/188 while verifying. will push tests to this branch shortly

oyvindberg commented 1 year ago

we can move forward with that one test commented out by the way, it's strictly not our problem and it may be hard to solve well in zio-jdbc.

oyvindberg commented 1 year ago

todo:

[x] update docs
[x] move ParameterMetadata from DSL to generated code. The DSL is meant to be optional, so it can't stay there. will need a slight refactor so DbLib can produce extra sc.Files in this case.
[x] published version of zio-jdbc

oyvindberg commented 1 year ago

I'm ready to merge this now, just need a new release of zio-jdbc.

🚀

guizmaii commented 1 year ago

I'm ready to merge this now, just need a new release of zio-jdbc.

I asked for a release

sideeffffect commented 1 year ago

new version has been released https://github.com/zio/zio-jdbc/releases/tag/v0.1.1 :tada:

oyvindberg / typo

Implement `zio-jdbc` backend #57

Iterate more on arrays: