search

Quafadas / dedav4s

Declarative Data Viz 4 Scala
https://quafadas.github.io/dedav4s/
Apache License 2.0
50 stars 4 forks source link

Boss Level: Generate a DSL #18

Closed Quafadas closed 2 years ago

Quafadas commented 2 years ago

Surprisingly, this might actually be tractable.

Scalably typed already does this in JS land. So it would be perfectly possible to have a typesafe DSL in scala JS.

However, the implementations dead end at js.native calls. Point is, that we don't need those. Vega is declarative, so we only need to be able to serialise the object and properties to JSON. We can discard it's actual implementation!

Solution sketch :

And then we'd generate the entire DSL automagically!

armanbilge commented 2 years ago

Alternatively, the Vega JSON schema is published here: https://github.com/vega/schema

If you can generate case classes from JSON schema (surely someone has solved this problem?) then you can use circe on JVM and JS to encode/decode the DSL.

armanbilge commented 2 years ago

Like this: https://cchandurkar.github.io/json-schema-to-case-class/

Quafadas commented 2 years ago

@armanbilge dude I think you just speedran my boss level!

I'll have a look at this when I get a little time to devote to this projcet once more, but if that generator works as advertised, I can't see any reason this wouldn't work.

This project is currently wedded to ujson / upickle... but they work in scala JS land too...

Quafadas commented 2 years ago

Alas, life is not quite so simple!

Limitations

As of now, it does not support a few latest JSON Schema features such as allOf/ anyOf/ oneOf.
Quafadas commented 2 years ago

But maybe we can do this; https://transform.tools/json-schema-to-typescript

And then wheel in scalably typed... ?

Quafadas commented 2 years ago

So this does indeed appear to be a proper boss level problem, having spent a little time with it

armanbilge commented 2 years ago

FWIW, I still suspect the best approach is JSON schema -> case classes.

Quafadas commented 2 years ago

I believe you're right... but after some hours looking into exactly the json schema -> case class issue, I have only the start of anyOf... and I'm not even that confident it's right. In fact I know it needs more work.

I fear this is just... hard.

armanbilge commented 2 years ago

There's still a lot of options to explore :) I guess my point is, focus on language/platform agnostic technologies.

For example, you posted this tool above: https://transform.tools

I see that it has features to transform JSON schema to:

Just tossing ideas out there ... there are many language-agnostic protocols, and many converter tools out there, and many many people with this problem. So, surely, there must be a way, but I am certain it won't involve Typescript nor ScalablyTyped because those are specific to JavaScript which is exactly what you want to avoid.

Quafadas commented 2 years ago

I'm listening, and I originally thought the same. In this case, that tools doesn't work with schema -> protobuf. And the API literally comes out the same as I paste it in .

The swagger generator is new for me though. I will give that route a go...

Quafadas commented 2 years ago

This is the most promising route I have so far;

https://app.quicktype.io

Apparently generates Kotlin classes from a JSON schema. It looks like it does a pretty good job. It is worth noting, that it does not capture the complete subtleties of the schema.

e.g. the Autosize enums / intersection / oneOf interactions.

sealed classes need to be re-written to intersection types.

data class = case class

param? : String = param : Option[String]]

But Kotlin syntax, is not so far from scala... and initial (manual) tests are promising.

I also think that as all modifications would be syntactic in nature, they can be done in serial, i.e. without needing to parse a (complicated) syntax tree...

As that bit, is already done... which is attractive...

Before putting in more effort - check serialisation....

armanbilge commented 2 years ago

Very nice find! Looks like once-upon-a-time there was a PR to add Scala to that tool:

Quafadas commented 2 years ago

I looked at this project and concluded that I'd start from kotlin... https://github.com/quicktype/quicktype/pull/1932

There's a certain amount of low hanging fruit above that I've already picked.

Let's see if it gets any response from the maintainers...

Quafadas commented 2 years ago

This is quite the journey. I believe the PR above produces syntactically valid scala code now, using intersection types for unions. Or at least I believe the case classes "compile".

But intersection types turned out to be a giant gotcha.

In order to include scala3 in the quicktype test suit, I believe I'd need to serialise, then deserialise their test suite. I have been bemused to discover that, as far as I can make out, scala JSON libraries, do not currently support intersection types. I guess it's a niche use case :-)!

So, I need to either revisit the way of encoding these types... stuck...

armanbilge commented 2 years ago

By "intersection types" do you mean | as in union types? :) https://dotty.epfl.ch/docs/reference/new-types/union-types.html

(I believe intersection types are & https://dotty.epfl.ch/docs/reference/new-types/intersection-types.html)

In theory it should be possible to derive encoder/decoders for union types, but not sure in practice. Off the top of my head I can think of two problems:

  1. Lack of compiler support e.g. https://github.com/lampepfl/dotty/issues/14770

  2. How to distinguish between types. E.g. consider:

    case class A(foo: Int, bar: Option[Int])
case class B(foo: Int, baz: Option[String])

    If you want to decode { "foo": 42 } to A | B, which concrete class should you instantiate? It could be either. If you give A preference for being first, it breaks the commutativity of |.

    Perhaps this is a contrived example, and such situations are rare in practice, but it does mean this is tricky to implement in general.

So, I need to either revisit the way of encoding these types... stuck...

Out of curiosity, is it possible to just used sealed trait like before the advent of the Scala 3 union? E.g.

sealed trait AorB
case class A(...) extends AorB
case class B(...) extends AorB
Quafadas commented 2 years ago

By "intersection types" do you mean | as in union types? :)

I do. I am mixing up the the "type intersection" and "set intesectoin" Venn diagrams in my head. I am probably going to call these the wrong names forever. Not good. I think I have the right mental model though, so I guess that's something.

Out of curiosity, is it possible to just used sealed trait like before the advent of the Scala 3 union? E.g.

It must be. It is ... less beautiful, but I think I need to simply accept this. I either wait or get dragged into problem which are not in my wheelhouse.

Current status; I have

Need to do the union implementation... but after that... I'm backing you circe...

Quafadas commented 2 years ago

https://github.com/com-lihaoyi/upickle/issues/386

and the same problem with circe.

https://scastie.scala-lang.org/Quafadas/Pay4wq3zSMCger0ScI9wdQ/38

That's game over for the time being...

armanbilge commented 2 years ago

The -Xmax-inlines? What happens if you add the compiler flag? :)

https://github.com/armanbilge/gcp4s/blob/bd8949adc84dc2b01a76a272bcb5c26df4880916/build.sbt#L31

Quafadas commented 2 years ago

I can barely believe it, but this appears to work!

i.e. we now have a complete, type safe DSL that works exactly the same in both JS and JVM land.

To be released after a little playing...

Quafadas commented 2 years ago

@armanbilge Thankyou so much for your guidance. I hope one day you have a need to plot something and find it useful in return!

armanbilge commented 2 years ago

@Quafadas that is absolutely mind-blowing 🤯 Phenomenal! Exactly, you thought I was sticking my nose in here for selfless reasons? 😉

Quafadas commented 2 years ago

It's so close... https://github.com/Quafadas/dedav4s/issues/21

I think that's a bug in the Vega spec... but it reproduces all other parts of the spec beautifully!