search

scala / slip

obsolete — archival use only
67 stars 15 forks source link

Adding standard Json AST - https://github.com/mdedetrich/scala-json-ast #28

Closed mdedetrich closed 7 years ago

mdedetrich commented 8 years ago

READ THIS FIRST

This discussion is about designing a library for a representation of JSON that can be shared between many different implementations of JSON parsing/pretty printing/transformation/.... As it's a SLIP (and not a SIP, so no language changes!), this shared API may eventually ship as a module that's included in the standard set of Scala libraries (e.g., it would be like a modern, minimalist scala-xml without the language support).

Everyone's welcome to join the discussion, provided your contribution is truly a contribution to the precise topic of this SLIP, as I tried to capture it above. This SLIP is not trying to solve the general problem of how to modularize the standard library, for example. I'm happy to refine my summary if needed. As always, there are plenty of other channels to discuss other (related) topics.

To ensure a productive discussion, please be kind to each other, stay as focussed as possible, and refrain from comments with very little technical content. As we are still refining this SLIP, it's too early to vote.

Thanks, @adriaanm

xuwei-k commented 8 years ago

-1

mdedetrich commented 8 years ago

-1

?

SethTisue commented 8 years ago

@sjrd want to weigh in on the Scala.js aspects of this, and/or call the Scala.js community's attention to it?

sjrd commented 8 years ago

Sure, I already did a surface reading of the proposal. I will have a deeper look at it later. I'll also mention the PR on the Scala.js gitter channel.

SethTisue commented 8 years ago

This question about key ordering in objects is interesting. In the proposal fast preserves key order, safe doesn't. I had expected the opposite.

If it's possible to preserve key order and still get good performance, great! But then I would expect safe to preserve it, too. (After all, the spec doesn't actually forbid order preservation.)

I imagine this will be the subject of further discussion...

xuwei-k commented 8 years ago

I believe Scala stdlib does not need json library.

odersky commented 8 years ago

@xuwei-k We see that sort comments a lot from the scalaz people. I must admit it is getting annoying. If you have serious concerns about a proposal let's hear them. But I am not interested in the commonly voiced opinion that no improvements should be made to the standard library. That's just obstructionism. It's like me telling you there should be no scalaz, I guess you would not like that either.

He-Pin commented 8 years ago

I think a standard way to work with json would be great,there are so many json there in the scala community,and look at the https://github.com/tminglei/slick-pg,it write extends for json4s, play-json, spray-json and argonaut json,and on the play side,I once looked at the google-group,@jroper,said there maybe go with json4j in Play3.0.and the current json4s way I think it's great,share some common thing in the scala community would really help,especially for those new comers.

milessabin commented 8 years ago

-1.

This is downright damaging when the FOSS community already contains several first class implementations, each with its own particular emphases and design goals, and all in use in production in systems large and small.

If you want to promote Scala for Json processing, get behind those libraries. By all means fork or create another one if none of the existing ones exactly meet your needs, but then recognize that you're doing this because one size doesn't fit all, which is precisely why promoting one as a so-called "standard" is harmful.

lihaoyi commented 8 years ago

We see that sort comments a lot from the scalaz people. I must admit it is getting annoying.

+1. It's totally unconstructive, and adds nothing to discussion.

It's even more annoying because oftentimes the objections come out before the exact thing being proposed has even been decided, which is totally ridiculous because by that point you clearly haven't even considered the proposal (because there isn't any yet!)

@hepin1989 OTOH this won't actually solve your use case. This is purely for interop in JSON ASTs, and doesn't provide any facilities for parsing or serializing.

IMHO that makes it considerably less useful. I don't find myself passing around parsed but un-converted JSON ASTs around enough for it to be a hassle; most of my work involves going straight from String -> case class or case class -> string, but maybe other people have different usage patterns.

Do people find themselves mangling, passing around and transforming parsed-by-untyped JSON ASTs between different libraries often?

seanparsons commented 8 years ago

As the maintainer of Argonaut I don't see a huge benefit to Scala itself in there being an inbuilt JSON library, mostly because it runs the usual risk of not changing once first created. Which means it can be a bit of a lowest common denominator implementation that gets rusty really quickly.

I agree with @lihaoyi that it's very rare to need to convert between implementations of ASTs, even to the point where I can't recall a time where I've actually had to do that.

It's also worth mentioning that the JSON standard as I think it stands now calls for arbitrary precision numbers which was a serious piece of work in Argonaut to support in an efficient way, which is why some of this gets hard really quickly.

odersky commented 8 years ago

@milessabin You are making a political argument, not a technical one. By the same yardstick I could argue that TypeLevel should not contain specs2 because users should choose freely between it and ScalaTest or one of the other available libraries. We have had this discussion before, and decided we will go with a batteries included model. Recycling this debate on individual SLIPs is obstructionist IMO.

[Some more thoughts] This in no way endorses the present SLIP which might be promising or not; I have not studied it. But I reject the argument that we should not even consider adding a JSON parser to the standard library, in particular since the one we have is so sorely lacking. Neither need the standard library stay monolithic and unchangeable. We are talking about modules that can be updated at their own pace, with not all sharing the same stability guarantees as the core.

On Wed, Nov 4, 2015 at 7:12 PM, Sean Parsons notifications@github.com wrote:

As the maintainer of Argonaut I don't see a huge benefit to Scala itself in there being an inbuilt JSON library, mostly because it runs the usual risk of not changing once first created. Which means it can be a bit of a lowest common denominator implementation that gets rusty really quickly.

I agree with @lihaoyi https://github.com/lihaoyi that it's very rare to need to convert between implementations of ASTs, even to the point where I can't recall a time where I've actually had to do that.

It's also worth mentioning that the JSON standard as I think it stands now calls for arbitrary precision numbers which was a serious piece of work in Argonaut to support in an efficient way, which is why some of this gets hard really quickly.

— Reply to this email directly or view it on GitHub https://github.com/scala/slip/pull/28#issuecomment-153813694.

Martin Odersky EPFL and Typesafe

lihaoyi commented 8 years ago

Here's a few more opinions:

milessabin commented 8 years ago

@odersky the very idea of a "standard" is a political one. This whole enterprise is political and it's appropriate to respond to it in kind.

SethTisue commented 8 years ago

If you want to promote Scala for Json processing, get behind those libraries

The code in this SLIP will literally be behind those libraries...!

densh commented 8 years ago

scala-offheap should have nothing to do with this. Zero. Nada. Making your core library dependent on a zero-user super-experimental library for no particular gain seems silly.

I wholeheartedly agree. It's way too young for getting into committed relationship with standard library.

milessabin commented 8 years ago

@SethTisue no, it will be a competitor with code which already exists in those libraries. I know you've see the relevant xkcd cartoon.

fommil commented 8 years ago

Why not just pick one that already works well? spray-json in really good. Importantly it uses arbitrary precision for numbers and reminds us all that JSON maps do not preserve ordering. If you really must create a new AST (replacing this one tiny file of awesome) then please do not forget these points.

The only change I would ask of spray-json is to remove the Root* traits, and it the marshalling layer to perhaps move all implicits to the companion of JsonFormat.

Also it has a super fast parser. And years of production usage with lots of tiny performance tweaks along the way.

Plus it has spray-json-shapeless ;-)

Let's just get everybody to standardise on spray-json

fommil commented 8 years ago

(Actually if the marshallers returned Either instead of throwing exceptions, it would significantly speed up Option marshalling, but that's a rather minor technical point).

milessabin commented 8 years ago

@fommil There's no need to "pick one" and give it an official stamp ... if you want to use or advocate for spray-json, that's fine, but there's no grounds for your (or anyone else's) preferences being legislated for.

fommil commented 8 years ago

BTW is this just about the AST or an entire JSON framework? Parsing/marshalling/querying/validation/etc?

I don't believe the standard library even has the power to provide what is needed for a proper marshalling framework. We need shapeless for that.

odersky commented 8 years ago

the very idea of a "standard" is a political one. This whole enterprise is political and it's appropriate to respond to it in kind.

It's good to know how you feel about it. My "politics" is to make Scala more accessible for newcomers and people who do not want to spend a lot of time trying out different alternatives. I stand by that. I am very happy co-opting a best of breed library that's out there; no need to reinvent the wheel. But I am deadset against leaving stdlib with the crippled JSON parser it has, and also leaving it without one.

lvicentesanchez commented 8 years ago

Unless I'm missing the point of this SLIP, this is just a proposal for a unified JSON AST and its goal is to allow different JSON libraries to interop between them; that is... no improvements to the existing JSON parser in the scala standard library.

If the whole purpose of this SLIP is just that, I don't thing it belongs to the standard library, not even as a module. If anything it's a library that the authors of the different JSON libraries might or might not want to use.

As an user I can't even imagine a use case where I would want to have two different JSON libraries in my classpath and rely on a unified JSON AST to interop between them.

SethTisue commented 8 years ago

is this just about the AST or an entire JSON framework?

This SLIP is about the ASTs only.

I suggest any broader discussion happen on scala-debate instead, but maybe those cows are already out of the barn.

SethTisue commented 8 years ago

I am deadset against leaving stdlib with the crippled JSON parser it has, and also leaving it without one.

Do you want to issue a separate call for a SLIP that addresses that? This one doesn't.

tperrigo commented 8 years ago

-1. I'm in agreement with @milessabin, and agree with @fommil that a proper JSON framework should provide parsing/marshalling/validation/etc (which, as he points out, would require something like shapeless...Circe, eg, is making excellent progress on this front). I simply don't see this as something that needs to be (or should be) in the standard library.

odersky commented 8 years ago

@SethTisue Yes, I think the current JSON parser is overdue for a replacement. It would be good to call for proposals.

lihaoyi commented 8 years ago

I don't think the current proposal is useful enough to warrant inclusion, but I think a good AST + a fast parser + a serializer, with the useful knobs (e.g. serialize with proper indentation) would be useful enough to put in there.

odersky commented 8 years ago

Almost every language library has a decent JSON parser. Ours sucks, as do quite a few modules that were introduced in the very early days of Scala when every contribution was welcome, because there was so little of it and the community was tiny anyway. It's time we corrected that, don't you think?

odersky commented 8 years ago

And, sure, maybe with shapeless one can do a more fancy thing with lots of typelevel stuff. That's no reason not to strive for something that's simple and works efficiently and can be put in the standard library.

lihaoyi commented 8 years ago

There are multiple stages that JSON-support can reach

  1. Just AST; bring your own parser and serializer and operations
  2. AST + operations + parser + serializer, bring your own case-class-mapper/etc.
  3. AST + operations + parser + serializer + case-class-mapper a.l.a. shapeless, uPickle, etc.
  4. Fancy operations: lenses, traversals, zippers, etc.

IMHO 1 is not useful enough to warrant inclusion, 2 is useful and straightforward to implement with current technology, 3 is probably just a bit too experimental, even as the author of a library which does exactly that, I would not want my code in the std lib just yet. 4 I have no experience with.

duncan commented 8 years ago

A standard built-in awesome JSON parser would be awesome. Yes, there's room for other parsers that do things beyond a threshold, for example all the stuff you can do with shapeless. But personally, I'm finding that I'm happier with projects that stick with the basics than with those that go too far down the rabbit hole.

wheaties commented 8 years ago

-1

We already had an XML library. It was arguably good and bad at the same time (Daniel's Anti-XML was an improvement.) That said, look how it all worked out. It was removed. There's already a ton of approaches to make with so many libraries to "solve" this issue. I have to agree with @milessabin.

That said, I see there being nothing wrong with having a Scala official JSON library the same as the Scala XML library, separate. I could +1 that approach.

ktoso commented 8 years ago

One datapoint (so please refrain from ranting about the JSR process. thank you.) which may be interesting to keep in mind is that the JDK is rolling in their own JSON types and parser. This may be meaningful in the long-term plan we foresee if we'd roll our own types (or rather not).

https://json-processing-spec.java.net/nonav/releases/1.0/fcs/javadocs/index.html https://json-processing-spec.java.net/

It might be just that this answers the inter-op story? I'm not sure if it will make it into JDK9 though.

UPDATE: The one considered for inclusion in JDK9 was JEP198 but seems like it may slip from the release. // pointed out by @soc, thanks! https://github.com/scala/slip/pull/28#issuecomment-153915854

UPDATE 2: The JSON JEP was dropped from JDK9, in theory it could come back for JDK10, but that's many many years in front of us. (details: https://github.com/scala/slip/pull/28#issuecomment-153917281 )

mandubian commented 8 years ago

A standard AST in Scala could be useful to stop having multiple AST in many projects... Specific language syntaxes, certainly not, XML was a failure... A parser I'm not so sure, there are different approaches to parsing, different requirements... "Official" lightweight extensions to (de)serialize Json AST using existing parsers from the Java world, why not... More about it, I don't see what: pure functional approaches to manipulate Json AST that are present in several libs will stay there or it would mean pure functional structures have appeared in standard...

djspiewak commented 8 years ago

The current JSON situation in the standard library is really awful. It's basically worse than nothing, since it's useless to anyone who knows what it is and how it works, and it entraps newcomers into using it "because it's there".

I think in general, standard library things more often skew towards entrapment than usefulness. This is not an indictment of Scala; most standard libraries fall into the same bucket (see Java). I think this is @milessabin's general point. If we take this observation to its natural conclusion, then the correct response is to apply the scala.xml treatment to JSON: delete with prejudice and don't replace.

With that said, I can see merits in the desire for a "batteries included" language. If you're a newcomer to Scala, you don't want to just fiddle around with encoding your own List and writing merge sort. You probably want to solve an actual problem with actual tasks that you would expect an actual language to have a solution for in the standard library. Like JSON parsing.

Exactly what set of tasks falls into the category of "batteries that should be included" is a very subjective question, and there is no way to answer this definitively in a way that will satisfy everyone. Personally, I don't want JSON parsing in the standard library any more than I want XML. But that's just me.

I certainly don't agree with the nuclear "freeze/delete the standard library" viewpoint.

odersky commented 8 years ago

@wheaties XML was put into a separate module, and that's what I would propose for a new JSON library as well. The days of a monolithic standard library are over. I would have loved to have Anti-XML as an alternative, in fact was trying to convince @djspiewak to contribute it, but it never got that far.

fommil commented 8 years ago

@odersky shapeless isn't needed for a fast parser, formatter or marshalling framework as spray-json demonstrates. But it is needed to automatically derive marshallers for user domain objects, as in spray-json-shapeless. Various other libraries have chosen to do this with their own macros, but fundamentally they are all doing the same thing. Without this, users need to provide a lot of boilerplate if they want these things to be defined and verified at compile time.

larsrh commented 8 years ago

If it's being put into a module, users would still need to declare it as a dependency in their build definition (just like for XML and parser combinators). What's the point prescribing one library when there are others out there which can be imported in exactly the same way?

milessabin commented 8 years ago

@odersky if it's done in the same way as Scala XML, separate repo, separate binary, different release cycles from the language distribution, then it's just another library. And if it's just another library then it should compete for mindshare on its own merits and not get a leg up from an "official" rubber stamp.

You should trust the community (which is much larger and more experienced than all of EPFL and Typesafe combined, by orders of magnitude) to make its own choices about which libraries to use.

odersky commented 8 years ago

@djspiewak Fair enough. But I want to re-iterate that we are talking about modules, not a monolithic jar. Modules can evolve at their own speed. There will be modules (such as the core) that are marked as stable, and we would not tolerate any major source incompatibilities between versions for them. Others, in particular more peripheral ones, could evolve at their own speed. And there would be space for alternatives in different libraries. My aim is simply to provide a least common denominator to get something acceptably useful out of the box. What constitutes that is a matter of debate and opinion.

OlegIlyenko commented 8 years ago

+1

this was my dream for quite a while now. Recently I was working on scala GraphQL library and had to provide adopters for different scala Json ASTs. To be honest, it becomes really frustrating to support them all (json4s, play-json, spray-json, argonaut, circe, etc.), especially considering how similar their AST representations are. Somehow reminds me of situation with futures a while ago.

In my case parsing/formatting of Json is not that important since library takes Json AST as an input and produces Json AST. Many of these Json libraries actually have very similar AST representation in form of case classes. I really wish for one unified AST which is adopted by different Json parsing/formatting libraries. I think this will make life much easier for libraries that need to work with Json, but don't do Json parsing/formatting themselves.

nafg commented 8 years ago

@larsrh good question. Maybe the answer is the "default distribution" -- i.e. the zip download?

davidhoyt commented 8 years ago

IMO, there is other work to be done and this is one thing that seems best to leave to the FOSS community to pursue. Put the expertise of Typesafe, EPFL, and other contributors to work in areas that will provide maximum benefit.

natewave commented 8 years ago

I wholeheartedly support @odersky's vision when it comes to a "batteries included" model. Scala should encourage newcomers and not just leave them there to the mercy of usually abandoned libs. I don't see any technical arguments supporting the other position, and thus it seems to me like a narrowed vision that can't get us anywhere. JSON is a big deal, there's no reason not to strive for support for it in stdlib.

A big +1

lrytz commented 8 years ago

If you're a newcomer to Scala, you don't want to just fiddle around with encoding your own List and writing merge sort.

This is not only about newcomers, I think it's really about the 90% of users / use cases. I cannot imagine that more people ever wanted to write their own List or merge sort.

Having something in the standard library doesn't prevent any other libraries (with different focuses, for the other 10%) from existing.

tixxit commented 8 years ago

A few concerns about a standard AST... I went to some lengths to ensure that arbitrary numbers were supported in Argonaut (and Circe by extension). This is not just using BigDecimal (eg spray-json), but also ensuring we only pay for the number type we use (eg don't force parsing to BigDecimal, if we're using Double). On top of this, it supports numbers that can't be stored in a BigDecimal, due to the limited exponent.

It's possible to create an AST that is open enough to allow these kind of optimizations, but I don't really hold out hopes that this will happen.

Edit: Just realized this is to make json4s-ast the standard. I'm actually pretty happy with that AST! That said, why not just promote json4s-ast? What other modules would benefit?

odersky commented 8 years ago

@milessabin You are thinking mindshare and competition, I am thinking usefulness "out of the box". Different concerns, obviously.

Also, your characterization of the SLIP committee as Typesafe + EPFL is somewhat disingenuous, given that a major Typelevel committer is on it.

milessabin commented 8 years ago

If you want a "batteries included" model make your choice(s) from the existing libraries, recommend them and put some effort and resources behind them.

fommil commented 8 years ago

I don't see any value in adding new modules to Scala. They are a good way to git rid of mistakes. Incorporating extremely stable codebases into the standard library has the advantage of getting in the door at large corporations with the language, thereby helping avoid the temptation to write it yourself. And if I've learnt anything from big corporates, it is that they have lots of very bad developers who just love to write things like JSON frameworks and logging APIs, but it's perhaps too late frankly.