search

whitequark / parser

A Ruby parser.
Other
1.59k stars 199 forks source link

Soft deprecation of the project? #1046

Open iliabylich opened 3 days ago

iliabylich commented 3 days ago

So, Prism is out and it's:

  1. correct
  2. comes out of the box with MRI (and IIRC with other Ruby implementations)
  3. faster (I think on TruffleRuby/JRuby it's absurdly faster because it doesn't run in interpreted mode like it does with parser)
  4. always up-to-date
  5. written in a language with perfect interop capabilities

On the other side backporting changes to parser becomes more and more difficult. MRI doesn't use yacc/bison anymore, instead there's Lrama that is also an LALR(1) parser generator but it has more "syntax sugar" that Racc doesn't have, and so the grammar will diverge more and more over time. And if (or once) parse.y is deprecated it will be even more complex to "translate" grammar changes.

I personally believe that parser should be soft-deprecated. What do you guys think? @kddnewton @bbatsov @koic @palkan @marcandre @mbj

And if you agree do you see anything we could do to parser to make transition easier?

mbj commented 3 days ago

The real use case of this gem left for me is multi version capability. Prism is not shipped with all the ruby versions my use cases still have to support, basically all non EOL (MRI) rubies.

If this gem where to stop supporting future ruby versions I'd be fine. Cannot speak for the rest of the ecosystem OFC.

kddnewton commented 3 days ago

In general I think this is the right long-term direction, but I'm uncomfortable soft deprecating until we mirror the tree rewriting functionality, since so many consumers use that. I think it could be added to the Ruby API of Prism relatively easily, but I haven't had time to look into this yet.

I'm not sure if there are any other APIs that I'm not thinking about, but that's the main one I know we are missing.

bbatsov commented 3 days ago

Indeed. I don't see how we can deprecated parser without the rewriting functionality. RuboCop also relies on the multi-version capability quite a lot, although I guess we can live without it if push comes to shove.

One other question - if parser gets soft deprecated will it still be updated to track newer Ruby releases? I think that a realistic time-frame for full migration to Prism is probably 2-3 years (for the projects that are still active, that is), so parser will definitely require some maintenance is the mean time.

iliabylich commented 3 days ago

One other question - if parser gets soft deprecated will it still be updated to track newer Ruby releases?

My initial thought was to stop doing it. At this point parser looks to me a lot like a (bad) polyfill and extending it more to be compliant with new grammar features feels like a disservice to the community.

mbj commented 3 days ago

I think we need to put evolutionary pressure on and NOT release compatibility with new versions.

bbatsov commented 3 days ago

I'm not super fond of measures that will be quite disruptive to the end users - to me it seems just wild to stop supporting something without providing a complete alternative for it. E.g. what happens if the rewriting is still not supported by Prism when Ruby 3.5 comes out?

At this point parser looks to me a lot like a (bad) polyfill and extending it more to be compliant with new grammar features feels like a disservice to the community.

Breaking existing apps is not exactly in the service of the community either. ;-)

I'm not arguing that parser should go away - I'm just concerned about abandoning the project prematurely and causing pain for the users of tool depending on it.

mbj commented 3 days ago

To play a bit of devils advocate here: How is not supporting a new future ruby release "breaking existing apps"?

bbatsov commented 3 days ago

Okay, "causing problems for the existing apps". I'm fairly certain RuboCop's users won't be exactly happy if we tell them we don't really support newer Ruby releases for whatever reasons. :-) And I'm not sure that simply stopping work on parser will motivate everyone to implement whatever is missing, etc. Also - how is the translation layer (Prism -> Parser) going to work when Parser stops tracking upstream changes and the available nodes in both libraries are out of sync?

Moving fully to Prism's native AST is going to be quite the time investment for many projects. I'd like for us to get there, but ideally I don't want us to be pressured into a timeline for this given that everyone working on RuboCop does this in their spare time.

mbj commented 3 days ago

Is there a translation layer from prosm to this parsers AST somewhere?

palkan commented 3 days ago

Is there a translation layer from prism to this parsers AST somewhere?

Prism supports AST translation for popular parsers: https://github.com/ruby/prism/tree/main/lib/prism/translation

And if (or once) parse.y is deprecated it will be even more complex to "translate" grammar changes.

I think, deprecating Parser as soon as parse.y is deprecated makes sense. Otherwise the effort required to port Ruby changes to Parser would be enormous.

how is the translation layer (Prism -> Parser) going to work when Parser stops tracking upstream changes and the available nodes in both libraries are out of sync?

From my experience, adding new nodes could be done relatively easy (compared to upgrading the grammar and lexer). I see, how, in theory, we can use Prism for newer Ruby versions as a parser and a Parser-compatible AST builder.

So, if we can keep Parser Ruby internals and APIs work in new Ruby versions (without providing the corresponding syntax support), that would smooth the transition.

To sum up, I would treat soft-deprecation as follows: stop adding new rubyXY.y files, but maintain compatibility with potential Ruby changes (I mean something breaking, like kwargs separation, etc.).

iliabylich commented 3 days ago

To sum up, I would treat soft-deprecation as follows: stop adding new rubyXY.y files, but maintain compatibility with potential Ruby changes (I mean something breaking, like kwargs separation, etc.).

Yes, that's more or less what I proposed (probably the word "deprecate" was a bit misleading). I'm not proposing to yank parser from rubygems.org, only to stop backporting new grammar changes.

I personally never touched rewriting API, shouldn't it "just work" with an AST constructed by Prism's compatibility layer?

Breaking existing apps is not exactly in the service of the community either. ;-)

My understanding is that it shouldn't break anything. If you are on Ruby 3.3+ use Prism (probably with a compatibility layer but it will be a part of the tooling, fully transparent for end users), otherwise use legacy parser gem. Or is it going to be a nightmare in terms of configuration and testing?

palkan commented 3 days ago

shouldn't it "just work" with an AST constructed by Prism's compatibility layer?

Yeah, it works right now. Well, at least, for Ruby Next.

I think, what could be done in terms of ensuring this compatibility is to make it possible to run rewriter tests via Prism and the translation layer. I can take a look at this (since I've already done the same for Ruby Next).

mbj commented 3 days ago

Is there an unparser equivalent yet?

kddnewton commented 3 days ago

Truthfully I don't know if parse.y is going to be deprecated, or if that will happen any time soon at all. I imagine it could stick around for quite a long time. That being said, it is getting further away from strict bison/racc syntax, so I know it's quite a maintenance burden to hold on to.

My understanding is that it shouldn't break anything. If you are on Ruby 3.3+ use Prism (probably with a compatibility layer but it will be a part of the tooling, fully transparent for end users), otherwise use legacy parser gem. Or is it going to be a nightmare in terms of configuration and testing?

I think this would work, but we will want to test the translation layer even more. I think there are issues with string escapes at the moment, but I'm not sure.

I think, what could be done in terms of ensuring this compatibility is to make it possible to run rewriter tests via Prism and the translation layer. I can take a look at this (since I've already done the same for Ruby Next).

That would be really great.

Is there an unparser equivalent yet?

I haven't managed to get it over the finish line yet, but yes https://github.com/ruby-syntax-tree/syntax_tree-prism is that work. It's almost done.

iliabylich commented 3 days ago

Is there an unparser equivalent yet?

Probably no, but I've found "untokenizer" made by @koic 😄

bbatsov commented 2 days ago

My understanding is that it shouldn't break anything. If you are on Ruby 3.3+ use Prism (probably with a compatibility layer but it will be a part of the tooling, fully transparent for end users), otherwise use legacy parser gem. Or is it going to be a nightmare in terms of configuration and testing?

To me it seems that'd be quite painful, especially if Prism doesn't support specifying target Ruby versions. And by "break" I mostly refer to "break the way" things are happening right now, not so much actually breaking existing apps. (and force clients to take some steps to ensure their tools will work with future Rubies that are not supported by parser) Anyways, I guess me and the rest of our team will have some thinking to do about the future and the next steps on our end.

To summarize, for us the biggest problems right now are:

I'm guessing that @koic and @marcandre will have more to add on the subject.

mbj commented 2 days ago

Probably no, but I've found "untokenizer" made by @koic 😄

unfortunately not enough for my use case.

@bbatsov Okay so I saw a world where parser would stop to add new ruby version support, and everyone who needs newer ruby version support would than conditionally use Prism via a compatibiltiy layer. But your use case of running one ruby version and targeting another breaks this option.

All my use cases would be covered but not yours, curious if there are plans for a version target @kddnewton?

marcandre commented 2 days ago

I think the rewriter could definitely be included in rubocop-ast, or even better be a separate gem altogether.