search

antlr / antlr4

ANTLR (ANother Tool for Language Recognition) is a powerful parser generator for reading, processing, executing, or translating structured text or binary files.
http://antlr.org
BSD 3-Clause "New" or "Revised" License
17.11k stars 3.28k forks source link

splitting version numbers for targets #3010

Closed parrt closed 3 years ago

parrt commented 3 years ago

Hiya: @pboyer, @mike-lischke, @janyou, @ewanmellor, @hanjoes, @ericvergnaud, @lingyv-li, @marcospassos

Eric has raised the point that it would be nice to be able to make quick patches to the various runtimes; e.g., there is a stopping bug now in the JavaScript target. He proposes something along these lines:

This is in optimal as people have criticized me in the past for bumping, say, 4.6 to 4.7 for some minor changes. It also has the problem that 4.9.x will not mean the same thing in two different targets possibly, as each target will now have their own version number.

Rather than break up all of the targets into separate repositories or similar, can you guys think of a better solution? Any suggestions? The goal here is to allow more rapid target releases, and independent of me having to do a major release of the tool.

ewanmellor commented 3 years ago

I do think you should reserve the first three numbers for base Antlr. Given your project cadence, 4.x -> 5.x basically means "we wrote a new book", so you need the other two numbers for normal software engineering purposes.

To version the runtimes separately, the conventions that spring to mind are:

Maybe the least confusing, but the least conventional, would be to give each runtime their own letter codes, so it would be 4.9.0-js1 for patchlevel 1 in the JavaScript runtime, and 4.9.0-swift1 for patchlevel 1 in the Swift runtime.

Whichever you prefer, I think people are generally used to the idea that the fourth number onwards is a bit subtle, so I don't think having skew between runtimes in the fourth number is a big problem. I do think that 4.9.x being different between runtimes would be more likely to confuse people.

mike-lischke commented 3 years ago

Why not simply do a quick follow up release? If everything is automated it should not mean much extra work for you. If not then it is about time to make everything in the release process automatic. That's the general approach these days in the software industry anyway: release quick, release often.

marcospassos commented 3 years ago

As the PHP target is maintained under another repo to avoid these issues, feel free to decide whatever makes sense to you guys.

On Sat, Dec 19, 2020 at 07:01 Mike Lischke notifications@github.com wrote:

Why not simply do a quick follow up release? If everything is automated it should not mean much extra work for you. If not then it is about time to make everything in the release process automatic. That's the general approach these days in the software industry anyway: release quick, release often.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/antlr/antlr4/issues/3010#issuecomment-748451677, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHGHPDEQ6OBYP74PMMQRE3SVR2YRANCNFSM4VBXPYUA .

parrt commented 3 years ago

@mike-lischke unfortunately not everything is automated. I have to go through and update a lot of stuff manually (api docs, website, examples, ...) and the process seems to break every time I do it. maven continues to confuse the crap out of me. there is the issue with builds failing randomly due to timeout. There is the issue that I have to wait for appveyor to give me a decent build just so I can publish C#. I have to publish quite a few of the targets manually. granted I try to write all this down, but it definitely has a high transaction cost is my point 👍

parrt commented 3 years ago

@ewanmellor I absolutely suggested 4.9.0-js1 type version numbers but @ericvergnaud found out that NPM and a bunch of other target publishing stuff will have a problem with anything that isn't major.minor.patch :(

this is a really tough problem if you ask me. I can't think of a great solution

ewanmellor commented 3 years ago

Yes, that's true, NPM is going to need the third digit to change for the package management to do its job properly.

It sounds like you're going to need to move the middle digit for every platform release, there's really no other way.

ericvergnaud commented 3 years ago

A somewhat cumbersome way of limiting the damage could be to agree on a convention, where multi-target patch numbers increase by 10, whereas target-specific ones increase by 1. So ANTLR would evolve as follows: 4.9.10, 4.9.20, 4.9.30 ... Whereas targets would evolve as follows: 4.9.10, 4.9.11, 4.9.12 ... (assuming no target gets more than 9 patches for a given ANTLR release) (I'm not a great fan of my own idea, just trying to be creative)

mike-lischke commented 3 years ago

@parrt Most of the problems seem to stem from the runtimes, as they require environments and knowledge you don't have (why should you?). ANTLR4 itself doesn't depend on any runtime, but it makes sense, of course, to bundle the Java runtime with it.

The individual runtimes are separate and can be released on their own. This would require to set up ANTLR4 repos for each runtime where only the tests specific for the particular runtime are executed. You would then remove the runtimes from the main repo and release both runtimes + tool separately. They are 2 very different pieces anyway. The only tricky part here is to have a shared source of all tests, so that you don't have to maintain them in several places (maybe a git submodule?).

Your task would only be to take care for the tool + Java target. Target authors have to take care for their language and can publish new releases at any time, independently. Version numbers would not correspond between tool and targets. Also, target authors could be allowed to merge patches on their own and don't need your intervention.

Github supports releases for a given repo and that would be your source of the runtimes when you want to publish them on the ANTLR4 homepage. But people can, at any time download a previously released version on a brand new one, even if you didn't do the full release yet.

How does that sound to you?

parrt commented 3 years ago

@mike-lischke Yep. I considered splitting apart the repository into multiple, one per target, but it's kind of drastic. And, at the beginning, I made the decision to put them all into one for ease of testing and to keep the runtimes and the tool consistent. Ha ha :)

@ericvergnaud not a bad idea. What if we took it further and made a digit for the runtime, but that would limit us to 10 runtimes. or maybe we use hexadecimal digits so we get 16 targets total? not sure npm or other tools could handle the hexadecimal though.

I am strongly motivated to delegate control of the targets to the respective maintainers, but I'm concerned about tool-target version matching for consistency. We also have to modify all of the testing infrastructure, but that is workable if we assume that all target repositories follow a naming scheme and are at the same level as the antlr repo. The git submodule thing really was driving me crazy for the PHP runtime; I literally am not smart enough to use git it to its fullest (or maven for that matter. haha).

ewanmellor commented 3 years ago

I'm pretty sure hex version numbers is going to be a non-starter.

Git submodules shouldn't be a problem. It's definitely a model you have to sit down and learn, but it's a rational model once you get it.

Maven is just unalloyed garbage, but then all the Java build systems are bad, so it's not like switching is going to make you happy.

The independent repos idea scares me a bit in this case, given how intertwined all the tests stuff is. You could do it, but I think you'll have to consider antlr-runtime-test-framework as a real package with a real interface with real dependents, so that each of the runtimes could consume it without constantly breaking.

lingyv-li commented 3 years ago

I feel that splitting patch versions doesn't mean that we have to separate the repos. It will be a major effort to update the testing framework, and require more effort to coordinate minor version upgrade if we separate repos and ownership. From a user's perspective, I like how we can generate code for all targets using the same command and IDE plugin. If some of the targets it generate don't have a runtime that supports it, that will be a major drawback.

Dart runtime can be released independently already. Just bump the patch version in pubspec.yaml and publish a release. Though that might cause confusion for users who sees inconsistent versions. So I don't see how separating repos can make it easier. For Dart target, the two things that needs updating during releases are the version check metadata and the package version. Both can be automated with RegEx. Unfortunately, the releasing process cannot be automated for now: dart-lang/pub#2227

This is in optimal as people have criticized me in the past for bumping, say, 4.6 to 4.7 for some minor changes. It also has the problem that 4.9.x will not mean the same thing in two different targets possibly, as each target will now have their own version number.

Right now major + minor version is used to check codegen and runtime compatibility. I think it still makes sense to make a release if the new codegen doesn't work for the old runtime. The 4.9 release for Dart probably isn't necessary under that criteria haha.

ericvergnaud commented 3 years ago

@lingyv-li seems what cannot be automated is publishing from a CI, and the problem mentioned is pretty much the same for any target (I also publish some other stuff to GitHub and Maven using a local script which reads non-committed files containing credentials). @parrt would you be able to describe what steps require manual intervention and maybe we can collectively address these (I agree with @mike-lischke that all this should be automated) Then releasing/publishing simply becomes a matter of running a script from your laptop?

lingyv-li commented 3 years ago

Yes, it cannot be fully automated especially the dart publish command literally opens a web page to login to the Google account. In npm for example, you can use access tokens which can be stored securely in the CI.

I imagine the ideal state to be using tools like semantic-release or auto, which can make the version bump and release fully automated in CI with PR labels. Though that's not possible right now at least for Dart runtime, but we should try to get as close as we can.

parrt commented 3 years ago

Hiya. Ok, it seems like we are converging on not splitting the repository, so at least that some progress. :) The human script that I use an update is here for releasing:

https://github.com/antlr/antlr4/blob/master/doc/releasing-antlr.md

A lot of it has to do with publishing the targets, which I guess is what we are trying to separate out. All I know is that every time I try to go through script, that something is different and I have all sorts of problems with different websites and CI and using maven properly and updating submodules etc..

As for submodules, I'm afraid I'm not willing to put the effort in the learn that crap. git is already shit in my mind and I don't want any more pollution. ;) Especially something I will never use again or elsewhere.

Maybe you guys can have some ideas after looking at that release script, but it still leaves the primary problem of version numbers.

ewanmellor commented 3 years ago

Speaking of CI and automation as we were, it looks like the Antlr CI is still using travis.org, which is being shut down in 8 days:

https://mailchi.mp/3d439eeb1098/travis-ciorg-is-moving-to-travis-cicom?e=%5BUNIQID%5D

mike-lischke commented 3 years ago

Migrating to travis.com actually...

And from the message I got there this should happen automatically.

KvanTTT commented 3 years ago

Maybe it makes sense to migrate to GitHub Actions.

parrt commented 3 years ago

I think we are already migrated to the .com: Installed 3 months ago Developed by travis-ci https://travis-ci.com. Actually, that looks like the organization is there but not all of the repositories. Let me see if I can transfer from .org

parrt commented 3 years ago

wow. I can't seem to add antlr org to travis-ci.com. I must be getting dumber by the week. it seems that I can add my parrt account but not the antlr organization, at least not properly. I think it said it queued removal of some old stuff so I will let this sit for a day.

@KvanTTT Where do the github actions run? Do we have to pay for it?

ericvergnaud commented 3 years ago

I expect Travis themselves would migrate accounts? Worth checking with them if that’s the case?

Le 26 déc. 2020 à 04:26, Terence Parr notifications@github.com a écrit :

wow. I can't seem to add antlr org to travis-ci.com. I must be getting dumber by the week.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/antlr/antlr4/issues/3010#issuecomment-751288064, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZNQJHARR2B27O25CZ6ACDSWTYO3ANCNFSM4VBXPYUA.

mike-lischke commented 3 years ago

I also had trouble when I opened my Github account in travis.com. Not all of my repos are listed there and instead they list now all the repos from the organizations I'm in, which is definitely not what I want. So I removed everything from there and now I'm waiting that travis.org is automatically migrated to travis.com.

KvanTTT commented 3 years ago

Where do the github actions run? Do we have to pay for it?

They work on GitHub servers and available on the "Actions" tab in a repository. It's free, @teverett has already integrated GitHub actions into grammars repository: https://github.com/antlr/grammars-v4/actions. He knows better how to integrate.

parrt commented 3 years ago

@mike-lischke ok, that's what I'll do though I think I messed that up by disconnecting too much stuff between github and travis. :(

parrt commented 3 years ago

Ah. I think i figured it out. shitty interface on github. antlr/antlr4 is hooked up to travis-ci.com now.

teverett commented 3 years ago

Grammars-v4 is now using GH Actions.

Terence Parr mailto:notifications@github.com December 26, 2020 at 11:03 AM

Ah. I think i figured it out. shitty interface on github. antlr/antlr4 is hooked up to travis-ci.com now.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/antlr/antlr4/issues/3010#issuecomment-751380792, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJLKZVF6JBBEZGWOYFSLXLSWYQOJANCNFSM4VBXPYUA.

-- Tom Everett tom@khubla.com mailto:tom@khubla.com https://www.postbox-inc.com

parrt commented 3 years ago

@teverett so can it do the complex builds we do for antlr tool with testing etc...? I see travis-ci.org still connected to the grammars-v4 repo, btw. this looks interesting:

https://github.com/antlr/grammars-v4/actions

Oh and i see this:

https://github.com/antlr/grammars-v4/blob/master/.github/workflows/main.yml

boy that sure looks like our travis yml file. Can you tell us more how well it works? Compare to travis?

teverett commented 3 years ago

I honestly don't know.  Grammars-v4 was easy to port b/c it's basically "run this maven build".   I do end up paying GH once in a while, less than $5 CDN for build capacity for grammars-v4.

Terence Parr mailto:notifications@github.com December 27, 2020 at 10:39 AM

@teverett https://github.com/teverett so can it do the complex builds we do for antlr tool with testing etc...? I see travis-ci.org still connected to the grammars-v4 repo, btw. this looks interesting:

https://github.com/antlr/grammars-v4/actions

Oh and i see this:

https://github.com/antlr/grammars-v4/blob/master/.github/workflows/main.yml

boy that sure looks like our travis yml file. Can you tell us more how well it works? Compare to travis?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/antlr/antlr4/issues/3010#issuecomment-751495379, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJLKZWK5MI6X4IFXURZ2XDSW5WL3ANCNFSM4VBXPYUA.

-- Tom Everett tom@khubla.com mailto:tom@khubla.com https://www.postbox-inc.com

parrt commented 3 years ago

Ok, thanks Tom.

KvanTTT commented 3 years ago

As for submodules, I'm afraid I'm not willing to put the effort in the learn that crap. git is already shit in my mind and I don't want any more pollution. ;) Especially something I will never use again or elsewhere.

I also think that submodules using is a good idea. On one hand, they solve the problem of runtimes splitting, on another hand, they can be synchronized easily. I use UI for git (GitExtensions) and submodules representation is very straightforward there. Maybe using a console is a bit confusing if use submodules.

parrt commented 3 years ago

I'm mainly worried about the version numbers and I just can't think of a good solution. Definitely seeing 4.9.1 Java and 4.9.2 Python versions could look weird and we would need to adjust to the version compatibility mechanism so that it only looked at the "4.9". There are tools that have different versions for different pieces of the same project, which is confusing, but at least they are very different numbers. What would you guys think about using 4.x.y for ANTLR too+java and then using completely different version numbers for the targets; maybe we start them all at 1.0? That way it's more complex but not confusing. Hmm... it would make the version compatibility check more complicated actually but should be simple enough to get a mapping.

ericvergnaud commented 3 years ago

Mmm I believe everybody out there perceives the last bit of a version number to express a patch rather than a full fledged version. Some people may look for say Python runtime 4.9.6 if they find JavaScript runtime 4.9.6 in npm (and not find the Python one) but I don’t perceive this as an issue if it’s properly documented. AFAIK the compatibility check does not yell for patch number mismatches does it? I would hate to have to remember that tool 4.10 requires JS runtime 2.3 and Python runtime 6.1.

Envoyé de mon iPhone

Le 29 déc. 2020 à 01:31, Terence Parr notifications@github.com a écrit :

 I'm mainly worried about the version numbers and I just can't think of a good solution. Definitely seeing 4.9.1 Java and 4.9.2 Python versions could look weird and we would need to adjust to the version compatibility mechanism so that it only looked at the "4.9". There are tools that have different versions for different pieces of the same project, which is confusing, but at least they are very different numbers. What would you guys think about using 4.x.y for ANTLR too+java and then using completely different version numbers for the targets; maybe we start them all at 1.0? That way it's more complex but not confusing. Hmm... it would make the version compatibility check more complicated actually but should be simple enough to get a mapping.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

parrt commented 3 years ago

Yeah, different version numbers are no matter what going to cause confusion. having radically different version numbers as you say would be more complex and confusing. dang.

You know the simplest solution here is simply to bump a single version, as we have now, when any of the targets change. That's more or less exactly what we do now. The difference would be that we need a way for individual targets to bump the overall version and also do the formal release, including updating the website and so on. ugh.

parrt commented 3 years ago

I'm going to try to get 4.9.1 released this weekend. Please bless any PRs for your various targets. JavaScript has an immediate need for release...

parrt commented 3 years ago

@ericvergnaud I'm going to try to tag PRs / bugs with 4.9.1 today and make a release

KvanTTT commented 3 years ago

Maybe it makes sense to merge #3018, #3019, #3021 pull requests. Also, I'm going to take a look at #3024 and I'm working on #2361.

parrt commented 3 years ago

trying to do a delete but once again git (may it die a horrible death) submodules are unhappy again:

beast:master:~/antlr/code/antlr4 $ git submodule update --recursive
beast:master:~/antlr/code/antlr4 $ git submodule update --remote --merge # might only need this last one but do both
beast:master:~/antlr/code/antlr4 $ vi runtime/PHP/src/RuntimeMetaData.php
beast:master:~/antlr/code/antlr4 $ 
(reverse-i-search)`': 
beast:master:~/antlr/code/antlr4 $ git add runtime/PHP
beast:master:~/antlr/code/antlr4 $ git commit -m "Update PHP Runtime to latest version"
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  (commit or discard the untracked or modified content in submodules)
    modified:   runtime/PHP (modified content)

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    antlr4-maven-plugin/gen/
    runtime-testsuite/gen/
    runtime/gen/

no changes added to commit (use "git add" and/or "git commit -a")
ewanmellor commented 3 years ago

runtime/PHP is a submodule. You need to do a commit inside the submodule.

cd runtime/PHP
git commit -m "Update PHP Runtime to latest version"
git push
cd ../..
git add runtime/PHP
git commit -m "Sync with latest PHP runtime"
git push
parrt commented 3 years ago

I tried that as well:

beast:master:~/antlr/code/antlr4 $ cd runtime/PHP
beast:~/antlr/code/antlr4/runtime/PHP $ git commit -m "Update PHP Runtime to latest version"
HEAD detached from 62fd3c3
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
    modified:   src/RuntimeMetaData.php

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    .fetch_time_cache

no changes added to commit (use "git add" and/or "git commit -a")
parrt commented 3 years ago

ah. -a helps

mike-lischke commented 3 years ago

It's not clear to me why you would need git add. When you pull the repo recursively, it will automatically also pull the submodule and hence have the latest code on disk. What is this git add for?

parrt commented 3 years ago

git commit -a -m "Update PHP Runtime to latest version" requires -a, although I thought that the git add would do it. changing the documentation

parrt commented 3 years ago

do I need to do a push inside?

mike-lischke commented 3 years ago

yes, see here: https://devconnected.com/how-to-add-and-update-git-submodules/ ("Fetch new submodule commits")

parrt commented 3 years ago

it appeared to do nothing as it says I'm up-to-date

parrt commented 3 years ago

aannnnnndddd we are back to the usual situation

beast:master⤒:~/antlr/code/antlr4 $ git status
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
  (use "git push" to publish your local commits)

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  (commit or discard the untracked or modified content in submodules)
    modified:   runtime/PHP (untracked content)

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    antlr4-maven-plugin/gen/
    runtime-testsuite/gen/

no changes added to commit (use "git add" and/or "git commit -a")
parrt commented 3 years ago

nothing to commit and nothing to push and yet the main repository thanks it is dirty

parrt commented 3 years ago

I wonder if there are people out there that think that ANTLR is as big of a piece of SHIT as I think git is

mike-lischke commented 3 years ago

Try a GUI client, which does all of this automatically (https://fork.dev/). Saves me brain cancer since a long time :-D

ewanmellor commented 3 years ago 
Your branch is ahead of 'origin/master' by 1 commit.
  (use "git push" to publish your local commits)

Why do you say "nothing to push"? There is clearly something to push.

parrt commented 3 years ago

@ewanmellor I meant that it is saying that the submodule is still dirty and has commits:

beast:master⤒:~/antlr/code/antlr4 $ git push origin master
Enumerating objects: 170, done.
Counting objects: 100% (170/170), done.
Delta compression using up to 8 threads
Compressing objects: 100% (70/70), done.
Writing objects: 100% (87/87), 6.48 KiB | 1.08 MiB/s, done.
Total 87 (delta 59), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (59/59), completed with 58 local objects.
To github.com:parrt/antlr4.git
   9c7b56361..eaa976e42  master -> master
beast:master:~/antlr/code/antlr4 $ 
beast:master:~/antlr/code/antlr4 $ git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  (commit or discard the untracked or modified content in submodules)
    modified:   runtime/PHP (untracked content)

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    antlr4-maven-plugin/gen/
    runtime-testsuite/gen/

no changes added to commit (use "git add" and/or "git commit -a")