search

antlr / antlr4

ANTLR (ANother Tool for Language Recognition) is a powerful parser generator for reading, processing, executing, or translating structured text or binary files.
http://antlr.org
BSD 3-Clause "New" or "Revised" License
17.11k stars 3.28k forks source link

splitting version numbers for targets #3010

Closed parrt closed 3 years ago

parrt commented 3 years ago

Hiya: @pboyer, @mike-lischke, @janyou, @ewanmellor, @hanjoes, @ericvergnaud, @lingyv-li, @marcospassos

Eric has raised the point that it would be nice to be able to make quick patches to the various runtimes; e.g., there is a stopping bug now in the JavaScript target. He proposes something along these lines:

This is in optimal as people have criticized me in the past for bumping, say, 4.6 to 4.7 for some minor changes. It also has the problem that 4.9.x will not mean the same thing in two different targets possibly, as each target will now have their own version number.

Rather than break up all of the targets into separate repositories or similar, can you guys think of a better solution? Any suggestions? The goal here is to allow more rapid target releases, and independent of me having to do a major release of the tool.

ewanmellor commented 3 years ago

The submodule is dirty. It has .fetch_time_cache there untracked.

parrt commented 3 years ago

Yes, but it is okay to have untracked files right?

parrt commented 3 years ago

intellij gui git tool says nothing to push

ewanmellor commented 3 years ago

Sure. I'd add it to the submodule's .gitignore myself, since it's clearly a cache, but it won't hurt if it's just sitting there.

parrt commented 3 years ago

so at this point I have tried all sorts of combinations of committing and pushing from various directories and subdirectories. status still shows that the PHP runtime has changes not staged for commit.

clearly this is going to happen every time I do a release which is why I specifically said no goddamn submodules.

I am open to suggestions but this is the last time I am ever going to do this so we need a new solution.

parrt commented 3 years ago

PS: I'm almost ready to go with 4.9.1

ewanmellor commented 3 years ago

@parrt Add .fetch_time_cache to the submodule's .gitignore. Commit and push that. Go to the parent module, commit and push the change to the submodule. Then the submodule will report as clean.

The bigger picture issue is that you're trying to release administrative control over runtime/PHP (i.e. making it have independent history) while at the same time having a huge dependency knot between Antlr, its runtime support library, its runtime tests, and the runtimes. Until you linearize these dependencies, no amount of submoduling or independent versioning is going to work properly.

parrt commented 3 years ago

@ewanmellor you win! that cleaned it up. hilarious.

Yep, with all of these different runtimes and build systems and dependencies, we have a serious mess. My only suggestion is that we limp along going forward as this is no longer my focus.

marcospassos commented 3 years ago

Unfortunately, I’m not a Git master either. Also, Composer does not support nested files, and doesn't seem to have plans to support. So, if we don't find any way to make this workflow work, the only way I see is manually copying the PHP runtime everytime or excluding it from the ANTLR release.

On Sun, Jan 3, 2021 at 15:22 Terence Parr notifications@github.com wrote:

PS: I'm almost ready to go with 4.9.1

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/antlr/antlr4/issues/3010#issuecomment-753657044, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHGHPHJXL6ETNRZTU454P3SYCYYDANCNFSM4VBXPYUA .

mike-lischke commented 3 years ago

I can only repeat my suggestion: take the runtimes out and let them exist separately, release them separately, test them separately, make separate downloads for them (oh that one exists already) etc.

The only problem you (or rather the runtime owners) have then to solve is how to test all runtimes with the same set of test cases, efficiently. But even if tests for a runtime fail, it's no longer your business Ter. The owner has to fix it.

And while you are at sourcing out this development, it might be time to find someone from the community to take care for ANTLR in the future, so you are not bound to it forever. Priorities shift and nobody can forever support a project. Maybe you can get it under the Apache Org roof?

parrt commented 3 years ago

yeah, what you say makes sense Mike, but would be a radical change. I'd have to think about splitting this up.

parrt commented 3 years ago

on an unrelated note, I literally cannot get travis-ci.com to see antlr/antlr4 repo. It sees a bunch of the other but not the one I want. so frustrating trying to hook this up. I deleted and reinstalled the github thing multiple times.

mike-lischke commented 3 years ago

Yes, I had the same issue and removed everything from travis-ci.com, hoping for an automatic solution in the future.

parrt commented 3 years ago

oh wow. you need to join the beta program... which I just asked for. ugh. Ok, it looks like it's trying to do it now. triggering test build now.

https://travis-ci.com/github/antlr/antlr4/builds/211699638

ewanmellor commented 3 years ago

I see it here: https://travis-ci.com/github/antlr/antlr4/builds/211698627, there's a build going on right now.

It did kind of glitch, like it wasn't in the list initially and then it found it.

parrt commented 3 years ago

Travis is now hooked up but it can't find the submodule (lovely)

https://travis-ci.com/github/antlr/antlr4/jobs/467886352:

$ git clone --depth=50 --branch=master https://github.com/antlr/antlr4.git antlr/antlr4
5.97s$ git submodule update --init --recursive
Submodule 'runtime/PHP' (https://github.com/antlr/antlr-php-runtime.git) registered for path 'runtime/PHP'
Cloning into '/home/travis/build/antlr/antlr4/runtime/PHP'...
fatal: remote error: upload-pack: not our ref 5f4930012c1601b0755ae5fc48828c3b0f3823d1
Fetched in submodule path 'runtime/PHP', but it did not contain 5f4930012c1601b0755ae5fc48828c3b0f3823d1. Direct fetching of that commit failed.
The command "eval git submodule update --init --recursive " failed. Retrying, 2 of 3.
ewanmellor commented 3 years ago

You haven't pushed the most recent commit in the submodule.

parrt commented 3 years ago

hahah. you sure? I could swear that I did do that. It also says that I am detached now when I go back into that directory. shit.

$ git status HEAD detached from 62fd3c3

parrt commented 3 years ago

Ok, so let's just add one more line to my human script which is to cp -r the PHP runtime over. I will have to jump over real quick and manually update the version number in that repository but that's no big deal. Is this the proper procedure for undoing submodules? https://gist.github.com/myusuf3/7f645819ded92bda6677

ewanmellor commented 3 years ago

If the submodule isn't checked out to an particular branch (i.e. it is a "detached HEAD") then you need to tell it which branch to push to.

Assuming that the correct branch is master, you probably want

git push origin HEAD:master

If you're going to be working on the submodule regularly, I'd check out the submodule onto the correct development branch (or master if that's right).

parrt commented 3 years ago

at this point I'm going to remove it if I can find a way to do it safely. ok, I have unhooked it using the GUI (Fork, actually a very nice tool).

parrt commented 3 years ago

I just realized that removing the submodule is going to break the testing of PHP as part of travis. On the other hand, it also has a build sequence that triggers upon push to its repository:

https://travis-ci.com/github/antlr/antlr-php-runtime/jobs/467888291

parrt commented 3 years ago

Looks like swift is failing at travis:

https://travis-ci.com/github/antlr/antlr4/jobs/467890059

(They seem to be working on my local mac though)

ewanmellor commented 3 years ago 
/Users/travis/build/antlr/antlr4/runtime/Swift/Tests/Antlr4Tests/RuntimeMetaDataTests.swift:30: error: -[Antlr4Tests.RuntimeMetaDataTests testGetMajorMinorVersion] : XCTAssertEqual failed: ("4.9") is not equal to ("4.9.1")

Looks like it needs a version number bump somewhere.

parrt commented 3 years ago

dang. ok. thanks. It's in there:

public static let VERSION: String = "4.9.1"

https://github.com/antlr/antlr4/blob/master/runtime/Swift/Sources/Antlr4/RuntimeMetaData.swift#L66

Maybe that build was old? Nope. current. also weird it works locally. ah. i see one small booboo and will try again.

ewanmellor commented 3 years ago

Yes, I see your change, that's the right one I think.

parrt commented 3 years ago

Crap. PHP tests are still attempted (on appveyor). trying to add php back and clone repo during install.

heh, swift is looking better https://travis-ci.com/github/antlr/antlr4/builds/211707047

parrt commented 3 years ago

Travis: "Builds have been temporarily disabled for public repositories due to a negative credit balance. Please go to the Plan page to replenish your credit balance or alter your Consume paid credits for OSS setting."

clicking on the links gives site error. hahaha.

Damn. ok, we have now consumed all credits with a few hours playing:

"-2810 available credits (purchase date: January 03, 2021) You have used 12810 of 10000 credits"

parrt commented 3 years ago

Looks like travis is now out. github actions? anybody want to take a crack at that? @KvanTTT ??

KvanTTT commented 3 years ago

I can try to integrate next week.

ewanmellor commented 3 years ago

https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing: "1 minute of mac build time costs 50 credits"

So the free plan now gives 2.5 test runs per month for our Swift tests alone. Wow.

It didn't even get a chance to run our extended test suite, which currently has macOS tests for .NET and C++ at approximately the same length.

Even if you disabled the macOS tests entirely and just ran the Linux ones, I think you're looking at 7 or 8 full test runs a month before the free plan caps out.

mike-lischke commented 3 years ago

Does this mean free Travis is going away, effectively? That would be a bummer.

ericvergnaud commented 3 years ago

It’s not going away, but the limitations make it unusable for ANTLR.

Le 4 janv. 2021 à 15:31, Mike Lischke notifications@github.com a écrit :

Does this mean free Travis is going away, effectively? That would be a bummer.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/antlr/antlr4/issues/3010#issuecomment-753808827, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZNQJBZ4VUL26QD42IJGWTSYFVGLANCNFSM4VBXPYUA.

parrt commented 3 years ago

I'll have to look at github actions (costs money). Can we get away with just AppVeyor?

parrt commented 3 years ago

I'm reading this on migrating to github actions: https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/migrating-from-travis-ci-to-github-actions

ericvergnaud commented 3 years ago

The problem I have with GitHub actions is the long history of Microsoft trapping people into a free product and then suddenly making it expensive… (appreciate that’s exactly what just happened with Travis CI on Mac)

I think the cost of migrating our CI from Travis to any CI will be small, rather I would suggest we focus on identifying the right one.

Travis, Github, AppVeyor, Gitlab all offer Linux/Wintel/MacOS builds (and there’s probably others out there)

I’d suggest, once 4.9.1 is out that we list them all and do a qui pros-and-cons. Happy to drive that effort.

Le 5 janv. 2021 à 07:07, Terence Parr notifications@github.com a écrit :

I'm reading this on migrating to github actions: https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/migrating-from-travis-ci-to-github-actions https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/migrating-from-travis-ci-to-github-actions — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/antlr/antlr4/issues/3010#issuecomment-754276244, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZNQJAB4YJNCB37P5N6UFTSYJCZXANCNFSM4VBXPYUA.

parrt commented 3 years ago

@ericvergnaud it would be great if you could leave that discussion after I push out 4.9.1 thanks

parrt commented 3 years ago

Ok, finishing up maven release and will update website etc... soon. closing this.

parrt commented 3 years ago

Anybody know npm? 

beast:master:~/antlr/code/antlr4/runtime/JavaScript $ npm run build

> antlr4@4.9.1 build /Users/parrt/antlr/code/antlr4/runtime/JavaScript
> webpack

sh: webpack: command not found
npm ERR! code ELIFECYCLE
npm ERR! syscall spawn
npm ERR! file sh
npm ERR! errno ENOENT
npm ERR! antlr4@4.9.1 build: `webpack`
npm ERR! spawn ENOENT
npm ERR! 
npm ERR! Failed at the antlr4@4.9.1 build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm WARN Local package.json exists, but node_modules missing, did you mean to install?

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/parrt/.npm/_logs/2021-01-05T21_02_05_436Z-debug.log
mike-lischke commented 3 years ago

Have you run "npm install" first?

parrt commented 3 years ago

hahah. I just figure that out and added it to the documentation after a web search. ugh. thanks Mike!

parrt commented 3 years ago

folks, we have 4.9.1 out the door :) thanks for the help!!

KvanTTT commented 3 years ago

The problem I have with GitHub actions is the long history of Microsoft trapping people into a free product and then suddenly making it expensive…

I don't think that we should be biased against any vendor. In this case, we should migrate to GitLab or another service because GitHub is also under Microsoft. In the end, any company tries to earn more money and there is no "Corporation of Good" in the modern world (even Google is not such a corporation). By the way, GitHub private repositories became free after GitHub had been acquired by Microsoft. If Microsoft or any other company became paid or expensive, we could switch to another service.

Also, I think we can try to ask for some advanced resources for free because ANTLR is a valuable project.

teverett commented 3 years ago

@parrt congrats on 4.9.1. the test plugin and grammars-v4 are updated.