antlr / antlr4

ANTLR (ANother Tool for Language Recognition) is a powerful parser generator for reading, processing, executing, or translating structured text or binary files.
http://antlr.org
BSD 3-Clause "New" or "Revised" License
17.03k stars 3.27k forks source link

Critical bug: Go target 4.13.1 is not published. version 4.13.0 was--possibly--published incorrectly. #4423

Closed kaby76 closed 1 year ago

kaby76 commented 1 year ago

I can't tell if 4.13.1 has been published or not. In fact, I can't tell if even 4.13.0 was published correctly because if I try to "go get" the runtime--with or without a version listed on the command--the code fetched is wrong, i.e., contains a wrong version string in recognizer.go, and has many changes in the code that I would have expected when browsing this repo, github.com/antlr/antlr4/runtime/Go/.... I also notice these problems in the repo github.com/antlr4-go/antlr, looking carefully at the tip of the main branch, and also for the git tagged version 4.13.0.

To test my hypothesis, before each "go get", I delete the ~/Go/pkg/mod cached files, and see the files created by the command. So, I know what it is fetching from github.com.

go 1.18

09/20-17:36:47 ~/issues/test-4406/pcre/Generated-Go $ go get github.com/antlr4-go/antlr/v4@v4.13.0 go: added github.com/antlr4-go/antlr/v4 v4.13.0 go: added golang.org/x/exp v0.0.0-20230515195305-f3d0a9c9a5cc 09/20-17:37:04 ~/issues/test-4406/pcre/Generated-Go $ pushd /c/Users/Kenne/go/pkg/mod/github.com/antlr4-go/antlr/v4@v4.13.0 ~/issues/test-4406/pcre/Generated-Go /c/Users/Kenne/Documents/GitHub/Domemtech.Trash 09/20-17:38:47 /c/Users/Kenne/go/pkg/mod/github.com/antlr4-go/antlr/v4@v4.13.0 $ ls antlrdoc.go dfa.go lexer.go recognizer.go atn.go dfa_serializer.go lexer_action.go rule_context.go atn_config.go dfa_state.go lexer_action_executor.go semantic_context.go atn_config_set.go diagnostic_error_listener.go lexer_atn_simulator.go statistics.go atn_deserialization_options.go error_listener.go LICENSE stats_data.go atn_deserializer.go error_strategy.go ll1_analyzer.go token.go atn_simulator.go errors.go nostatistics.go token_source.go atn_state.go file_stream.go parser.go token_stream.go atn_type.go go.mod parser_atn_simulator.go tokenstream_rewriter.go char_stream.go go.sum parser_rule_context.go trace_listener.go common_token_factory.go input_stream.go prediction_context.go transition.go common_token_stream.go int_stream.go prediction_context_cache.go tree.go comparators.go interval_set.go prediction_mode.go trees.go configuration.go jcollect.go README.md utils.go 09/20-17:38:49 /c/Users/Kenne/go/pkg/mod/github.com/antlr4-go/antlr/v4@v4.13.0

This command fetches code that should be "v4.13.0" but returns in the ~/Go cache code that for all I can tell is NOT v4.13.0. If I grep for "4.13" and the older version "4.12", I get some quite bad results.

09/20-17:50:03 /c/Users/Kenne/go/pkg/mod/github.com/antlr4-go/antlr/v4@v4.13.0 $ grep 4.12 antlrdoc.go:list the release tag such as @4.12.0 - this was confusing, to say the least. antlrdoc.go:As of 4.12.1, the runtime is now available as a go module in its own repo, and can be imported as github.com/antlr4-go/antlr antlrdoc.go: │ ├── antlr-4.12.1-complete.jar antlrdoc.go: alias antlr4='java -Xmx500M -cp "./antlr4-4.12.1-complete.jar:$CLASSPATH" org.antlr.v4.Tool' recognizer.go: runtimeVersion := "4.12.0" 09/20-17:56:29 /c/Users/Kenne/go/pkg/mod/github.com/antlr4-go/antlr/v4@v4.13.0 $ grep 4.13 README.md:For instance, the current Go runtime release, which is tagged with v4.13.0 in antlr/antlr4 is retrieved by go get as: README.md: github.com/antlr/antlr4/runtime/Go/antlr/v4 v4.13.0 09/20-17:56:37 /c/Users/Kenne/go/pkg/mod/github.com/antlr4-go/antlr/v4@v4.13.0

Note the `runtimeVersion` is 4.12.0. Note the other occurrences of 4.12.

Now, let's compare the cached source from the `go get` command, and see how that compares with source in the two different repositories https://github.com/antlr/antlr4 and https://github.com/antlr4-go/antlr.

09/20-18:07:09 /c/Users/Kenne/go/pkg/mod/github.com/antlr4-go/antlr/v4@v4.13.0 $ pushd /c/Users/Kenne/Documents/GitHub/antlr4-dev /c/Users/Kenne/go/pkg/mod/github.com/antlr4-go/antlr/v4@v4.13.0 09/20-18:07:18 /c/Users/Kenne/Documents/GitHub/antlr4-dev $ git status . HEAD detached at 4.13.0 nothing to commit, working tree clean 09/20-18:07:22 /c/Users/Kenne/Documents/GitHub/antlr4-dev $ pushd /c/Users/Kenne/go/pkg/mod/github.com/antlr4-go/antlr/v4@v4.13.0 /c/Users/Kenne/Documents/GitHub/antlr4-dev 09/20-18:07:25 /c/Users/Kenne/go/pkg/mod/github.com/antlr4-go/antlr/v4@v4.13.0 $ diff -r . /c/Users/Kenne/Documents/GitHub/antlr4-dev/runtime/Go/antlr/v4/ Only in .: .gitignore diff -r ./antlrdoc.go /c/Users/Kenne/Documents/GitHub/antlr4-dev/runtime/Go/antlr/v4/antlrdoc.go 20c20 < list the release tag such as @4.12.0 - this was confusing, to say the least.

list the release tag such as @4.13.0 - this was confusing, to say the least. 22c22 < As of 4.12.1, the runtime is now available as a go module in its own repo, and can be imported as github.com/antlr4-go/antlr

As of 4.13.0, the runtime is now available as a go module in its own repo, and can be imported as github.com/antlr4-go/antlr 52c52 < │ ├── antlr-4.12.1-complete.jar

  │     ├── antlr-4.13.0-complete.jar

74c74 < alias antlr4='java -Xmx500M -cp "./antlr4-4.12.1-complete.jar:$CLASSPATH" org.antlr.v4.Tool'

  alias antlr4='java -Xmx500M -cp "./antlr4-4.13.0-complete.jar:$CLASSPATH" org.antlr.v4.Tool'

diff -r ./comparators.go /c/Users/Kenne/Documents/GitHub/antlr4-dev/runtime/Go/antlr/v4/comparators.go 21c21 < // allows us to use it in any collection instance that does not require a special hash or equals implementation.

// allows us to use it in any collection instance that does nto require a special hash or equals implementation. diff -r ./lexer.go /c/Users/Kenne/Documents/GitHub/antlr4-dev/runtime/Go/antlr/v4/lexer.go 208a209 ttype := LexerSkip 210c211 < ttype := b.safeMatch()

                  ttype = b.safeMatch()

Only in .: README.md diff -r ./recognizer.go /c/Users/Kenne/Documents/GitHub/antlr4-dev/runtime/Go/antlr/v4/recognizer.go 59c59 < runtimeVersion := "4.12.0"

  runtimeVersion := "4.13.0"

09/20-18:07:54 /c/Users/Kenne/go/pkg/mod/github.com/antlr4-go/antlr/v4@v4.13.0 $


In other words, we see that the code fetched and cached from `go get ..... v4.13.0` isn't version 4.13.0 code.

In fact, the code in repo https://github.com/antlr4-go/antlr/tree/v4.13.0 is version 4.12.0 code, not 4.13.0, which is what I would have expected.

$ diff -r . /c/Users/Kenne/Documents/GitHub/antlr4-dev/runtime/Go/antlr/v4/
Only in .: .gitignore
diff -r ./antlrdoc.go /c/Users/Kenne/Documents/GitHub/antlr4-dev/runtime/Go/antlr/v4/antlrdoc.go
20c20
< list the release tag such as @4.12.0 - this was confusing, to say the least.
---
> list the release tag such as @4.13.0 - this was confusing, to say the least.
22c22
< As of 4.12.1, the runtime is now available as a go module in its own repo, and can be imported as `github.com/antlr4-go/antlr`
---
> As of 4.13.0, the runtime is now available as a go module in its own repo, and can be imported as `github.com/antlr4-go/antlr`
52c52
<       │     ├── antlr-4.12.1-complete.jar
---
>       │     ├── antlr-4.13.0-complete.jar
74c74
<       alias antlr4='java -Xmx500M -cp "./antlr4-4.12.1-complete.jar:$CLASSPATH" org.antlr.v4.Tool'
---
>       alias antlr4='java -Xmx500M -cp "./antlr4-4.13.0-complete.jar:$CLASSPATH" org.antlr.v4.Tool'
diff -r ./comparators.go /c/Users/Kenne/Documents/GitHub/antlr4-dev/runtime/Go/antlr/v4/comparators.go
21c21
< // allows us to use it in any collection instance that does not require a special hash or equals implementation.
---
> // allows us to use it in any collection instance that does nto require a special hash or equals implementation.
diff -r ./lexer.go /c/Users/Kenne/Documents/GitHub/antlr4-dev/runtime/Go/antlr/v4/lexer.go
208a209
>                       ttype := LexerSkip
210c211
<                       ttype := b.safeMatch()
---
>                       ttype = b.safeMatch()
Only in .: README.md
diff -r ./recognizer.go /c/Users/Kenne/Documents/GitHub/antlr4-dev/runtime/Go/antlr/v4/recognizer.go
59c59
<       runtimeVersion := "4.12.0"
---
>       runtimeVersion := "4.13.0"
09/20-17:41:22 /c/Users/Kenne/go/pkg/mod/github.com/antlr4-go/antlr/v4@v4.13.0

A plain ol' go get github.com/antlr4-go/antlr should fetch the latest code, but it produces the same result. The code fetched is NOT version 4.13.1. And, in fact, does not contain the changes introduced in 4.13.0 nor 4.13.1.

Conclusion: likely 4.13.0 was not published correctly. And, version 4.13.1 was never published.

jimidle commented 1 year ago

4.13.1 has not yet been published because there is an outstanding issue for Google that I need to fix. I have not had time to do that as I am working on another contract. I hope to do this at the weekend, at which time I will decide with Ter/Google how to publish 4.13.1 or 4.13.2 or whatever makes the most sense. It is quite involved to do this as I am sure you realize.

4.13.1 was created specifically to include some changes to make life easier for Google - nobody else is waiting upon it. For Go, you need to use the jar for 4.13.10 until 4.13.1 is published. The changes in dev incorporate the changes made for Google, but will not appear in the release only repo until they are verified. There is a race condition that nobody else is likely to see, but that a Google stress test uncovered - it won't be a 30 second fix.

Procedure:

cd tmp mkdir test go mod init go get github.com/antlr4-go/antlr/v4 go: added github.com/antlr4-go/antlr/v4 v4.13.0

cat go.mod module test

go 1.21.1

require ( github.com/antlr4-go/antlr/v4 v4.13.0 ...

Conclusion: you do not understand go versioning and you are not a go programmer.

Garbage in -> Garbage out.

I request that you tone down the acerbic nature of your comments (perhaps you don't read them as such), both to me (I have a thick skin and am older, so I don't really care) and others (who may not be so resilient). Please remember that we are all doing this for free and some of us have to earn a crust at the same time. You don't need to make bold assertions, you can just ask me if all is OK with the published release, and then I can guide you to the correct procedure.

That said, I will check the documentation and make sure that it is not confusing - honestly, it won't be to go programmers - and that the /v4 part has been included therein, and emphasised enough. I will also adjust the README in the release repo to show the correct go get command.

Jim

On Thu, Sep 21, 2023 at 6:29 AM Ken Domino @.***> wrote:

I can't tell if 4.13.1 has been published or not. In fact, I can't tell if even 4.13.0 was published correctly because if I try to "go get" the runtime--with or without a version listed on the command--the code fetched is wrong. Before each "go get", I delete the ~/Go/pkg/mod cached files, and see the files created by the command. So, I know what it is fetching from github.com.

  • Try go get @.*** What does it do?

09/20-17:36:37 ~/issues/test-4406/pcre/Generated-Go $ ls build.ps1 clean.ps1 desc.xml go.mod parser/ run.ps1 Test.go test.sh build.sh clean.sh examples/ makefile perf.sh run.sh test.ps1 09/20-17:36:43 ~/issues/test-4406/pcre/Generated-Go $ cat go.mod module example.com/myparser

go 1.18

09/20-17:36:47 ~/issues/test-4406/pcre/Generated-Go $ go get @. go: added github.com/antlr4-go/antlr/v4 v4.13.0 go: added golang.org/x/exp v0.0.0-20230515195305-f3d0a9c9a5cc 09/20-17:37:04 ~/issues/test-4406/pcre/Generated-Go $ pushd @. ~/issues/test-4406/pcre/Generated-Go /c/Users/Kenne/Documents/GitHub/Domemtech.Trash 09/20-17:38:47 @. $ ls antlrdoc.go dfa.go lexer.go recognizer.go atn.go dfa_serializer.go lexer_action.go rule_context.go atn_config.go dfa_state.go lexer_action_executor.go semantic_context.go atn_config_set.go diagnostic_error_listener.go lexer_atn_simulator.go statistics.go atn_deserialization_options.go error_listener.go LICENSE stats_data.go atn_deserializer.go error_strategy.go ll1_analyzer.go token.go atn_simulator.go errors.go nostatistics.go token_source.go atn_state.go file_stream.go parser.go token_stream.go atn_type.go go.mod parser_atn_simulator.go tokenstream_rewriter.go char_stream.go go.sum parser_rule_context.go trace_listener.go common_token_factory.go input_stream.go prediction_context.go transition.go common_token_stream.go int_stream.go prediction_context_cache.go tree.go comparators.go interval_set.go prediction_mode.go trees.go configuration.go jcollect.go README.md utils.go 09/20-17:38:49 @.

This command fetches code that should be "v4.13.0" but returns in the ~/Go cache code that for all I can tell is NOT v4.13.0. If I grep for "4.13" and the older version "4.12", I get some quite bad results.

09/20-17:50:03 @. $ grep 4.12 antlrdoc.go:list the release tag such as @4.12.0 - this was confusing, to say the least. antlrdoc.go:As of 4.12.1, the runtime is now available as a go module in its own repo, and can be imported as github.com/antlr4-go/antlr http://github.com/antlr4-go/antlr antlrdoc.go: │ ├── antlr-4.12.1-complete.jar antlrdoc.go: alias antlr4='java -Xmx500M -cp "./antlr4-4.12.1-complete.jar:$CLASSPATH" org.antlr.v4.Tool' recognizer.go: runtimeVersion := "4.12.0" 09/20-17:56:29 **@. $ grep 4.13 README.md:For instance, the current Go runtime release, which is tagged with v4.13.0 in antlr/antlr4 is retrieved by go get as: README.md: github.com/antlr/antlr4/runtime/Go/antlr/v4 v4.13.0 09/20-17:56:37 **@.***

Note the runtimeVersion is 4.12.0. Note the other occurrences of 4.12.

Now, let's compare the cached source from the go get command, and see how that compares with source in the two different repositories https://github.com/antlr/antlr4 and https://github.com/antlr4-go/antlr.

09/20-18:07:09 @. $ pushd /c/Users/Kenne/Documents/GitHub/antlr4-dev @. 09/20-18:07:18 @./20-18:07:18> /c/Users/Kenne/Documents/GitHub/antlr4-dev $ git status . HEAD detached at 4.13.0 nothing to commit, working tree clean 09/20-18:07:22 /c/Users/Kenne/Documents/GitHub/antlr4-dev $ pushd @. /c/Users/Kenne/Documents/GitHub/antlr4-dev 09/20-18:07:25 @.*** $ diff -r . /c/Users/Kenne/Documents/GitHub/antlr4-dev/runtime/Go/antlr/v4/ Only in .: .gitignore diff -r ./antlrdoc.go /c/Users/Kenne/Documents/GitHub/antlr4-dev/runtime/Go/antlr/v4/antlrdoc.go 20c20 < list the release tag such as @4.12.0 - this was confusing, to say the least.

list the release tag such as @4.13.0 - this was confusing, to say the least. 22c22 < As of 4.12.1, the runtime is now available as a go module in its own repo, and can be imported as github.com/antlr4-go/antlr http://github.com/antlr4-go/antlr

As of 4.13.0, the runtime is now available as a go module in its own repo, and can be imported as github.com/antlr4-go/antlr 52c52 http://github.com/antlr4-go/antlr52c52 < │ ├── antlr-4.12.1-complete.jar

  │     ├── antlr-4.13.0-complete.jar

74c74 < alias antlr4='java -Xmx500M -cp "./antlr4-4.12.1-complete.jar:$CLASSPATH" org.antlr.v4.Tool'

  alias antlr4='java -Xmx500M -cp "./antlr4-4.13.0-complete.jar:$CLASSPATH" org.antlr.v4.Tool'

diff -r ./comparators.go /c/Users/Kenne/Documents/GitHub/antlr4-dev/runtime/Go/antlr/v4/comparators.go 21c21 < // allows us to use it in any collection instance that does not require a special hash or equals implementation.

// allows us to use it in any collection instance that does nto require a special hash or equals implementation. diff -r ./lexer.go /c/Users/Kenne/Documents/GitHub/antlr4-dev/runtime/Go/antlr/v4/lexer.go 208a209 ttype := LexerSkip 210c211 < ttype := b.safeMatch()

                  ttype = b.safeMatch()

Only in .: README.md diff -r ./recognizer.go /c/Users/Kenne/Documents/GitHub/antlr4-dev/runtime/Go/antlr/v4/recognizer.go 59c59 < runtimeVersion := "4.12.0"

  runtimeVersion := "4.13.0"

09/20-18:07:54 @.*** $

In other words, we see that the code fetched and cached from go get ..... v4.13.0 isn't version 4.13.0 code.

In fact, the code in repo https://github.com/antlr4-go/antlr/tree/v4.13.0 is version 4.12.0 code, not 4.13.0, which is what I would have expected.

$ diff -r . /c/Users/Kenne/Documents/GitHub/antlr4-dev/runtime/Go/antlr/v4/ Only in .: .gitignore diff -r ./antlrdoc.go /c/Users/Kenne/Documents/GitHub/antlr4-dev/runtime/Go/antlr/v4/antlrdoc.go 20c20 < list the release tag such as @4.12.0 - this was confusing, to say the least.

list the release tag such as @4.13.0 - this was confusing, to say the least. 22c22 < As of 4.12.1, the runtime is now available as a go module in its own repo, and can be imported as github.com/antlr4-go/antlr http://github.com/antlr4-go/antlr

As of 4.13.0, the runtime is now available as a go module in its own repo, and can be imported as github.com/antlr4-go/antlr 52c52 http://github.com/antlr4-go/antlr52c52 < │ ├── antlr-4.12.1-complete.jar

  │     ├── antlr-4.13.0-complete.jar

74c74 < alias antlr4='java -Xmx500M -cp "./antlr4-4.12.1-complete.jar:$CLASSPATH" org.antlr.v4.Tool'

  alias antlr4='java -Xmx500M -cp "./antlr4-4.13.0-complete.jar:$CLASSPATH" org.antlr.v4.Tool'

diff -r ./comparators.go /c/Users/Kenne/Documents/GitHub/antlr4-dev/runtime/Go/antlr/v4/comparators.go 21c21 < // allows us to use it in any collection instance that does not require a special hash or equals implementation.

// allows us to use it in any collection instance that does nto require a special hash or equals implementation. diff -r ./lexer.go /c/Users/Kenne/Documents/GitHub/antlr4-dev/runtime/Go/antlr/v4/lexer.go 208a209 ttype := LexerSkip 210c211 < ttype := b.safeMatch()

                  ttype = b.safeMatch()

Only in .: README.md diff -r ./recognizer.go /c/Users/Kenne/Documents/GitHub/antlr4-dev/runtime/Go/antlr/v4/recognizer.go 59c59 < runtimeVersion := "4.12.0"

  runtimeVersion := "4.13.0"

09/20-17:41:22 @.***

A plain ol' go get github.com/antlr4-go/antlr should fetch the latest code, but it produces the same result. The code fetched is NOT version 4.13.1. And, in fact, does not contain the changes introduced in 4.13.0 nor 4.13.1.

Conclusion: 4.13.0 was not published correctly. And, version 4.13.1 was never published.

— Reply to this email directly, view it on GitHub https://github.com/antlr/antlr4/issues/4423, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ7TMBEGUNLRULH37XUOHLX3NU4DANCNFSM6AAAAAA5ATPWWQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

kaby76 commented 1 year ago

Sorry, I'm not trying to be critical of anyone. You're work is excellent and your comments appreciated.

Antlr 4.13.1 was announced. I assumed that all targets were released because the 4.13.1 release page mentions the changes in the Go target.. There is often a long a lag time between tool release date and target release date, which can be painful when trying to update something in grammars-v4. But, sometimes the release for a target is never published, e.g., PHP 4.13.0.

It was confusing why there should be a difference between the source code between https://github.com/antlr/antlr4 and https://github.com/antlr4-go/antlr with the 4.13.0 tag. E.g., https://github.com/antlr/antlr4/blob/4.13.0/runtime/Go/antlr/v4/lexer.go#L209-L211 and https://github.com/antlr4-go/antlr/blob/v4.13.0/lexer.go#L210 . But, it's expected. You're actively trying to keep those two sources in synch. The difference was rolled back to https://github.com/antlr/antlr4 a couple of weeks ago to the dev branch.

jimidle commented 1 year ago

NO problem Ken. We all have our styles of writing.

I expect that the fix will be trivial, but the finding of it won't be and I have been just too busy.

On Thu, Sep 21, 2023 at 7:01 PM Ken Domino @.***> wrote:

Sorry, I'm not trying to be critical of anyone. You're work is excellent and your comments appreciated.

Antlr 4.13.1 was announced. I assumed that all targets were released because the 4.13.1 release page mentions the changes in the Go target. https://github.com/antlr/antlr4/releases/tag/4.13.1. There is often a long a lag time between tool release date and target release date, which can be painful when trying to update something in grammars-v4. But, sometimes the release for a target is never published, e.g., PHP 4.13.0.

It was confusing why there should be a difference between the source code between https://github.com/antlr/antlr4 and https://github.com/antlr4-go/antlr with the 4.13.0 tag. E.g., https://github.com/antlr/antlr4/blob/4.13.0/runtime/Go/antlr/v4/lexer.go#L209-L211 and https://github.com/antlr4-go/antlr/blob/v4.13.0/lexer.go#L210 . But, it's expected: you're actively managing keeping those two sources in synch. The difference was rolled back to https://github.com/antlr/antlr4 a couple of weeks ago to the dev branch.

— Reply to this email directly, view it on GitHub https://github.com/antlr/antlr4/issues/4423#issuecomment-1729340287, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ7TMB3KSCZHAVN4TAKXODX3QM7BANCNFSM6AAAAAA5ATPWWQ . You are receiving this because you commented.Message ID: @.***>