spec: index expressions should allow a newline before the closing square bracket

[mvdan]

What version of CUE are you using (cue version)?

$ cue version
cue version v0.0.0-20231220085713-fd90dbba81e1

go version devel go1.22-a2a2c5b947 Wed Dec 20 02:18:50 2023 +0000
      -buildmode exe
       -compiler gc
  DefaultGODEBUG httplaxcontentlength=1,httpmuxgo121=1,panicnil=1,tls10server=1,tlsrsakex=1,tlsunsafeekm=1
     CGO_ENABLED 1
          GOARCH amd64
            GOOS linux
         GOAMD64 v3
             vcs git
    vcs.revision fd90dbba81e10273bde54178225fb4e1e70c99f3
        vcs.time 2023-12-20T08:57:13Z
    vcs.modified true

Does this issue reproduce with the latest stable release?

Yes; tested with v0.7.0

What did you do?

exec cue fmt in.cue

-- in.cue --
bar[
    "baz"
]

What did you expect to see?

Success; per the spec at https://cuelang.org/docs/references/spec/#tokens:

White space, formed from spaces (U+0020), horizontal tabs (U+0009), carriage returns (U+000D), and newlines (U+000A), is ignored except as it separates tokens that would otherwise combine into a single token. Also, a newline or end of file may trigger the insertion of a comma. While breaking the input into tokens, the next token is the longest sequence of characters that form a valid token.

What did you see instead?

$ testscript -v repro-cmd.txtar 
> exec cue fmt in.cue
[stderr]
expected ']', found newline:
    ./in.cue:2:10
missing ',' in struct literal:
    ./in.cue:3:1
[exit status 1]

[mvdan]

@rogpeppe correctly points out that this is indeed working as expected per the spec. https://cuelang.org/docs/references/spec/#commas explains that a newline inserts a comma if it immediately follows tokens like a string literal, so the parser effectively tries to interpret:

bar[
    "baz",
]

However, the spec defines index expressions as:

Index          = "[" Expression "]" .

Note the lack of a comma token, unlike lists, meaning that the inserted comma above isn't allowed:

ListLit       = "[" [ ElementList [ "," ] ] "]" .

I think we would have to amend the spec grammar to:

Index          = "[" Expression [ "," ] "]" .

Then it would be correct for the parser to allow a newline between an index expression and its closing square bracket.

[mvdan]

To support my proposal, the Go spec allows a comma in index expressions too:

Index          = "[" Expression [ "," ] "]" .

And it works as expected, allowing the splitting of an index expression with newlines:

$ cat f.go
package main

func main() {
    x := []int{3}
    println(x[0])
    println(x[0,])
    println(x[
        0,
    ])
}
$ gofmt f.go
package main

func main() {
    x := []int{3}
    println(x[0])
    println(x[0])
    println(x[0])
}