parser: make dot operators (eg:`.?`) same precedence/associativity as `.`

[timotheecour]

originally proposed in https://github.com/nim-lang/Nim/pull/16924#issuecomment-776542934

proposal 1

we defined dot like operators as operators starting with ., but excluding operators starting with .. like ..<, ..^ etc.

Dot like operators shall have same precedence as .

example

when true:
  import macros
  macro deb(a: untyped): untyped = echo a.lispRepr
  deb a.b.c # parsed as: (a.b) . c
  deb a.b.?c # parsed as: (a.b) .? c
  deb a.?b.c # parsed as: a .? (b.c) currently; parsed as `(a .? b).c` under this RFC

current output:

(DotExpr (DotExpr (Ident "a") (Ident "b")) (Ident "c"))
(Infix (Ident ".?") (DotExpr (Ident "a") (Ident "b")) (Ident "c"))
(Infix (Ident ".?") (Ident "a") (DotExpr (Ident "b") (Ident "c")))

under this RFC, a.?b.c would instead parse as: (a.?b).c, and .? would then be a valid replacement for user-defined ., in particular for jsffi or nimpy, without having to mess around with builtin .:

let j = (a: 1, b: 2).toJson
let foo = j.?bar.?baz

note

this reduces the need for dotOperators (see also https://github.com/nim-lang/Nim/pull/16996 which removed dotOperators in std/wrapnils) but this RFC isn't about deprecating dotOperators (this can be discussed elsewhere); in particular, there may be valid remaining use cases for type punning.

After this RFC, jsffi/nimpy + any other library/API that was using dotOperators as a means to provide dynamic field access would then be encouraged to use .? (or similar) instead of ..

This would sidestep issues like https://github.com/nim-lang/Nim/issues/7777, https://github.com/nim-lang/Nim/issues/15607, https://github.com/nim-lang/Nim/issues/13063 for those use cases.

proposal 2 (on top of proposal 1)

all of https://nim-lang.github.io/Nim/manual_experimental.html#special-operators would apply to dot-like operators, except that the flag {.experimental: "dotOperators".} would not be needed for those (except for . itself).

in particular, these can be defined:

template `.?`(a, b): untyped
template `.?()`(a, b): untyped
template `.?=`(a, b, c): untyped

(ditto with replacing .? by another dot like operator, eg .!, .$$ etc)

breaking change discussion

• proposal 1 is a breaking change but it's unclear whether this is a problem in practice, as this is probably how user would expect it to work, and IIRC dot like operators are rare/nonexistant; data welcome. A PR implementing this could assess damage on important_packages; a migration strategy could be possible, by controlling this via a scopable pragma, possibly returning warnings in places where the parser change would change semantics and prompt for adding parens to enforce old associativity
• this parser change should be backported to 1.0 branch so that libraries involving dot like operators can work there too

links

• https://nim-lang.github.io/Nim/manual_experimental.html#special-operators-dot-operators
• misc dotOperators · Issue #582 · timotheecour/Nim
• Force experimental switch to be able to use call/dot operators by hlaaftana · Pull Request #16924 · nim-lang/Nim
• see also D's approach, using opDispatch https://tour.dlang.org/tour/en/gems/opdispatch-opapply

[Araq]

this parser change should be backported to 1.0 branch so that libraries involving dot like operators can work there too

Maybe but we know from the survey results that most people are on version 1.4 and happy to upgrade. Maybe it's time to make the next Nim release version 2.0, keeping semver. Or dropping semver altogether as it's just terrible, see also https://news.ycombinator.com/item?id=19137896

[mratsim]

There should be a stability focused 1.x version where everything that was or will be introduced in 1.x works decently (views, arc/orc, IC,...). People need to be able to rollback without losing too much.

Now regarding, Nim 2.0, I don't see the value of the "2", unless there is a new strategy to decouple the standard library from the compiler, because there is nothing 2.0 in the standard library: views are not used, threadpools need revamp, concepts/interfaces should be used as well, strutils and sequtils need to embrace dup and maybe have an alternative view/stream/iterator API as well.

[Araq]

There should be a stability focused 1.x version where everything that was or will be introduced in 1.x works decently (views, arc/orc, IC,...).

Yes and we are working on that. But should the community be hold back until we have this version? Maybe some people prefer to work on 2.0-worthy things.

[metagn]

Relevant line for first proposal:

https://github.com/nim-lang/Nim/blob/554fe8f88fc6d146b17726acbab415f77e346a72/compiler/lexer.nim#L924

Change could be done today, seems pretty uncontroversial. Should be done IMO.

Second proposal is interesting, but = must be the most edge cased expression in the language lol. I mean, .= is a ternary operator, if we extend it to specifically other dot operators, then why stop there? Why use operators when you can use names?

proc `.?=`(x, y, z: int) = echo x * y * z

1 .? 2 = 3

# becomes

proc `foo=`(x, y, z: int) = echo x * y * z

1.foo(2) = 3

I don't know if it's worth making these kinds of changes unless = is generalized for custom implementations, which is another topic whether or not it should be.

[timotheecour]

Change could be done today, seems pretty uncontroversial. Should be done IMO.

yes, PR welcome for proposal 1!

if we extend it to specifically other dot operators, then why stop there?

you're right, and maybe dotOperators is an un-necessay hack:

• complicates compiler, buggy implementation
• yet, not flexible enough, as it (currently) can't handle any of these:

  a.?foo = bar
  1.foo(2) = 3
  a.foo += bar # this will be parsed as `+=`(a.foo, bar) and `.=` can't help here [1]

In fact, dotOperators is a poor-man's attempt at TRW (term rewriting macros, https://nim-lang.github.io/Nim/manual_experimental.html#term-rewriting-macros).

What if we instead do this:

template ex1{a.foo = c}(a: JsonNode, foo: untyped, c: JsonNode) = a[astToStr(foo)] = c
x.bar = y # rewritten as a["bar"] = y

and these would also work:

# this replaces a setter, like `foo=`:
template ex1{a.foo = c}(a: Foo, c: int) =
  echo "setting real field fooImpl"
  a.fooImpl = c

except it's more general than a setter as, unlike setters, it can be made to work for module scope variables as shown in https://github.com/nim-lang/Nim/issues/14674#issuecomment-778570371

furthermore, unlike setters, it can be used for more general assignments, eg += (we don't have foo+= setters):

template ex1{a.foo += c}(a: Foo, c: int) = ...

or for dynamic fields:

template ex2{a.foo += c}(a: Foo, foo: untyped, c: int) = ...

Note that TRW currently can't allow some of the things mentioned here (but https://github.com/nim-lang/Nim/issues/14674#issuecomment-778570371 works), but it's close; maybe it could be a different syntax than the current TRW though.

Although TRW adds another level of complexity and magic, I'd argue the magic is already present in dotOperators except it's worse as the rewrite rule is written in the compiler instead of in library code via the declaration in the TRW eg template ex1{a.foo += c}(a: Foo, c: int) = ...; by making it more general, it can be made in fact less buggy. Similarly, using TRW could make the []= array indexing operators less magic/buggy (see https://github.com/nim-lang/Nim/issues/15911). And since you have symbol names for each rewrite rules, you can import/export them individually, unlike []= + similar.

notes

[1] eg, suppose you have a proxy DSL for some operations that happen in a remote server (eg mongodb/sql), you'll want a.foo += bar to happen entirely on the remote, you can't have it parsed as: a.foo = a.foo + bar (eg you can't get a var reference to a.foo which is an abstract remote operation)

links

• On term rewriting macros - Nim forum
• Nim Experimental Features

[Araq]

TRWs are not well designed either: They slow down compilations significantly (I have ideas how to fix that) and have been designed for custom optimizations. For your purpose it's not a good fit.

[n0bra1n3r]

What do you guys think of this forum post about dotOperators?

Like @hlaaftana said, the case of .?= is essentially a ternary operator, and from my newbie perspective modifying Nim's binary operators to act like ternary ones for special cases seems like a work-around. @timotheecour's TRW macro idea seems like it can model the ternary case well, but like @Araq said it's designed for other things. The idea in the forum post might be an alternative, as it can also cover .?= in a way similar to a TRW macro:

type Obj = object

macro `=call`(o: Obj, field: untyped, args: varargs[untyped]) =
  echo $field

let obj = Obj()
obj.?test = 1 # prints '?test='
# same as
`?test=`(obj, 1)

I don't know if the above is even possible, but I just feel like extending dot operators like in this RFC would more fit languages that have the "methods go inside class definitions" concept, and Nim doesn't do this. I think maybe leveraging Nim's UFCS capabilities would fit Nim more, instead of modelling after Dlang's opDispatch. You're the experts, but just wanted to voice this idea.

[Araq]

This has been implemented.

[planetis-m]

Is it ok to add `.[]` and `.[]=` to the list? This way you can wrap object fields that are arrays. Edit dont really need it.