Nim Roadmap 2024 and beyond

[Araq]

Life has a melody, Gaius. A rhythm of notes that become your existence once 
they're played in harmony with God's plan. Come. See the face of the shape of things to come.
-- "6" from Battlestar Galactica

NOTE: This plan covers 2024 and beyond as it is very ambitious.

Version 2.2

• [ ] Fix or mitigate the remaining bugs labelled as "showstopper" and release versions 2.2.0 and 2.0.2 early in 2024.

C/C++ Backends based on NIR

NIR is Nim's upcoming intermediate language that is sufficiently low level as a compilation target while maintaining all the information required for optimizations, including type based alias analysis. NIR enables more effective evolution of the Nim language as different Nim frontend versions can all target the same NIR layer:

Nim v2 ---> NIR --> Optimizer including  --> NIR +---> C / C++ code
            ^       CPS transformations          |---> LLVM
            |                                    |---> Interpreter / REPL
            |                                    |---> GCC (libjit)
            |
Nim v3 -----+

This will be achieved by a set of huge compiler code refactorings, among them:

• [ ] Port the iterator inliner to work on NIR.
• [ ] Port the VM to work on NIR.
• [ ] Refactor the compiler's type graph to be less error prone to use and to consume less memory and to be faster to serialize for IC support.
• [ ] Refactor the compiler's abstract syntax tree to consume less memory and be faster to serialize for IC support.

Upcoming Versions

Language

Upcoming versions will finally address some long standing pain points of Nim:

• [ ] Incremental compilation (IC). It should not even be possible to run the compiler in a non-incremental mode.
• [ ] Cyclic modules via an explicit .cyclic annotation on all modules that participate in the cycle group. A cycle group is always compiled as a single step producing a single .rod file for the IC mechanism. Since every module in the cycle has to be annotated the import order cannot have unkown effects for IC.
• [ ] Calling routines without forward declarations.
• [ ] Type checked generics. A key insight here is that an unconstrained generic type variable T should simply match "any" used operation in the generic routine's body. But as soon as T is constrained by a concept, it's checked thoroughly.
• [ ] Deprecate case objects and offer real sum types as a replacement.
• [ ] Make not nil the default for ref and ptr.
• [ ] Experiment how a more modern macro system could be provided with the help of a compiler plugin system based on NIR.

Library

• [ ] Improve the string implementation to be more flexible with custom memory management. More details in an RFC to come.

• [ ] Deemphasize the importance of string by offering a write operation that all collection types support. This makes the libraries more aligned with embedded devices and it makes it easier to write high-performance Nim code.

  • [ ] Every collection should offer a write (for text streams/files) and a store (for binary streams/files) operation.
  • [ ] Every collection should offer a read (for text streams/files) and a load (for binary streams/files) operation.

• [ ] Change the seq type so that it disables the assignment operation that copies. Instead one has to use an explicit copy operation. In the vast majority of cases, the compiler can move the seq however, so that should rarely make a difference. The same principle is applied to any collection type like Table or HashSet that is based on seq.

• [ ] Rework the collection libraries once again to make use of type checked generics and concepts.

[mratsim]

I see CPS, is there some further plan for it?

No vtable plan? The concepts/interfaces/heterogenous collection of same behavior has been in limbo since 2017.

[Araq]

There is already experimental: "vtables" which enables VTables for method. There are no current plans to turn concept into anything else but type constraints. In order to convert a concept into an interface use a macro.

So far nothing convinced me that there is a better solution. Converting int implicitly to a runtime Number interface is only convenient for newcomers which are then disappointed by the poor performance and we end up with another closure-like feature: "Yes, it's there, FP people rejoice! But don't use it, it's slow!"

[omentic]

Very exciting! Some questions and notes:

Calling routines without forward declarations.

Does this also apply for types? That is more of a pain point in my experience.

Deprecate case objects and offer real sum types as a replacement.

Love to see it. What form will this take? metagn's proposal in https://github.com/nim-lang/RFCs/issues/527? Are there plans to make use of them when refactoring the compiler AST? (better question: how high of a priority are these?)

Rework the collection libraries once again to make use of type checked generics and concepts.

Also exciting. With regards to lists, however: any thoughts about laziness? Chaining iterators is currently extremely inefficient, to my knowledge. Are there any plans to rework std/sequtils and similar to either follow a Rust-like model (lazy iterators in the types system, explicitly collected) or take some compiler magic inspiration from zero-functional?

Also, any plans regarding tooling?

[omentic]

Curious about the distinction between interfaces and concepts v2 too, because I've never quite seen what makes them distinct. (or really, have a concrete idea of what "interface" entails here: implementations differ so much across languages.) I like the term "interface" better to describe the behavior of concepts v2, but bah...

[jangko]

We have remove case objects from our serializer library because it is hard to handle automatically by the library and needs user intervention. I hope the new sum types can be more friendly for both serializer library implementer and users.

[Araq]

@jangko A great point, we should be careful here.

[mratsim]

So far nothing convinced me that there is a better solution. Converting int implicitly to a runtime Number interface is only convenient for newcomers which are then disappointed by the poor performance and we end up with another closure-like feature: "Yes, it's there, FP people rejoice! But don't use it, it's slow!"

I never mentioned implicit conversion though. The collection would had to be a ref say seq[ref Stringifiable]

[Araq]

Love to see it. What form will this take? metagn's proposal in https://github.com/nim-lang/RFCs/issues/527? Are there plans to make use of them when refactoring the compiler AST? (better question: how high of a priority are these?)

527 is definitely close to become a winner. The refactorings do not depend on these better sum types and are prioritized over feature developments.

[Araq]

The collection would had to be a ref say seq[ref Stringifiable]

Where is the benefit over Stringifyable = ref object; method toString(x: Stringifyable): string; ... which you can generate from a concept body via a macro.

[radkesvat]

any plans for improving nim views and checkings...?

[Araq]

any plans for improving nim views and checkings...?

We think we can build a good borrow checker based on NIR, yes. But it's not of a particularly high priority.

[miguelmartin75]

Very cool roadmap! Excited for the REPL.

Interpreter / REPL

It is not explicit, but will the "REPL" support include LLVM or libgccjit backends? It seems possible (and done with nlvm with ORC), but it is not explicitly stated.

Change the seq type so that it disables the assignment operation that copies. Instead one has to use an explicit copy operation

This is just a thought/muse on the language design: it would be a nice feature to have "explicit" blocks: where certain operations such as copies/moves/allocations/etc. must be explicit or is disallowed (this could be extended to you must provide types for vars or other implicit features of the language). For performance-critical code, it would be a good feature to include such that you know exactly when a copy is occurring / memory alloc. This is related to this discussion on allocations within the effect system: perhaps the effect system can be used or an extension on it, but from my understanding, the effect system does not work per variable (var on noSideEffect/func is the exception).