search

nim-lang / RFCs

A repository for your Nim proposals.
135 stars 26 forks source link

Roadmap for Nim #437

Closed Araq closed 1 year ago

Araq commented 2 years ago

"The Cylons Were Created by Man. They Rebelled. They Evolved. They Look and Feel Human. Some are programmed to think they are Human. There are many copies. And they have a Plan."

It seems to come up regularly that people don't know where Nim is heading, where our focus of development is and what's the PLAN. I'm sorry for not communicating it well but the plan is solid, doesn't change much and it "only" takes longer than I hoped it would take.

Incremental compilation (IC) / nimsuggest / IDE tooling

This is a single point as for me it's a single thing. If we had a compiler that only recompiles the module that was touched the "nimsuggest" tool can be a simple command line tool that answers a single question like "where is symbol X used?".

The good news is that we have this tool already, it's nim check --navigate, the bad news is that its progress is tied to PR https://github.com/nim-lang/Nim/pull/17814. Which is in the state "on hold" until version 1.6.2 is out.

IC is the most important thing for me personally. It's not scheduled for any particular release as it's not about the language nor about its library. Ideally IC will stabilize and be backported to the 1.6.x line.

Update: Good IC relies on refactorings of the compilation pipeline. We want to finish these refactorings, release version 2.0 and then finish IC in version 2.x.

Version 1.6.2

Version 2.0

Hard goals

For version 2.0 we will make --gc:orc, mimalloc and --threads:on the new defaults.

Furthermore, we consider the following experimental features to be stable enough for version 2.0:

Stretch goals

Language

Cleanup of experimental language features

Stable experimental language features

These features are stable and should be part of the manual.

Library

  1. Clean up the system module, move these submodules into their own modules:

  2. Clean up the OS module, make it more type safe, split it in up in e.g.:

  3. The JSON module is also in a poor shape, the API exposes public fields, sometimes is not intuitive (the %* operator) and has poor performance. It could be split up into:

    • [-] jsonbuilder
    • [-] jsonparser
    • [-] jsonlexer
    • [-] jsontree
    • [-] jsonmapper: it maps Nim types (objects) to Json types

  4. More of the standard library should be moved to external Nimble packages. How far we get with this is unclear.

Misc (in comments)

Version 2.x

Every stretch goal of 2.0 that didn't make it into version 2.0, should be considered a goal for version 2.x.

Hard goals

We should remove the requirement for forward declarations and allow for recursive module dependencies. Then we can remove these experimental features:

  1. "Package level objects"
  2. "code reordering"

see also https://github.com/nim-lang/Nim/pull/18818

Stretch goals

  1. Find a better design for "Term rewriting macros". Once done, deprecate the old feature.
  2. Find a better design for "Special Operators". Once done, deprecate the old feature.
  3. Implement "Aliasing restrictions in parameter passing". The compiler can always produce warnings first.
  4. "Strict funcs" / "view types" should be made stable for Nim 2.x.
  5. Concepts: Find a way of how to infer static values and stabilize the feature.
  6. Find a way of allowing a syntax like of Value(let x, var y) and embrace an official pattern matching solution that makes use of this syntax.

Nimble

We hope to be able to ship a new Nimble with the 2.0 release but we are happy to ship it whenever it's ready. See https://github.com/nim-lang/RFCs/issues/398 for good ideas of how to further evolve Nimble. The most important missing feature of Nimble is "lock files" -- Nimble implements it, but not in a way that is compatible with the 1.6.0 Nim compiler.

How can you help?

The standard library cleanups should be rather easy to do.

Vindaar commented 2 years ago

"AST based overloading" -- Should be removed.

Not sure how I feel about this. Would this also imply removal of the same feature for NimNodes? I rather like that (i.e. specifying a CT proc is only for idents).

To be fair I didn't know this could be used with runtime procedures.

metagn commented 2 years ago

Find a way of allowing a syntax like of Value(let x, var y) and embrace an official pattern matching solution that makes use of this syntax.

Reminds me, default values for object fields was quite popular, is this planned as well? (just saw this higher up in the post) are any object variants improvements from many proposals going to be considered? I've also seen a lot of complaints about object construction, but I don't know which useful changes there are other than default values, if there was going to be a =init I'm guessing it'll probably be more coherent for it to be similar to =destroy instead of a complete override of initialization (after zeroing memory/initializing default field values).

The JSON module ... could be split up into:

Would json/builder, json/lexer be better (with a re-export of every submodule in a module named json)? I remember this being discussed for other modules but vaguely something about it not working or being inconvenient. I believe it was proposed for the xml modules and times, but deemed problematic for os.

Araq commented 2 years ago

Would json/builder, json/lexer be better

Yes, most people would like that better but I don't want to implement and support package.module.name just for a refactoring so I'd go with the name prefixes.

konsumlamm commented 2 years ago

"Void type" -- We should optimize away the "empty tuple" type in the backend and remove this feature.

So eventually, void would be replaced by the empty tuple, so that e.g. proc test() = discard would return () (like it is the case in Rust)?

Araq commented 2 years ago

So eventually, void would be replaced by the empty tuple, so that e.g. proc test() = discard would return () (like it is the case in Rust)?

No, for the return type and discard-checking there would still be a void.

levovix0 commented 2 years ago

+1 for moduleGroup/module instead of moduleGroupmodule

I don't want to implement and support package.module.name

why implementing and supporting package.module.name needed for package/module/name?

Araq commented 2 years ago

Update: Expanded the section on Nimble.

juancarlospaco commented 2 years ago

"Parallel & Spawn" -- Should be removed. Better done by external packages.

But stdlib keeps std/tasks ?, I think a way to run multiple stuff at the same time in stdlib is useful !, at least with a basic API, maybe think of removing Parallel and keep Spawn?

Araq commented 2 years ago

std/tasks is a "building block", so it should stay.

Varriount commented 2 years ago

+1 for moduleGroup/module instead of moduleGroupmodule

I don't want to implement and support package.module.name

why implementing and supporting package.module.name needed for package/module/name?

I believe that this is because import only considers the last path component as the module name. The code from json/lexer import nil allows accessing all exported routines/types in "./json/lexer.nim" through the lexer symbol, not the expression json.lexer.

If an expression like json.lexer won't be supported, then the symbol "jsonLexer" may be preferable to "lexer", since "jsonLexer" is less ambiguous.

Araq commented 2 years ago

Update:

juancarlospaco commented 2 years ago

Would be nice to give the jsgen some love too!, at least improve jsgen a little, the JS stdlib improved a lot and wont produce Showstopper complex bugs, and the few JS bugs are actually jsgen related bugs actually so ... 🤔

a-mr commented 2 years ago

What is policy for breaking changes in 2.0?

I see 2 extreme options:

  1. almost no breaking allowed as it is currently in Nim 1.x. The transition to gc:orc is an exception.
  2. any breaking changes are potentially allowed if it is the right thing. Why would you call it version 2.0 otherwise...

Some concrete examples I know:

Araq commented 2 years ago

We should fix as much as reasonable, but we need to have a good upgrade path. The worst thing continues to be subtle changes in runtime behavior. That means:

zedeus commented 2 years ago

Very cool, nice roadmap. What's the current status, what is the primary project being worked on?

The good news is that we have this tool already, it's nim check --navigate, the bad news is that its progress is tied to PR nim-lang/Nim#17814. Which is in the state "on hold" until version 1.6.2 is out.

1.6.2 is now out, but there hasn't been any updates to that PR. What's the status of it?

Araq commented 2 years ago

1.6.2 is now out, but there hasn't been any updates to that PR. What's the status of it?

I never know how to update these RFCs. 1.6.2 is out, next week work on 2.0 will begin.

Araq commented 2 years ago

There is nim-lang/threading which is official plus a couple of external packages that do it better than we could hope to. There are currently no plans to bring this into the stdlib for these reasons:

We merely offer the building blocks with destructors, atomics and std/tasks.

metagn commented 2 years ago

Note for the checklist: guards & locks are in both experimental features to clean up and stable experimental features. Since guards were moved to the stable documentation, it can be marked under the stable experimental features section.

Some RFCs covered by the items:

A few simple non-breaking features that could be done at any time, but would be nice to have soon, probably also under 1.6 line:

There are also tons of RFCs related to default object fields and pattern matching but they tend to be larger in scope and discuss other things.

I think "optimize away empty tuple" will take a while compared to the steps for removing the other experimental features. But since it's not tied to a super well known experimental feature it should be fine if it's not 2.0.

Does the var in of Value(let x, var y) mean var y: T or y: var T? I don't think it should be the first one given for loops and proc parameters presumably don't have it for a reason. Second one is something the language could do that macros couldn't.

"Not nil annotation" -- Should be planned for Nim 2.x but removed from the "experimental" manual.

Don't understand this. Is it supposed to be kept in as experimental but undocumented?

beef331 commented 2 years ago

I just got reminded, is there any chance marshal will get removed from the stdlib for 2.0, or at least removing the VM support for it. Almost every json library made can be used at compile time without hard coded VM support, and as most know marshal is just a json serializer. https://github.com/nim-lang/Nim/blob/devel/compiler/vmgen.nim#L2057-L2065 can cause 10% slower compilation in compile time heavy code(macros included). Since every single call is done with two string searches to see if it's from marshal.

ringabout commented 2 years ago

Alternative way is to support vmmarshal properly with callback if it is worth.

Araq commented 2 years ago

@beef331 wow... that should be fixed indeed. It doesn't necessarily mean we need to disable marshal on the VM though, it should simply either use its own opcodes or the existing callback mechanism.

ringabout commented 2 years ago

A simple POC using callback mechanism: https://gist.github.com/xflywind/551d84567cb39d95d0ddcd71f8279e6a

ringabout commented 2 years ago

Path related RFCs：

https://github.com/nim-lang/RFCs/issues/54 new path handling module that uses Nim's type system, using Path instead of string https://github.com/nim-lang/RFCs/issues/121 os module should have an "exists" function https://github.com/nim-lang/RFCs/issues/71 [compiler/pathutils] AbsoluteFile,AbsoluteDir,RelativeFile,RelativeDir is too complicated https://github.com/nim-lang/RFCs/issues/112 joinPath("foo", "/bar") should return "/bar"

Menduist commented 2 years ago

Could we consider https://github.com/nim-lang/RFCs/issues/89 for nim V2?

A few options have been considered, eg the linked RFC, or deprecating naked excepts, all of them being better than the current status quo, imo

Araq commented 2 years ago

Version 2 or not, the guiding principle remains "no secret change in semantics" so we are deprecating except:. We could make it not compile altogether for version 2 though.

juancarlospaco commented 2 years ago

In Python, naked except is allowed, but most linters, IDEs, and tools will complain about it.

qaziquza commented 2 years ago

Any plans for nre/re? Removal of one or the other? Cleanup, speedup? (see https://forum.nim-lang.org/t/6008)

Araq commented 2 years ago

They should both be moved out of the stdlib, IMO.

metagn commented 2 years ago

Some tidbits from the forum thread mentioned up to this point for 2.0 breaking features ( https://forum.nim-lang.org/t/9132 ):

Also a lot of discussion about stdlib organization and tooling/error messages, but these already seem to be a focus. Other non-breaking but commonly discussed stuff include sugar as default import with more macros, and more concept use in stadnard library.

ringabout commented 2 years ago

Suggestions for Nim v2 (thanks to @j-james)

ref https://forum.nim-lang.org/t/9132

https://gist.github.com/j-james/08cd36b7475d461d6291416381ce98ad

arnetheduck commented 2 years ago

Whatever else happens, adding more things, specially more things that don't exist yet (such as random nice-to-haves from v2 forum threads) to 2.0 is a good way to never ship a good 2.0 - shipping orc/arc is already a massive undertaking: it fundamentally changes the way libraries / Nim code should be written (because they now need to think different about memory management, destructor ordering, performance profile, a fresh set of compiler bug workarounds etc etc).

Even enabling ORC/ARC for a simple, small codebase like the compiler itself has proven to be a major difficulty when trying to retain it's previous performance characteristics (or even keep it working without hitting bugs) - to get it across the finish line, the learnings from that ordeal need to be carried back to the standard library itself, which, as such, needs significant work if it is to not be terrible under the new memory regime, let alone to take advantage of the significant improvements and play well with them.

If anything, a line in the sand around further addtions and 100% focus on system.nim to make that the best system possible in a post-2.0-language world would be key targets to look at, such that there emerges a small and stable core / foundation upon which further work can be done.

juancarlospaco commented 2 years ago

I would like to re-consider ideas proposed before this Nim 2.0 trend, like:

beef331 commented 2 years ago

Slightly related to arne's point. In my quest to make code more reusable I did attempt to make parseutils use openarray[char] but it's showed a few issues with openarray/string/cstring that will need to get resolved. Presently due to implicit conversion from string to cstring the following does not work.

proc doThing(oa: openarray[char]) = discard
proc doThing(str: cstring) = discard
doThing("hello")

Similarly the following perhaps should work. Where openArray[char] is considered equivalent to string in the case of converters.

type NotString = distinct string
converter toString(s: NotString): string = string(s)
proc doThing(oa: openarray[char]) = discard
doThing(NotString("hello"))
juancarlospaco commented 2 years ago

The Nim 2.0 can also be a good opportunity to make a decision about Nimble packages clean out, packages that do not exist anymore and give 404 errors, but they are still in the packages list, theres multiple issues open and closed in the repo about the problem, and removing the packages from the JSON looks like it is not allowed (?), but it is useless because the repo do not exist anymore, and sometimes the user also do not exist anymore or is not coding anymore since years ago.

CyberTailor commented 1 year ago

"Parallel & Spawn" -- Should be removed. Better done by external packages

What about bootstrapping build systems/package managers that use threads? Will they need to supply their own implementation of threadpool?

Araq commented 1 year ago

No, package managers and bulid systems can use Nimble packages as dependencies if they do the "git clone" commands directly in the build script and/or can use git submodules. A package manager is not like a compiler, it doesn't have to "bootstrap" itself.

CyberTailor commented 1 year ago

This is not possible in network-sandboxed environments, and support for them matters.

Araq commented 1 year ago

The sandbox would prevent nimble install nimble just as it would prevent git clone depsThatNimbleRequires.

CyberTailor commented 1 year ago

I've successfully packaged lots of Nim libraries and programs for Gentoo, which package manager is network-sandboxed (this is the standard in *nix packaging). Neither git nor nimble are needed for it.

Seems like this discussion is off-topic and should be over.

Araq commented 1 year ago

Fine with me, you started the subdiscussion.

tomasky commented 1 year ago

I like nim, but cross-compiling is inconvenient. I wish I could cross-compile directly like Golang or ziglang, which would attract more users

Araq commented 1 year ago

Just use a CI, cross-compilation won't let you run any tests anyway... I wish we would attract more users who understand software development.

tomasky commented 1 year ago

Ok, I spent some time cross-compiling using CI:

- name: build-x86
  image: dockcross/linux-x86
  commands:
  - nimble build --passL:-static -o:out-linux-x86 -d:release

- name: build-x86-64
  image: dockcross/linux-x64
  commands:
  - nimble build --passL:-static -o:out-linux-amd64 -d:release

- name: build-linux-arm64
  image: dockcross/linux-arm64
  commands:
  - nimble build --cc:env --os:linux --cpu:arm64 -o:out-linux-arm64 -d:release

- name: build-android-arm64
  image: ghcr.io/tomasky/drones/arm64:nim
  commands:
  - nimble build --cc:env --os:android --cpu:arm64 -o:out-android-arm64 -d:release
Araq commented 1 year ago

Postponed:

Araq commented 1 year ago

Postponed:

Araq commented 1 year ago

I'm closing this since version 2.0 is effectively done, we'll only fix even more ORC bugs and release 2.0 in early 2023. The plan for 2023 will be published as an RFC here in the next couple of days.

ShalokShalom commented 1 year ago

I like nim, but cross-compiling is inconvenient. I wish I could cross-compile directly like Golang or ziglang, which would attract more users

https://github.com/enthus1ast/zigcc

Just use the Zig compiler 😉