`nim dump` is roughly 100x slower in 1.3 versus 1.2

[disruptek]

See title.

[Araq]

I'm pretty sure it's caused by our NimScript config evaluation.

[disruptek]

Confirmed. This is also the source of the extra output problem detailed in #13912.

@timotheecour

[timotheecour]

@Araq as always, the 1st thing to try is git bisect

3d2459bdc0b6d6236a2cd9209ed81c965ee411a5 is the first bad commit
commit 3d2459bdc0b6d6236a2cd9209ed81c965ee411a5
Author: Araq <rumpf_a@web.de>
Date:   Thu Apr 30 20:23:42 2020 +0200

    fixes the regression #12860 caused; hotfix

 lib/pure/os.nim          | 2 +-
 lib/system/nimscript.nim | 3 +++
 2 files changed, 4 insertions(+), 1 deletion(-)
bisect run success

and indeed git bisect is correct once again:

https://github.com/nim-lang/Nim/commit/3d2459bdc0b6d6236a2cd9209ed81c965ee411a5 adds a import os dependency on nimscript which introduces the regression:

• nim dump . processes $nim/config/config.nims
• $nim/config/config.nims imports system => nimscript.nim => (since that commit) os.nim
• os.nim itself is expensive, causing a 6X slowdown

But that's not the end of the story.

Because of https://github.com/dom96/choosenim/issues/193 (which I reported here https://github.com/nim-lang/Nim/issues/13986#issuecomment-614611262 as a confounding factor for another issue), $nim/config/config.nims is not copied over to $nim_toolchain_D/nim-1.2.0/config/config.nims, so for nim 1.2.0 installed via choosenim you instead get:

• nim dump . processes ... no nims file at all since processes $nim/config/config.nims doesn't exist

=> 23X difference

timings:

$nim_120_X dump --skipusercfg --hint:conf:on .  0.01s user 0.01s system 88% cpu 0.015 total
$nimb dump --skipusercfg --hint:conf:on .  0.05s user 0.01s system 97% cpu 0.062 total # right before 3d2459bdc0b6d6236a2cd9209ed81c965ee411a5
$nimb dump --skipusercfg --hint:conf:on .  0.33s user 0.02s system 99% cpu 0.352 total # right after 3d2459bdc0b6d6236a2cd9209ed81c965ee411a5

@disruptek

Confirmed. This is also the source of the extra output problem detailed in #13912.

unrelated AFAIU

proposal

• https://github.com/dom96/choosenim/issues/193 is a bug that keeps creeping up and should be fixed, $nim/config/config.nims is meant to be read for correctness even if it currently doesn't contain much; users who know what they're doing can always pass --skipCfg to skip it
• https://github.com/nim-lang/Nim/commit/3d2459bdc0b6d6236a2cd9209ed81c965ee411a5 should be reverted to avoid adding os dependency in nimscript
• https://github.com/nim-lang/Nim/pull/12860 should be reverted, and https://github.com/nim-lang/Nim/pull/14143 should be merged instead, it's the more general fix for https://github.com/nim-lang/Nim/issues/12835 since it also fixes https://github.com/nim-lang/Nim/issues/14142 etc

That will fix all above problems including https://github.com/nim-lang/Nim/issues/14142.

• I also have a PR that turns nimscript into an import module (not include file) and allows a nimscript module to skip auto-importing it, which makes nims code more like regular nim code without all the "baggage" like cd, etc. I just need to see how well it fits with https://github.com/nim-lang/Nim/pull/14235. https://github.com/timotheecour/Nim/issues/307

local imports

https://github.com/timotheecour/Nim/issues/169 That's a topic for another discussion, but allowing local imports (eg at proc scope) would solve many dependency issues, including the one that causes this regression, making importing more lazy. I even have a working branch for that.

[genotrance]

dom96/choosenim#193 is a bug that keeps creeping up and should be fixed, $nim/config/config.nims is meant to be read for correctness even if it currently doesn't contain much; users who know what they're doing can always pass --skipCfg to skip it

This is not a choosenim issue which simply downloads the .zip or .xz file from nim-lang.org and extracts it. The file is missing in the archive posted on nim-lang.org.

[timotheecour]

The file is missing in the archive posted on nim-lang.org.

fixing that part in https://github.com/nim-lang/Nim/pull/14267