Increased memory during compiling on 1.6.x with `orc`

[ThomasTJdev]

Upgrading to Nim 1.6.x using orc increases the memory usage with x10 during compiling.

We are in the process of upgrading from 1.4.x to 1.6.x, but have hit a problem, where the memory usage explodes. On both Nim 1.6.0 and 1.6.2 (and 1.7.1) the compiling process suddenly reaches a point where the memory will increase rapidly and continues through the roof.

• The problem is only present on Nim +1.6.x with orc.
• We can compile fine with refc on all Nim versions
• We can compile fine with orc on Nim 1.4.x.

We are heavily using asyncdispatch and jester throughout the code.

Output

[..xxx]

/home/runner/work/nightlies/nightlies/nim-1.6.2/compiler/lookups.nim(294, 12) compiler msg initiated here [MsgOrigin]
/home/user/tool/tool.nim(1213, 3) template/generic instantiation of `routes` from here
/home/user/.nimble/pkgs/jester-0.5.0/jester.nim(1304, 35) template/generic instantiation of `async` from here
/home/user/.choosenim/toolchains/nim-1.6.2/lib/pure/asyncmacro.nim(41, 30) Warning: not GC-safe: 'cast[proc () {.closure, gcsafe.}](matchNimAsyncContinue_486659554)' [GcUnsafe]
              next.addCallback cast[proc() {.closure, gcsafe.}](identName)
                               ^
/home/runner/work/nightlies/nightlies/nim-1.6.2/compiler/sempass2.nim(209, 10) compiler msg initiated here [MsgOrigin]
/home/user/tool/tool.nim(1213, 3) template/generic instantiation of `routes` from here
/home/user/.nimble/pkgs/jester-0.5.0/jester.nim(1331, 39) template/generic instantiation of `async` from here
/home/user/.choosenim/toolchains/nim-1.6.2/lib/pure/asyncmacro.nim(41, 30) Warning: not GC-safe: 'cast[proc () {.closure, gcsafe.}](matchErrorHandlerNimAsyncContinue_486767074)' [GcUnsafe]
              next.addCallback cast[proc() {.closure, gcsafe.}](identName)
                               ^
/home/runner/work/nightlies/nightlies/nim-1.6.2/compiler/sempass2.nim(209, 10) compiler msg initiated here [MsgOrigin]
/home/user/.nimble/pkgs/jester-0.5.0/jester.nim(1360, 24) Hint: global variable declared here [GlobalVar]
    let jesIdent = genSym(nskVar, "jes")
                         ^
/home/runner/work/nightlies/nightlies/nim-1.6.2/compiler/semstmts.nim(661, 16) compiler msg initiated here [MsgOrigin]
Hint: libpq.so(.5|) [Dependency]
/home/runner/work/nightlies/nightlies/nim-1.6.2/compiler/cgen.nim(704, 17) compiler msg initiated here [MsgOrigin]
/home/user/.choosenim/toolchains/nim-1.6.2/lib/pure/asyncdispatch.nim(1313, 39) Hint: passing 'adata.readList' to a sink parameter introduces an implicit copy; if possible, rearrange your program's control flow to prevent it [Performance]
        adata.readList = newList & adata.readList
                                        ^
/home/runner/work/nightlies/nightlies/nim-1.6.2/compiler/injectdestructors.nim(462, 14) compiler msg initiated here [MsgOrigin]

#
# !! It happens right here !!
#

/home/runner/work/nightlies/nightlies/nim-1.6.2/compiler/injectdestructors.nim(462, 14) compiler msg initiated here [MsgOrigin]
/home/user/.nimble/pkgs/jester-0.5.0/jester/patterns.nim(115, 34) Hint: passing 'matchNamed' to a sink parameter introduces an implicit copy; if possible, rearrange your program's control flow to prevent it [Performance]
        result.params[node.text] = matchNamed

/home/user/.choosenim/toolchains/nim-1.6.2/lib/pure/httpcore.nim(170, 25) Hint: passing 'headers.table[toCaseInsensitive(headers, key)]' to a sink parameter introduces an implicit copy; if possible, rearrange your program's control flow to prevent it [Performance]
      return headers.table[headers.toCaseInsensitive(key)].HttpHeaderValues
                          ^
/home/runner/work/nightlies/nightlies/nim-1.6.2/compiler/injectdestructors.nim(462, 14) compiler msg initiated here [MsgOrigin]
/home/user/.choosenim/toolchains/nim-1.6.2/lib/pure/uri.nim(190, 12) Hint: passing 'name' to a sink parameter introduces an implicit copy; if possible, rearrange your program's control flow to prevent it [Performance]
      yield (name, value)
             ^
/home/runner/work/nightlies/nightlies/nim-1.6.2/compiler/injectdestructors.nim(462, 14) compiler msg initiated here [MsgOrigin]
/home/user/.choosenim/toolchains/nim-1.6.2/lib/pure/uri.nim(190, 18) Hint: passing 'value' to a sink parameter introduces an implicit copy; if possible, rearrange your program's control flow to prevent it [Performance]
      yield (name, value)

[..xxx]

Additional Information

$ nim -v
Nim Compiler Version 1.4.8 [Linux: amd64]
Compiled at 2021-05-25
Copyright (c) 2006-2021 by Andreas Rumpf

git hash: 44e653a9314e1b8503f0fa4a8a34c3380b26fff3

$ nim -v
Nim Compiler Version 1.6.0 [Linux: amd64]
Compiled at 2021-10-19
Copyright (c) 2006-2021 by Andreas Rumpf

git hash: 727c6378d2464090564dbcd9bc8b9ac648467e38

$ nim -v
Nim Compiler Version 1.6.2 [Linux: amd64]
Compiled at 2021-12-17
Copyright (c) 2006-2021 by Andreas Rumpf

git hash: 9084d9bc02bcd983b81a4c76a05f27b9ce2707dd

$ nim -v
Nim Compiler Version 1.7.1 [Linux: amd64]
Compiled at 2021-12-26
Copyright (c) 2006-2021 by Andreas Rumpf

git hash: fdbec969d8601af0f666426b523e8ebba644ab56

$ gcc -v
gcc version 11.1.0 (GCC) 

$ clang -v
clang version 13.0.0

[Araq]

@Clyybber do you think this is related to your refactorings?

[Clyybber]

Yeah, most likely.

[ringabout]

Hello @ThomasTJdev do you have some code example to reproduce the issue? If the code is not public, could you try https://github.com/nim-lang/Nim/pull/19975?

[ThomasTJdev]

Hi @xflywind,

Can confirm that our code compiles fine with recent devel and also quite fast. Thank you!

$ choosenim devel
   Switched to Nim #devel

$ nim -v
Nim Compiler Version 1.7.1 [Linux: amd64]
Compiled at 2022-07-08
Copyright (c) 2006-2022 by Andreas Rumpf

git hash: ad0aee535435ac9b22c2ee9ef02085ffdc901d48
active boot switches: -d:release

$ gcc -v
gcc version 12.1.0 (GCC)