search

c-blake / cligen

Nim library to infer/generate command-line-interfaces / option / argument parsing; Docs at
https://c-blake.github.io/cligen/
ISC License
501 stars 23 forks source link

Question: Have you noticed a slowdown in compilation? #137

Closed pb-cdunn closed 4 years ago

pb-cdunn commented 4 years ago

I think it got slower when I switched to nim-1.2.0. Actually, I think every compilation unit is much slower now, but I noticed it for the top CLI file, which is the slowest of all.

I think Nim is compiling everything imported in a single compilation unit, and since the cligen CLI imports everything, it compiles everything. Does that sound accurate?

I wonder if we can find a way to delegate into the sub-command files without actually importing them.

I'll post some numbers soon ...

pb-cdunn commented 4 years ago

Hmmm. I'm not sure 1.0.0 was any faster.

One problem is that some unrelated change in an imported file can cause the top-level to be recompiled, even though nothing changed in the API.

c-blake commented 4 years ago

There is Nim compilation and optimizing C compilation. For me almost all the time is in the C compiles and that is done a per-Nim-module basis in .cache/nim and in terms of wall clock time that is done in parallel. That said, all my multi-commands are same-module. So, there may be some effect that I haven't noticed.

For me, personally, the biggest slow down is that tcc no longer works..any compile using that bombs out at the end of lib/system/atomics.nim for a couple months now. So, when doing compiles during writing/debugging, I have to use gcc which is 5-10x slower even with -O0.

As per your idea of delegating without importing..that sounds impossible. Nim needs to know their types at least. It may already be possible to "forward declare" some types, but at some point everything that gets called needs to be compiled into the program. You can factor the code into an external .so and then have thin wrapper code call that, I guess. There is loadLib and loadSym as used by e.g. my lc.nim.

There could be some kind of code caching gotchas/weaknesses. I don't pay that much attention to these.

c-blake commented 4 years ago

Anyway, we can keep discussing in this thread, but I am closing the issue per-cligen as I really doubt there is anything to be done in this package in particular to address these troubles.

pb-cdunn commented 4 years ago

How you tried zig for the back-end?

https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-clang.html

pb-cdunn commented 4 years ago

For cligen, I might switch to a lot of sub-executables, plus a top-level that can execute any of them. The only reason I use a single executable is that at my company it is very difficult to expose new executables, but I might have an easier time just installing them and calling them by relative path.

I definitely have to find a solution. It's taking over a minute to rebuild sometimes.

c-blake commented 4 years ago

I have not, though I am aware Zig is supposed to be fast. Not a bad idea. I have to install zig first, though. :-) It might also help your compile-performance problems, too. tcc is really lightning fast, though with its own inline ELF linker and everything. I'd be surprised if zig came close.

c-blake commented 4 years ago

Well, Zig is finicky about LLVM having been compiled with AArch64 support amongst other things. So, it will be a while before I try that.

I'm not sure what to advise you about your company's deployment constraints and how that interacts with Nim development. Back in y2k era I had a friend who had to wait hours for his big C++ compiles. A minute isn't so bad, but I agree I miss my fast tcc compiles and on my main devel box I can compile the whole of Nim in 2.5 minutes.

One of the other active cligen users is @kaushalmodi who, as it happens, has just been asking about "wrapper commands" in https://github.com/c-blake/cligen/issues/138 which might have some relevance to your mentioned solution. Kaushal has no power to change backend commands for other reasons (p4 being some proprietary version ctl thing), but the cligen concerns "rhyme" a bit.

pb-cdunn commented 4 years ago

Please let me know if/when tcc is working with Nim again.

c-blake commented 4 years ago

Will, do, Chris. I think --threads:on never quite worked with tcc, but otherwise the basic devel mode/edit/compile/run cycle always worked fine for me for years. When it first broke back in January, the error pointed immediately to some TODO/FIXME comment at the end of atomics.nim. So, I did not file an issue or look into it much, hoping rather that whoever broke it would fix it. It's been a long time now, though. Maybe it was forgotten about (or maybe there is something very idiosyncratic about my tcc set up that breaks it, but I doubt it...).

Right now when I try tcc the error is some error reported weirdly 64 lines past the end of atomics.nim. And the actual C compiler error is about the generated stdlib_io.nim.c:230 saying "invalid type" but the C code just looks like static N_INLINE(NIM_BOOL, nimAddInt)(NI a, NI b, NI* res); which is pretty generic. Maybe one of those N* cpp macros is being mis-defined.