nonstandard node.js build system

[cyisfor]

I know, I know, but hear me out. I've been sitting on this a while, and I think people might not be aware of the subtle issues I'm seeing with using node.js to build cjdns. So I'd like to mention those here, and perhaps suggest my reasoning in why going through all the trouble to remove it might be worth doing.

Generated code is amazing and versatile, and we should by all means use it to its fullest extent... but... in system critical super secure applications like cjdns? It adds a layer of complexity, where when you see #define RootTest_mainName <?js return '"' + file.RootTest_mainName + '"' ?> there's no way to determine what the value of that is, without parsing it. So you can't precisely say what your code is going to do, until after it's been processed with javascript. The C preprocessor is terrible and limited, but... in this case that sort of limit can be a good thing? I don't have to go looking through every other single file in the build tree to divine what "main" is in the file.RootTest_mainFunc = RootTest_toStr(main); expression, which has no indication where it got the variable in the first place. With CPP, I look at the includes, then the includes of that. It's not as easy to follow as modules, but... node.js is a lot worse. Magic variables like "main" in critical code like this make me really uneasy.

So... I tried to compile cjdns the other day, and it required me to install openssl. Why? Because nodejs is a mess. When you depend on a third party project like node.js, you inherit their politics, and if they want to use proven insecure routines in OpenSSL, because LibreSSL doesn't provide build instructions for MSVC, then you (and by extension we) are stuck with that. Depending on third party libraries and applications is what I'd call strong coupling with some very unreliable, inconsistent and human interfaces. Yes, it's nice to be able to get something downloaded over https, but coupling your preprocessor in together with that? Better to have two separate tools, so one can be switched out easily, in the case of upstream drama and/or better alternatives.

I'm not suggesting it's possible to replace node.js with something standard and stable like (ignores screams of the damned) autotools. I wasn't kidding about magic variables, and I'm at a loss to say how to recreate that source in a less incomprehensible manner. I'm just saying that it'd be really nice to do that, and if I myself want to spend the hours (days) it takes to unspool all that spaghetti, I'd like to at least make a case here that there's a good reason for doing so.

I'm more of a writer than a coder, if that wasn't obvious, but I think I'm good enough to judge that node.js is giving us some worrying problems. Complicated, political, and upstream software. Generated code that's truly inscrutable in an application that should be very scrutable. And departure from any standard, which makes new contributors to cjdns balk at the difficult learning curve, wasting time trying to figure out what "main" is, instead of coding to increase reliability or performance. It might not be possible to remove node.js, but... it is an issue worth mentioning, I think.

[cyisfor]

...and now I realize that "main" is just the string "main" because a CPP macro stringified it, before a javascript macro assigned it. ...dang that's confusing.

[cyisfor]

Oh, now that I think on it, openssl might not be required. We'd just have to make a phony "Crypto" module because of builder.js's absolutely ridiculous use of cryptographic hashes to (fail to) track dependencies. I tried making one that just calls sha256sum as a subprocess, and it actually works fine... the point still stands about overcomplication though.

Honestly, the vast majority of this javascript "magic" seems to be in tracking what objects link where (which could easily (and probably more clearly) be listed in a Makefile.am instead of scattered inside each .c file), and in making those tests fancier, redefining "main" so that each test can be dispatched from a single executable. It just really feels like unnecessarily added complexity to me. Not having luck refactoring it so far, but... I'll keep poking at it.

[cyisfor]

Okay, so openssl isn't required, and there's an unofficial libressl fork of node.js, so there's no physical problem anymore. And in learning autotools, I found that they decided it was a good idea to disallow reassigning variables, and they also preemptively reassemble every Makefile, copying and pasting all included files into it, so separate files are not treated like... separate files. So basically even I'm sick of trying to get autotools to work, when there have to be workarounds for some versions of "make" which don't even allow you to include files.

The node.js build sucks, is broken, and doesn't track dependencies right, but totally recompiling cjdns every time is cheap anyway, and it's not really clear how using CPP for a godawful macro parser is worse than using javascript. Also all the tests have javascript embedded in their C code, which CPP cannot replicate (complex compile time logic).

Whatever fixes the build system, stopping it from recompiling everything every time, this is not gonna be it. And whatever removes the extra layer of complexity, allowing cjdns to be plain C code, rather than javascript templates that sorta look like C code, but are meaningless until processed into C code, autotools isn't gonna help with that one bit.