Spec: console output

[zkochan]

I am not sure we should standardize this. However, logging is such a pain.... Just look at npm's logging, it has like dozens of supporting libraries and a lot of complexity to work with them.

I think we can have a standard like TAP and different reporters could make it prettier.

[alexanderGugel]

What about using something like ndjson? budo uses it and it looks great!

There are tons of formatters for it as well I think, e.g. garnish.

[zkochan]

ndjson looks good. We'll have to describe in what format should the logs be and some standard set of log types.

[billiegoose]

I like newline delimited JSON. I first encountered it in bunyan logging utility, but have seen it many places since, and have used it for many of my own projects since (and not just for logging, but any time series data).

But I'm not sure it's right for this use case. I assume the point is to standardize installer output so that debugging failed installs is easier. In my CI build steps I used to run npm installwith the verbosity level at 'silly', so I can try to figure out whether the install failed due to a network timeout, a package being unavailable (404), my npm cache server (50x), a package's node-gyp build step failing, or the OOM killer. However, the npm output would be enormous and be unmanageable, particularly in a CI web interface showing console output. I eventually wrote my own installer that reads package.json and executed 'npm install' for each dependency separately. This greatly reduced npm's memory usage, AND allowed me to handle errors individually. Instead of bailing at the first failed install, it would keep going, and at the end present a report listing which packages failed to install, so I could decide whether I actually needed to rebuild the image (I may not be testing the part of the app that requires those modules), instantly informed me whether the problem was with the new dependencies I just added, a problem with npmjs itself (symptom: 3-10 packages at random out of 100 failed), or with my build environment (node-gyp failed to compile X, or installing a dependency from Github failed because the git+ssh port was blocked).

[billiegoose]

So I had never thought of it before you mentioned TAP, but I think it would be really handy for installers to have a --output-tap flag. I'm less convinced that something as rich and flexible as ndjson would actually result in interoperability, because it would be so tempting to keep adding new fields.

Just to be clear, your not thinking of trying to standardize interactive installs right? I think there's too many options in terms of CLI interfaces... someone may decide to do more than just progress bars and write a full nurses interface. Or have dancing nyan cats.

[zkochan]

What do you mean by interactive installs?

I think we can standardize some basic log types like progress events, resolving, HTTP responses, errors. Everything else can be logged as some nonstandard log type maybe... But if these basic log types will be standardized, I think we could have some pretty nice, tool-agnostic install reporters

[billiegoose]

I mean installs run in a TTY. To quote an example from the Node manual

$ node -p "Boolean(process.stdout.isTTY)"
true
$ node -p "Boolean(process.stdout.isTTY)" | cat
false

In a TTY, I'd like to have spinners, progress bars, prompts ("This will install 3 million packages... Continue? [Y/n]"), ANSI colors, and other fancy features that look like absolute shit if written directly to a log file. I don't recall how many times, but I had to scroll through log files with npm2 that were 90% filled with that little spinner...

\
|
/
-
\
...

[billiegoose]

I think we can standardize some basic log types like progress events, resolving, HTTP responses, errors. ... But if these basic log types will be standardized, I think we could have some pretty nice, tool-agnostic install reporters

Sounds ambitious, but I'm not against trying, if only because it will make us carefully consider what type of events our programs could expose. Another possibility, I just want to bring up, would be to ditch the limitations of communicating through stdout and use an EventEmitter. If we choose to write our core installers (e.g. modinst) as a separate package from our cli programs (e.g. modinst-cli), then it would be easy for others to write not just reporters, but also input handling. For instance, right now there are big differences in the command names used by npm (npm install) and yarn (yarn add). I think the yarn names are attempting to improve upon the npm command names, but in doing so they've created friction. Converting old scripts that use "npm install -S" to use yarn would take effort. Instead, we could have one CLI interface, and allow specifying the underlying backend to use! So for example,

nodeinstall express --save --use=pnpm
nodeinstall express --save --use=ied --ied-specific-option
(and to round out)
nodeinstall express --save --use=mi --max-concurrent=4 --retry=3 --continue-on-fail --max-memory=200M

[zkochan]

Wouldn't that limit the supported package managers to be written in Node? ied is already being rewritten to Go

[billiegoose]

Grmph. Well.... good point. It has to use whatever the baseline for interprocess communication between languages is these days. Do we have something better than the old stdout & stderr yet? No? Darn.

</rant Like seriously. It's been decades since stdin, stdout & stderr were invented, and there's still no "better" way of passing a number between two processes that doesn't involve serializing it to text and back again? Why can't we at least pass floats... IEEE 754 standardized floating point numbers in 1985. What have Operating Systems designers been working on since then? /rant>

[zkochan]

I added ndjson to pnpm! Here's the PR: https://github.com/rstacruz/pnpm/pull/540

@alexanderGugel, it was a really good idea!