Make ksdump output only JSON on success

[wader]

Hi, i think ksdump -f json would be easier to use with jq etc and be more what i would expect if only outputted JSON on stdout. Currently it also outputs some log messages on stdout making the output invalid JSON. Maybe on error it could print logs on stderr?

Reproduction:

$ docker run -v "$PWD:/share" -it --entrypoint=ksdump kaitai/ksv -f json /dev/zero some.ksy
Compilation OK
... processing substring_diff.ksy 0
...... loading substring_diff.rb
Classes loaded OK, main class = SubstringDiff
{
  "a": ""
}

Workaround i use at the moment is to pipe thru tail ksdump ... | tail +5 | jq ....

If good ide I guess same would apply for XML and YAML output.

I also noticed that there is no trailing new line for the last JSON output line, making it a bit awkward in a shell.

[generalmimon]

@wader:

Hi, i think ksdump -f json would be easier to use with jq etc and be more what i would expect if only outputted JSON on stdout. Currently it also outputs some log messages on stdout making the output invalid JSON.

Weird, this may have something to do with the fact that you're running ksdump in the Docker container. Need to investigate further.

I know for sure that at least when you clone https://github.com/kaitai-io/kaitai_struct_visualizer and run bin/ksdump directly, the log messages are printed to stderr and indeed only the JSON goes to stdout as expected (I'm redirecting stdout to separate .json files all the time and then processing the .json files further).

[wader]

Ah yeap your correct, tried in a shell inside the container and log messages end up on stderr 👍 tricky i was pretty sure docker can separate them from working the daemon API, and it turned out without -t it do work.

Would it still make sense to silence logs to stderr on success and only print things if some verbose flag is used?

[generalmimon]

@wader:

Would it still make sense to silence logs to stderr on success and only print things if some verbose flag is used?

Yes, it does. It's a bit ironic that ksdump has these verbose "dumps" turned on (and there's not even a way to turn them off), but kaitai-struct-compiler by default doesn't print anything.

I don't like this default quiet mode of KSC: --verbose file is much better because it informs you what target it's currently compiling to and what files were generated (and in what subdirectory of the specified "output directory" they are - when you compile with -t java and specify --java-package io.kaitai.struct.testformats, generated files will be put into the io/kaitai/struct/testformats/ subfolder, for example).

But this doesn't translate well to ksdump - in fact, nobody cares about the Ruby sources it generated under the hood (they're only generated into a temporary folder anyway which is deleted when ksdump exits, so the user can't do anything about this).

I actually think it's OK to just remove these debug logs from ksdump entirely. They have never been interesting to me either - only errors from KSC should be printed, otherwise it's just expected that it succeeds and we don't need any more useless ballast around it.

Yeah, and on the other hand, I'd really want --verbose file to become the default for kaitai-struct-compiler ;)

[wader]

My very subjective opinion is that is like tools that are quiet unless you tell them or something goes wrong :) but yeah in the case of ksdump it would expect such a tool to not output anything extra

[wader]

🎉

BTW sorry for little progress on the fq kaitai front. Ended up being involved in getting jq maintained again after 5 years without any release and hopefully jq 1.7 will be out soon!

[generalmimon]

Ended up being involved in getting jq maintained again after 5 years without any release and hopefully jq 1.7 will be out soon!

The pre-release version actually popped up in my feed recently - that's super exciting! jq is amazing for all sorts of JSON manipulation and it's easy and quick to use, but yeah, I wasn't sure whether there would be a successor of jq 1.6 after this time 😉