Open mterron opened 5 years ago
In brief, @base64d
has a documented limitation:
Note: If the decoded string is not UTF-8, the results are undefined.
This could be restated as follows:
Let $B be an arbitrary base64 string, then `$B | @base64d` is undefined
if `base64 -D <<< $B` is not a valid UTF-8 string.
Here base64
refers to the command-line program of that name, and I'm assuming that base64 -D
acts as the inverse of base64
.
Using the moreutils
program isutf8
to test this condition, we see
that the given base64 string does NOT satisfy it:
$ S="V0MsL2hwbyCcGs2AMrFAKaSjPl8OuX4OWAEip+idGVU="
$ base64 -D <<< "$S" | xxd -p
57432c2f68706f209c1acd8032b14029a4a33e5f0eb97e0e580122a7e89d
1955
$ base64 -D <<< "$S" | isutf8
(standard input): line 1, char 8, byte 8: Expecting bytes in the following ranges: 00..7F C2..F4.
Fair enough. I'd make the suggestion to document that the results of @base64d
is incorrect instead of merely undefined when decoding non UTF-8 strings.
Having put this to rest, is there any plans to have a way to "shell out" to call external utilities to aid in processing this edge cases?
@mterron - I believe the author or authors of this particular section of the official manual did not want to commit to any particular behavior at the time of writing, no doubt because, as you say, there is much to be said for raising an error condition.
As for shelling out -- yes, there are plans to support this (see e.g. https://github.com/stedolan/jq/issues/147 and https://github.com/stedolan/jq/issues/1614), but it won't help much in the present case, for the reason already stated.
@pkoppstein give a way to shell out I'll make it work :)
Thanks!
@mterron wrote:
I'll make it work :)
It appears you are not quite grasping the fact that the implementation of the shell-out function will (of necessity) be designed to prevent what I understand you want to do.
The "j" in jq can be understood as a commitment that, with two exceptions, every jq filter should produce strictly valid JSON, the exceptions being the values for NaN
and Infinite
, but even these two values are ultimately converted to valid JSON on output (e.g. echo Nan | jq .
#=> null).
This is not to say that every version of jq is guaranteed to reject non-JSON strings, but that's not for want of trying :-)
Oh I understand very well. xxd will convert the binary back to an hexadecimal string that is valid json
@mterron - Excellent. You might want to mention your use case on one of the shell-out tracking issues.
I think this is worth fixing and it does not break compatibility. It is useful if we can decode the image data from aws ec2 get-console-screenshot
with jq only.
Congratulations to @itchyny on gojq's @base64d, which passes the test using https://www.w3.org/2001/06/utf-8-wrong/UTF-8-test.html with flying colors. With B as a local copy of this file:
$ base64 < B > B.base64
$ diff <(gojq -Rrj '@base64d' B.base64) B
$
@pkoppstein does that mean that the Go version of jq
is now beating the 'original' jq
at its own game? I have made a very simple test, and gojq
surprisingly seemed to be much faster at decoding 118K of base64-encoded raw binary data, compared to jq
(which, of course, will produce garbage).
Now I'd really love to see a benchmark comparing the two :) [note: I'm well aware that gojq
does not fully implement everything that jq
does...]
Just tested it on macOS. gojq
worked beautifully.
@GwynethLlewelyn - gojq is unquestionably better than jq in several important respects -- not least that it is being actively maintained -- but unfortunately gojq has one major intrinsic drawback compared to jq, namely that in many cases that jq can handle, it simply runs out of memory. My understanding is that this is, in effect, by design -- that is, the problem apparently cannot be addressed within the scope of gojq's current design.
My experience has been that, in general, gojq tends to trade memory for speed, at least compared to jq, and the following statistics suggest that might be the case for @base64d too:
/usr/bin/time -lp gojq -Rrj '@base64d' B.base64 > /dev/null
3,436,544 maximum resident set size
1,499,136 peak memory footprint
/usr/bin/time -lp jq -Rrj '@base64d' B.base64 > /dev/null
1,937,408 maximum resident set size
1,204,224 peak memory footprint
There is also the matter of retaining the order of keys within objects, which for some users and applications is unimportant, but that is not always the case.
As a user of jq I'm sorely disappointed that a simple note on the limitation of this function isn't in the manpage with over 5 years of lead time.
I spent a good couple hours debugging why my base64 decoded strings were getting corrupted only to find that it's a known limitation of jq.
There is a note in the manpage about how interpolation behaves and nothing else.
@hachi - In the section on @base64d, the man page (https://jqlang.github.io/jq/manual/) says:
The inverse of @base64, input is decoded as specified by RFC 4648. Note\: If the decoded string is not UTF-8, the results are undefined.
Describe the bug The output of jq's @base64d does not match the output of
base64 -d
for the same stringTo Reproduce
Expected behavior The outputs of
echo "V0MsL2hwbyCcGs2AMrFAKaSjPl8OuX4OWAEip+idGVU=" | jq -Rr '@base64d' | xxd -p -c64
andecho "V0MsL2hwbyCcGs2AMrFAKaSjPl8OuX4OWAEip+idGVU=" | base64 -d | xxd -p -c64
should be equal.Environment (please complete the following information):
Additional context Seems to be related to binary values being encoded.