Closed jirutka closed 7 months ago
Sorry, fat-fingered the post button.
Hi, this is caused by a change in behavior of the bin/base64
utility since 0.5.0. In order to be bug-compatible with "standard" base64
, it now always outputs a newline after the output. Since \n
is not a valid base64 char, an error is thrown when the output is piped back into the decoder.
The fix is to run it as echo foo | ./build/bin/base64 -w0 | ./build/bin/base64 -d
. The -w0
option suppresses line-buffered output.
I hope you don’t consider this totally unexpected and confusing behaviour a feature. Since newline is not a valid character in base64, you can easily strip the trailing newline in the input to the decoder.
The base64
utility is intended as a demo project for the base64
library, to show how to integrate the library into a codebase. I don't consider it a core feature. Before the last release, the base64
library was very simple and behaved as a dumb pipe. stdout
could be piped to stdin
and all was well.
But over the years a few issues have been raised (#8 is the one I could find quickly) that were caused by the fact that the behavior was not identical to the system base64
. So in issue #115 I decided to fix that complaint and go for bug compatibility. So now the program always emits a trailing newline and always line-breaks the output by default. Gross, I agree, because I like dumb pipes, but it upholds the principle of least surprise.
But, and this is where you have a point, it seems like I didn't do a perfect job, because upon checking, the system base64
does seem to eat newlines, as demonstrated by:
echo foo | base64 | base64 -d
So I guess this issue is now about stripping newlines from the input.
Possible connection to #33.
Windows 7 x64 user here. Looking forward to test a fixed version.
I pushed an issue126
branch which fixes this issue, but in benchmarks it's about twice as slow as the system base64
's decoder on the same large input file. Probably because the algorithm is really naive and checks the input bytewise for \n
.
I guess I can merge it because a working but slow program is preferable over a non-working program, but it sucks to admit defeat to the system base64
:)
@aklomp, you mean I have to download that branch and compile it myself? o_O
@sergeevabc Well yes, how else do you get a binary? This project does not publish any binaries.
But anyway, I think I'll merge the branch today because it fixes the issue, and create a follow-up issue to deal with the slow decoding. The fix works, it's pretty trivial and I tested it locally. It's just quite slow because it scans the input twice.
Found the speed regression. Don't write to stdout
in the inner loop :) Anyway, now the timing is competitive again. Will merge soon.
Testing base64 0.5.2 built with w64devkit on Windows 7 x64. Oh my, the executable turned out to be almost 10 times larger than the source (116658 vs 12913 bytes).
$ busybox echo -n Z2l0aHVi | base64.exe -d
github
$ busybox echo -n Z2l0aHViIAo= | base64.exe -d
github
$ busybox echo -n Z2l0aHViCg== | base64.exe -d
github
$ busybox echo -n Z2l0aHViIA0K | base64.exe -d
github
I guess the devil is in the details like spaces, carriage returns and line feeds?
@sergeevabc I don't quite understand your comment. Are you reporting a bug?
Those strings seem to decode fine for me, including the whitespace, as shown with xxd
:
$ echo -n Z2l0aHVi | bin/base64 -d | xxd
00000000: 6769 7468 7562 github
$ echo -n Z2l0aHViIAo= | bin/base64 -d | xxd
00000000: 6769 7468 7562 200a github .
$ echo -n Z2l0aHViCg== | bin/base64 -d | xxd
00000000: 6769 7468 7562 0a github.
$ echo -n Z2l0aHViIA0K | bin/base64 -d | xxd
00000000: 6769 7468 7562 200d 0a github ..
About the binary size, not sure why it's so much larger, but I think static linking might have something to do with it.
There's an issue indeed, @aklomp.
@sergeevabc This issue has been closed. Please open a new issue if you want to file a bug report. By the way, I can't reproduce this in Linux; are you sure you are using the latest version?
@aklomp, by the way, you should consider displaying the version so that we don't have to guess, redownload and recompile the source code. Yes, I have the latest version (0.5.2). And I suspect that issue is related to handling CRLF.
base64 v0.5.0 works fine, v0.5.1 is broken on Alpine Linux Edge (both with gcc and cmake):