Closed gperciva closed 7 months ago
There's a typo in libarchive-2.7 which stopped the code from doing what it was supposed to.
Modern libarchive uses a different method of doing the same thing (so the typo wasn't explicitly "fixed", but it's not broken any more).
To investigate this, I created utf8-good.tar
and utf8-bad.tar
(in the attached zip file, because github doesn't allow us to upload tar
files).
Both tar files contain a single 0-byte file called filename-😀
. (that's U+1F600, a smile emoji)
In the "good" version, the tar file begins:
00000000: 5061 7848 6561 6465 722f 6669 6c65 6e61 PaxHeader/filena
00000010: 6d65 2df0 9f98 8000 0000 0000 0000 0000 me-.............
whereas the bad one encodes the smile emoji as a surrogate pair, and begins:
00000000: 5061 7848 6561 6465 722f 6669 6c65 6e61 PaxHeader/filena
00000010: 6d65 2ded a0bd edb8 8000 0000 0000 0000 me-.............
The diff on those two lines is:
-00000010: 6d65 2df0 9f98 8000 0000 0000 0000 0000 me-.............
+00000010: 6d65 2ded a0bd edb8 8000 0000 0000 0000 me-.............
(The full diff has more: the filename is repeated 3 times, and the checksum and path length changes. But those aren't important.)
I tested libarchive-2.7, libarchvie 3.6.0 (the default in freebsd 12.4), and gnu tar 1.35, with tar -tf utf8-good.tar
and tar -tf utf8-bad.tar
.
utf8-good.tar
.utf8-bad.tar
.utf8-bad.tar
.
$ gtar -tf utf8-bad.tar
filename-\355\240\275\355\270\200
(that's the bytes in octal)
$ ~/src/libarchive-2.7/b/bsdtar -tf utf8-bad.tar
bsdtar: Pathname in pax header can't be converted to current locale.
filename-\355\240\275\355\270\200
bsdtar: Error exit delayed from previous errors.
When I tried applying this fix to libarchive-2.7, it worked fine:
$ ~/src/libarchive-2.7/b/bsdtar -tf utf8-bad.tar
filename-😀
(that's the modified libarchive-2.7)
Amusingly, the code which is described as:
has a typo: 0xDC00 should be 0xD800.
The comment mentions a "leading surrogate", which is a synonym for a high-surrogate code unit:
What libarchive is doing here is adjusting for an invalid conversion of UTF16 to UTF8; this adjustment is now known as the Compatibility Encoding Scheme for UTF-16: 8-Bit (CESU-8), published as the Unicode Technical Report #26 [1].
[1] https://www.unicode.org/reports/tr26/
Essentially, if libarchive detects a surrogate pair (not allowed in UTF-8 [2]), it tries to construct the desired unicode value (as per CESU-8).
[2] That unicode value should be encoded with 4 octets, whereas the surrogate pair requires 6 octets.
Caveat:
I used the term CESU-8 because later libarchive calls it that -- in fact, they moved this functionality into cesu8_to_unicode() in:
However, "cesu" does not appear in the tarsnap git repo.