p5pRT
commented
20 years ago From @jlokier
Created by @jlokier
The "use bytes" pragma is useful for code which only wants to handle bytes.
substr()\, length()\, index()\, pos() and regex matching all ignore the UTF-8 flag on strings in the scope of this pragma.
However\, string concatenation does not take this pragma into account. Just like without the pragma\, it upgrades strings to UTF-8 if any of them are UTF-8.
This is quite inconsistent with the algebraic properties expected of byte strings\, such as:
length(substr($a\,0\,1).substr($a\,1)) == length($a)
Here's an example program which illustrates this:
$x="\x{100}abc"; $y="\x{80}def"; use bytes; print length($x)\, "\,"\, length($y)\, "\n"; $z = $x.substr($x\,0\,1).substr($x\,1).$y; print length($x)\, "\,"\, length($y)\, "\,"\, length($z)\, "\n";
The program prints:
5\,4 5\,5\,17
Those numbers make no sense. In bytes\, length($x) is 5 and length($y) is 4. After the concatenation\, the total is 17\, when it should logically be 14.
(This also shows length($y) is modified simply by $y being read\, reported as [perl #26901]. In this case\, length($y) is 4 before the concatenation but 5 after.)
Summary: I think string concatenation should _not_ upgrade non-UTF-8 strings to UTF-8 when they are concatenated inside the scope of "use bytes". A warning or even an exception may be appropriate.
Migrated from rt.perl.org#26905 (status was 'resolved')
Searchable as RT26905$