p5pRT
commented
8 years ago From dwheeler@cpan.org
I have a file to which Iâve written:
"\xc3\xa5\xc3\xa5\xc3\xa5\x0a"
These bytes correspond to the UTF-8 string:
"Ă„Ă„Ă„\n"
I want to read this file into a scalar\, so I wrote this Perl:
sub slurp { my ($file) = @_; open my $fh\, "\<:encoding(UTF-8)"\, $file or die $!; return '' if eof $fh; local $/; return \<$fh>; }
my $fn = shift or die "Usage: $0 [file]\n"; slurp $fn;
Works great on my Mac and on *nix machines\, but not on Windows. There it emits a warning:
utf8 "\xA5" does not map to Unicode at try.pl line 6
I can fix it by changing the I/O layer to :raw:encoding(UTF-8). My guess is that it reads it in as raw bytes\, first\, then does the conversion. I can also break it on my Mac by changing the layer to :crlf:encoding(UTF-8).
But I donât understand why the encoding layerâs parsing of a file with \x0a line endings should vary by platform. Sure\, line ending on Windows might typically be \x0d\x0a\, but why would the I/O layer care when Iâve told it that the file encoding is UTF-8?
Note that this does not occur with data in memory. This works fine on both platforms:
use Encode qw(decode_utf8); my $data = "\xc3\xa5\xc3\xa5\xc3\xa5\x0a"; decode_utf8 $data;
Is the I/O layer assuming that\, because weâre on Windows\, line endings need to be converted to \r\n before decoding? Is it implicitly using :crlf on Windows? Doesnât seem like itâd be necessary if Iâve already told it what encoding to use\, and shouldn't bork that encoding in any event.
Migrated from rt.perl.org#127668 (status was 'open')
Searchable as RT127668$