Open sesc opened 8 years ago
Hi!
Sorry it took us so long but the relevant commit seems to do with Tie::File and is not related here - see commit 21e63fa9a4ff7fe876f13adfff659c8ea628d0d9 . Can you file a new pull-req?
Hi!
Sorry for the late reply but in the branch I am getting test failures:
Test Summary Report
-------------------
test/in-place.t (Wstat: 2304 Tests: 0 Failed: 0)
Non-zero exit status: 9
Parse errors: Bad plan. You planned 5 tests but ran 0.
test/os.t (Wstat: 0 Tests: 14 Failed: 0)
TODO passed: 5-6
test/tie_file.t (Wstat: 2304 Tests: 2 Failed: 0)
Non-zero exit status: 9
Files=56, Tests=373, 6 wallclock secs ( 0.19 usr 0.04 sys + 2.99 cusr 0.34 csys = 3.56 CPU)
Result: FAIL
/home/shlomif/apps/perl/brew/perls/perl-5.24.0/lib/site_perl/5.24.0/auto/share/dist/Zilla-Dist/Makefile:75: recipe for target 'test' failed
make: *** [test] Error 1
'make -f /home/shlomif/apps/perl/brew/perls/perl-5.24.0/lib/site_perl/5.24.0/auto/share/dist/Zilla-Dist/Makefile test' failed: No such file or directory
can you look into fixing them?
Ah, sorry! I can reproduce your issue and have found a working alternative. Will update my branch and let you know.
Okay, I'm done.
My new approach has Tie::File open the file itself and set the binmode, using that module's (unfortunately undocumented) 'discipline' option (see perldoc Tie::File
L112ff).
I have also expanded the tie_file test to verify the utf8 functionality a bit.
Should we politely ask the owner of Tie::File to document the discipline option now?
I went looking for a place to execute your suggestion, @akarelas, and I think I now know why they didn't mention it... UTF-8 support in Tie::File is seriously buggy! (However, I think they ought to have warned about the bug instead, since the problem same no matter if one tie()s with my 'discipline' approach here or via an utf8-enabled filehandle as in the bug report above.)
My test happened to exercise the one way that works: Appending multi-byte data to the end of the file. But if you change data in the tied array, the lengths of the old and replacement strings, and thus the range within the file to be updated, are calculated in bytes, corrupting the file.
For instance, if line 1 has an "ä" (two-byte UTF-8 char) on it and you replace it with "a" (one byte), then there is an extra empty line before the old line 2. On the other hand, if there is an "a" and you replace it with "ä", the first character of line 2 is clobbered by the newline that now has moved to the right.
The bug has been known and acknowledged but unfixed for 2 1/2 years, which implies that it isn't trivial.
@shlomif, I'm not quite sure how we are to handle this problem in our module, until Tie::File is fixed. Maybe we should add a bold warning of our own that tie()ing in UTF-8 mode only is safe for reading?
This pull request ensures correct encoding by first opening the file normally, with all encoding layers being applied, before its file handle is tied to an array reference.