assarbad
commented
2 years ago The following .tar file contains the file names that cause the error renamer-error.tar.gz.
This should hopefully ensure that the file names appear on the target system the same as on mine. All files have been truncated to zero size for convenience.
The issue appears to be around the second code point in the file name starting with Z.
After unpacking this .tar file on a n ext4 and btrfs partition, respectively, I am seeing the same issue with renamer, so it's not specific to NTFS or the ufsd (commercial) driver from Paragon.
For reference, this is the renaming I am attempting:
From
$ xxd -g 1 before-renaming.txt
00000000: 44 69 65 20 49 6e 76 65 73 74 6d 65 6e 74 62 61 Die Investmentba
00000010: 6e 6b 20 42 61 72 69 6e 67 73 20 42 61 6e 6b 20 nk Barings Bank
00000020: 5b 31 37 31 30 31 38 5f 62 61 72 69 6e 67 73 5f [171018_barings_
00000030: 69 6e 66 5d 2e 6d 70 34 0a 44 69 65 20 49 6e 76 inf].mp4.Die Inv
00000040: 65 73 74 6d 65 6e 74 62 61 6e 6b 20 4c 65 68 6d estmentbank Lehm
00000050: 61 6e 20 42 72 6f 74 68 65 72 73 20 5b 31 37 31 an Brothers [171
00000060: 30 31 38 5f 6c 65 68 6d 61 6e 5f 69 6e 66 5d 2e 018_lehman_inf].
00000070: 6d 70 34 0a 5a c3 b6 6c 69 62 61 74 20 2d 20 44 mp4.Z..libat - D
00000080: 65 72 20 6b 61 74 68 6f 6c 69 73 63 68 65 20 4c er katholische L
00000090: 65 69 64 65 6e 73 77 65 67 20 5b 31 30 38 30 34 eidensweg [10804
000000a0: 36 34 35 5d 2e 6d 70 34 0a 645].mp4.To
$ xxd -g 1 after-renaming.txt
00000000: 44 69 65 20 49 6e 76 65 73 74 6d 65 6e 74 62 61 Die Investmentba
00000010: 6e 6b 20 42 61 72 69 6e 67 73 20 42 61 6e 6b 20 nk Barings Bank
00000020: 5b 31 37 31 30 31 38 5f 62 61 72 69 6e 67 73 5f [171018_barings_
00000030: 69 6e 66 5d 2e 6d 70 34 0a 44 69 65 20 49 6e 76 inf].mp4.Die Inv
00000040: 65 73 74 6d 65 6e 74 62 61 6e 6b 20 4c 65 68 6d estmentbank Lehm
00000050: 61 6e 20 42 72 6f 74 68 65 72 73 20 5b 31 37 31 an Brothers [171
00000060: 30 31 38 5f 6c 65 68 6d 61 6e 5f 69 6e 66 5d 2e 018_lehman_inf].
00000070: 6d 70 34 0a 5a c3 b6 6c 69 62 61 74 20 2d 20 44 mp4.Z..libat - D
00000080: 65 72 20 6b 61 74 68 6f 6c 69 73 63 68 65 20 4c er katholische L
00000090: 65 69 64 65 6e 73 77 65 67 2e 6d 70 34 0a eidensweg.mp4.The respective part (Zölibat) seems not to change at all, yet it's the part renamer complains about:
5a c3 b6 6c 69 62 61 745a c3 b6 6c 69 62 61 74
As far as I could ascertain the c3 b6 translates fine to ö (in HTML ö).
mtimkovich
When on Ubuntu 20.04 with UTF-8 as code page I am seeing the following:
For the issue to surface I need to:
I am still collecting more information on this on my end, but I suspect some sort of normalization issue. This happens on an NTFS drive (which may play a role) with the commercial version of the Paragon NTFS driver (may also play a role). Could also have to do with NTFS as storage (which uses UTF-16 internally and may use other Unicode normalization rules).
I'll have to try to narrow this down further, but wanted to already bring this to your attention to the extent to which I have gathered information.
What would be great to see, however, would be the real bytes underneath. I mean I trust Rust to figure out if we end up in the middle of a surrogate or so, but since environmental factors like (potentially) normalization on part of the editor, the file system etc could play a role, it would be great to see input and output as raw bytes in case of this sort of panic.