simsong
commented
6 years ago Good work. Thanks for the kind mention of my Usenix Unicode article; it’s one of my favorites.
On Jul 20, 2018, at 4:11 PM, Alex Nelson notifications@github.com wrote:
DFXML can be generated for file systems that do not use UTF-8 encoding, or even that use arbitrary bytes. (For instance, HFS (not HFS+), allows any byte in a file name except the ASCII colon character.) DFXML has three objectives in case these bytes are encountered:
The original bytes should be preserved. The original bytes should be decoded into human-readable strings. The decoding process should present UTF-8 character strings (not byte strings) to a DFXML consumer without requiring additional scripting work to recognize transcodings (e.g. no mode should be required when opening a file that presents accented Latin characters originally encoded in macos-roman). When transcodings are done, though, they should be encoded in the DFXML and thus accessible by the DFXML API being used. Within the DFXML schema, these constructs will be added to any encodable/transcodable string:
The string element will have an optional attribute "original_encoding" to indicate a transcoding occurred. The original bytes encountered in the parse will be recorded in the optional attribute "original_bytes_base64". Some new semantics will result from this, because there are now 8 states (represented as sets of 3 conditions) we can have for any of these transcoding states, based on presence or absence of (A) the original bytes in base64, (B) the transcoding, and (C) the element's child text. Let absence of one of these conditions be represented as an underscore below as we walk through the state space:
[_] It could be that a fileobject was not meant to be named, such as with DFXML being used to distribute file hashes (e.g. with some modes of hashdeep). [__C] Absence of the original bytes and original encoding attributes implies the string was recorded in DFXML exactly as it was encountered. [B] This state is a bit uninformative, and should be avoided. [_BC] This implies original bytes were not recorded. This state should be avoided in case of an error with the script that performed the transcoding. If such an error occurred, the original bytes may not be derivable from the DFXML record. [A] The original bytes could not be rendered as a UTF-8 string. This can occur in an example where control characters are embedded as a file name. (H/t to @dd388 for finding a case where an HFS file system recorded "^C^B^AMove&Rename", special control characters for a Mac OS somewhere around version 7.) [AC] If original bytes are present and UTF-8 text is recorded, this shall imply the original encoding was UTF-8. This may be desired in cases where a unicode character could be encoded in multiple ways, such as with unicode combining characters. [AB] This state implies the original bytes are decodable, but do not have a corresponding point in unicode space. [ABC] All t's crossed, all i's dotted. In short, the preferred states are to include original bytes (conditions A), and include UTF-8 encodings when they are reachable (conditions C). If there is nothing more complex than ASCII, __C (omitting original bytes and original encoding) would be fine. If the character data are more complex than ASCII, and there is no chance of ambiguity, the original encoding can be omitted (condition A_C). If the file names are all ASCII, or unicode where all characters only have one representation, condition A_C would suffice; however, this may be unnecessarily difficult to determine on the fly, so unicode filenames may be best represented verbosely (condition ABC).
Thanks to @dd388 for assistance drafting this description, to @timothyryanwalsh for raising the matter, and @simsong for discussion and an article on Programming in Unicode. The original structure proposed in this Issue is close to what came from discussion in the DFXML library Issue.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.
ajnelson-nist
joachimmetz
DFXML can be generated for file systems that do not use UTF-8 encoding, or even that use arbitrary bytes. (For instance, HFS (not HFS+), allows any byte in a file name except the ASCII colon character.) DFXML has three objectives in case these bytes are encountered:
Within the DFXML schema, these constructs will be added to any encodable/transcodable string:
original_encoding" to indicate a transcoding occurred.original_bytes_base64".Some new semantics will result from this, because there are now 8 states (represented as sets of 3 conditions) we can have for any of these transcoding states, based on presence or absence of (
A) the original bytes in base64, (B) the transcoding, and (C) the element's child text. Let absence of one of these conditions be represented as an underscore below as we walk through the state space:___] It could be that afileobjectwas not meant to be named, such as with DFXML being used to distribute file hashes (e.g. with some modes ofhashdeep).__C] Absence of the original bytes and original encoding attributes implies the string was recorded in DFXML exactly as it was encountered._B_] This state is a bit uninformative, and should be avoided._BC] This implies original bytes were not recorded. This state should be avoided in case of an error with the script that performed the transcoding. If such an error occurred, the original bytes may not be derivable from the DFXML record.A__] The original bytes could not be rendered as a UTF-8 string. This can occur in an example where control characters are embedded as a file name. (H/t to @dd388 for finding a case where an HFS file system recorded "^C^B^AMove&Rename", special control characters for a Mac OS somewhere around version 7.)A_C] If original bytes are present and UTF-8 text is recorded, this shall imply the original encoding was UTF-8. This may be desired in cases where a unicode character could be encoded in multiple ways, such as with unicode combining characters.AB_] This state implies the original bytes are decodable, but do not have a corresponding point in unicode space.ABC] All t's crossed, all i's dotted.In short, the preferred states are to include original bytes (conditions
A**), and include UTF-8 encodings when they are reachable (conditions**C). If there is nothing more complex than ASCII,__C(omitting original bytes and original encoding) would be fine. If the character data are more complex than ASCII, and there is no chance of ambiguity, the original encoding can be omitted (conditionA_C). If the file names are all ASCII, or unicode where all characters only have one representation, conditionA_Cwould suffice; however, this may be unnecessarily difficult to determine on the fly, so unicode filenames may be best represented verbosely (conditionABC).Thanks to @dd388 for assistance drafting this description, to @tw4l for raising the matter, and @simsong for discussion and an article on Programming in Unicode. The original structure proposed in this Issue is close to what came from discussion in the DFXML library Issue.