search

djc / rnc2rng

RELAX NG Compact to regular syntax conversion library
MIT License
16 stars 13 forks source link

Open rnc files using UTF-8 encoding #14

Closed brechtm closed 6 years ago

brechtm commented 6 years ago

rnc2rng currently uses the platform's default encoding when reading rnc files. This causes platform-dependent issues: brechtm/citeproc-py#81. I think it would be safe to assume that rnc files will be UTF-8 encoded, so it would make sense to always load rnc files using UTF-8.

djc commented 6 years ago

Thanks for the feedback! Your proposal makes sense to me. I see you've already prepared a fix, do you want to submit that as a PR? I can merge it and publish a new release if that helps.

brechtm commented 6 years ago

I haven't been able to test the patch yet. I have asked @yurivict who ran into the issue to verify whether it works.

brechtm commented 6 years ago

From the RELAX NG Compact Syntax reference, it seems that rnc files loaded from the filesystem should indeed always be opened as UTF-8 of UTF-16, depending on the BOM:

The textual representation of the RELAX NG schema in compact syntax may be either a sequence of Unicode characters or a sequence of bytes. In the latter case, the first stage is to transform the sequence of bytes to the sequence of characters. The sequence of bytes may have associated metadata specifying the encoding. One example of such metadata is the charset parameter in a MIME media type [RFC 2046]. If there is such metadata, then the specified encoding is used. Otherwise, the first two bytes of the sequence are examined. If these are #xFF followed by #xFE or #xFE followed by #xFF, then an encoding of UTF-16 [Unicode] will be used, little-endian in the former case, big-endian in the latter case. Otherwise an encoding of UTF-8 [Unicode] is used. It is an error if the sequence of bytes is not a legal sequence in the selected encoding.

brechtm commented 6 years ago

Passing encoding='utf-8' when opening the file is not enough, since rnc files could also be UTF-16 encoded.

djc commented 6 years ago

I suspect we could use codecs.open(), instead.

brechtm commented 6 years ago

I'm not sure how that would work. It doesn't seem to be able to detect the encoding of the file. The RNC spec suggests that files are either UTF-8 of UTF-16 encoded. The latter only when a BOM is present. So we could open the file in binary mode and read the first two bytes to determine the encoding to use (UTF-8, UTF-16 BE or UTF-16 LE).

brechtm commented 6 years ago

codecs.open() handles newlines differently apparently, so I used io.open() instead.