gb18030 encoder using index gb18030 ranges pointer

[chncc]

https://encoding.spec.whatwg.org/commit-snapshots/75b988c17cc2b90266e69526f399c7916c3e0ef0/#index-gb18030-ranges-pointer https://encoding.spec.whatwg.org/commit-snapshots/75b988c17cc2b90266e69526f399c7916c3e0ef0/#gb18030-encoder

index-gb18030-ranges.txt maps more-or-less continuous pointers to Unicode values. Reverse process may produce invalid results due to the fact that Unicode values are not continuous: U+00A4, U+00A5 maps to 81 30 84 36; U+00A7, U+00A9 maps to 81 30 84 38; U+00A8, U+00AA maps to 81 30 84 39; Euro U+20AC, U+20AD maps to 81 36 B3 32, and so on.

A verification should be added to gb18030 encoder, like: 8a. Set verification code point to the index gb18030 ranges code point for pointer. 8b. If verification code point is not code point then return error with code point.

or have I missed something?

[annevk]

It only uses index gb18030 ranges if index gb18030 produced no result. So, e.g., I don't see how U+00A4 would reach that point.

[chncc]

I'm sorry for my lack of precision. At the beginning of my post I've included some ''irreversible'' codes as an example of such ``irreversibility''. I agree that only surrogates (among all strange codes) pass to that point at the moment. It is not a problem: they are denied already by an assertion at process an item.

I'm rather wondering about some kind of protection from inconsistencies of two data files (or promise of consistency). Potential holes at pointer => code point are explicitly eliminated by gb18030 decoder, next at index gb18030 ranges code point and finally while looking at index-gb18030-ranges.txt description.

If we consider code point => pointer direction (encoding) there is no obvious way to state a consistency. Especially the standard says: we are using two files for [only/primarily] space saving purposes. At the first view we have 1:1 map and next not always reversible function for remnants. It rings the bell: ``what if some irreversible value passed?'' (those bells in my head when I'm returning to the standard every time ;-).

I know that an adding my verification is too heavy and unnecessary. But please consider some light-weight solution: maybe your short and concise sentence ``It only uses...'' as a note after point 7. in gb18030 encoder; some comment at index table description or so. Thank you for your answer and time.

[annevk]

Apologies for the slow follow-up here, maybe something like this for the description:

... It therefore only superficially matches the GB18030-2005 standard for code points encoded as four bytes. It complements index gb18030 described above. See also ...

I suppose we could add this step between step 8 and 9 to the gb18030 encoder but I'd like @hsivonen or @inexorabletash's input as to whether that seems okay:

Assert: code point is the index gb18030 ranges code point for pointer.

If we make these changes, would you like to be acknowledged as chncc?

[inexorabletash]

@jungshik may be able to reason about this w/r/t ICU