This PR has complete support for characters upto 3 byte codepoints.
If a source dataset has characters which are longer than 3 codepoint-bytes there could be an exception due to characters that cant be mapped.
A followup to fix this limitation is under the works.
Overview
In order to do parallel range queries in order to read a table with String column in it's index, we need to split the space of string from min to max into chunks. Unlike integers, strings can't be directly
The logic for mapping the string has to take care of various database nuances like:
Control Characters like control-z are ignored for comparisons at all positions.
Space is ignored for comparison at trailing positions. Depending on the character-set, there could be more than one character that represents a space like EM quad, non-breaking space.
Space is not ignored at non-trailing positions in the comparison and characters like tab or new-line could compare less then space at non-trailing positions depending on the collation.
Note on Unit testing to collation order query
It's not possible to directly unit test the collation-order query on derby. To circumvent this, we have dumped the output of
utf8mb4_0900_ai_ci (case and accent insensitive), utf8mb4_0900_as_cs (case and accent sensitive) and utf8mb4_unicode_ci (pad space) collations for the purpose of unit tests. The process to add such an output to the tests is documented as well.
TODOS (for current PR)
End to End migration manual test.
TODOS (for followup PR)
MySQL 5.7 compatible Collation order and index-discovery queries.
Making the side-input at a DB level (instead of current table level)
Implementing Logic for Splitting of strings.
Limitation
This PR has complete support for characters upto 3 byte codepoints. If a source dataset has characters which are longer than 3 codepoint-bytes there could be an exception due to characters that cant be mapped. A followup to fix this limitation is under the works.
Overview
In order to do parallel range queries in order to read a table with String column in it's index, we need to split the space of string from min to max into chunks. Unlike integers, strings can't be directly The logic for mapping the string has to take care of various database nuances like:
control-z
are ignored for comparisons at all positions.tab
ornew-line
could compare less then space at non-trailing positions depending on the collation.Note on Unit testing to collation order query
It's not possible to directly unit test the collation-order query on derby. To circumvent this, we have dumped the output of
utf8mb4_0900_ai_ci
(case and accent insensitive),utf8mb4_0900_as_cs
(case and accent sensitive) andutf8mb4_unicode_ci
(pad space) collations for the purpose of unit tests. The process to add such an output to the tests is documented as well.TODOS (for current PR)
TODOS (for followup PR)