This new select works fine in most cases, as long as you don't try to paginate using something other than the primary key. When trying to paginate using a non-primary key it will return columns in an order which is different from the table schema. Ghostferry have before this bug was detected assumed that data is read and written in the table structure column order. But since USING(FallbackColumn) will put FallbackColumn first in the result set, it will confuse Ghostferry and lose track of which value needs to go where since the batch writer only have access to the table structure column order. The inline verifier caught this and crashed Ghostferry correctly.
This PR resolves this by passing the column order in the RowBatch struct. Making it possible for the batch writer to generate a correct INSERT query. This allow us to do changes to our queries fetching data without having to be too careful with in which order columns are returned.
Resolving #285.
After optimizing
ShardedCopyFilter.BuildSelecta while back we switched to using a sub-select.This new select works fine in most cases, as long as you don't try to paginate using something other than the primary key. When trying to paginate using a non-primary key it will return columns in an order which is different from the table schema. Ghostferry have before this bug was detected assumed that data is read and written in the table structure column order. But since
USING(FallbackColumn)will putFallbackColumnfirst in the result set, it will confuse Ghostferry and lose track of which value needs to go where since the batch writer only have access to the table structure column order. The inline verifier caught this and crashed Ghostferry correctly.This PR resolves this by passing the column order in the
RowBatchstruct. Making it possible for the batch writer to generate a correct INSERT query. This allow us to do changes to our queries fetching data without having to be too careful with in which order columns are returned.