Open revans2 opened 6 months ago
Just to be clear this also happens for schemas that have a STRUCT with strings in them, not just LISTS.
Created PR #16731 as fix.
List children should be "element" instead of "LIST".
Besides that, PR #16545 will fix this issue (repro added as unit test in this PR).
Describe the bug This is very similar to https://github.com/rapidsai/cudf/issues/14239, and because that is not done, then it is fine for this to be a dupe of that.
In Spark we are handed a read schema and some JSON data. Our goal is to pull out the parts of the JSON data that match the read schema. But for strings, this gets to be a little complicated, and any type can be coerced into a string. If the data is an array it is coerced into a string by converting the tokens to a JSON formatted string, if the data is a dict it is coerced into a string the same way.
mixed_types_as_string
was added in part to help make this happen, especially in the case of nested types. But that appears to only work at a top level column.Throws an exception about trying to create a nested column using a fixed width column factory.