Open JkSelf opened 1 month ago
@mbasmanova @majetideepak @rui-mo Do you have any input? Thanks.
@JkSelf I met the same issue and please check the discussion in this issue https://github.com/facebookincubator/velox/issues/9821.
I think we will need to support writing non-flat vectors in Parquet writer. @majetideepak what do you think?
We do support writing regular dictionaries to Parquet. See https://github.com/facebookincubator/velox/pull/7025 I am curious as to why complex types have a problem. I'll look into this. We know constant vectors are flattened until Arrow supports writing REE encoding to Parquet. There is some issue somewhere for this.
https://github.com/facebookincubator/velox/pull/9406
I have a draft PR to support flatten complex vector.
Bug description
When we execute the SQL statement below using Gluten, it raises an exception with the message 'An unsupported nested encoding was found.
System information
Velox System Info v0.0.2 Commit: 467812fb0a0220fd6b479502dd89b21973a4a3b3 CMake Version: 3.28.3 System: Linux-5.4.0-167-generic Arch: x86_64 C++ Compiler: /usr/bin/c++ C++ Compiler Version: 9.4.0 C Compiler: /usr/bin/cc C Compiler Version: 9.4.0 CMake Prefix Path: /usr/local;/usr;/;/usr/local/lib/python3.8/dist-packages/cmake/data;/usr/local;/usr/X11R6;/usr/pkg;/opt
Relevant logs