search

facebookincubator / velox

A C++ vectorized database acceleration library aimed to optimizing query engines and data processing systems.
https://velox-lib.io/
Apache License 2.0
3.42k stars 1.12k forks source link

Parquet reader error "Write past Buffer capacity() 4000" #7617

Open qqibrow opened 10 months ago

qqibrow commented 10 months ago

Bug description

Expected behavior:
able to read the parquet file with array type that contains 30000 empty arrays. Both parquet-tools and presto parquet reader are able to read the file

lniu@devrestricted-lniu:~$ parquet-tools inspect /tmp/velox_test_data/native_parquet_reader_test7669682075122168858parquet_1700174964628

############ file meta data ############
created_by: parquet-mr version 1.12.2 (build 77e30c8093386ec52c3cfa6c34b7ef3321322c94)
num_columns: 1
num_rows: 30000
num_row_groups: 1
format_version: 1.0
serialized_size: 247

############ Columns ############
array_element

############ Column(array_element) ############
name: array_element
path: test.bag.array_element
max_definition_level: 3
max_repetition_level: 1
physical_type: INT32
logical_type: None
converted_type (legacy): NONE
compression: GZIP (space_saved: -39%)

Actual behavior: velox native parquet reader crashed. Not sure it's a tuning issue or bug.

System information

Velox System Info v0.0.2 Commit: 1e186e548833750cdee4b95d829711ddad78aba1 CMake Version: 3.16.3 System: Linux-5.4.0-1063-aws Arch: x86_64 C++ Compiler: /usr/bin/c++ C++ Compiler Version: 9.4.0 C Compiler: /usr/bin/cc C Compiler Version: 9.4.0 CMake Prefix Path: /usr/local;/usr;/;/usr;/usr/local;/usr/X11R6;/usr/pkg;/opt

Relevant logs

E1116 23:53:56.036094 4076080 Exceptions.h:69] Line: ../.././velox/buffer/Buffer.h:506, Function:checkEndGuardImpl, Expression:  Write past Buffer capacity() 4000, Source: RUNTIME, ErrorCode: INVALID_STATE
terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError'
  what():  Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Write past Buffer capacity() 4000
Retriable: False
Function: checkEndGuardImpl
File: ../.././velox/buffer/Buffer.h
Line: 506
Stack trace:
# 0  std::shared_ptr<facebook::velox::VeloxException::State const> facebook::velox::VeloxException::State::make<facebook::velox::VeloxException::make(char const*, unsigned long, char const*, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, bool, facebook::velox::VeloxException::Type, std::basic_string_view<char, std::char_traits<char> >)::{lambda(auto:1&)#1}>(facebook::velox::VeloxException::Type, facebook::velox::VeloxException::make(char const*, unsigned long, char const*, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, bool, facebook::velox::VeloxException::Type, std::basic_string_view<char, std::char_traits<char> >)::{lambda(auto:1&)#1})
# 1  facebook::velox::VeloxException::VeloxException(char const*, unsigned long, char const*, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, bool, facebook::velox::VeloxException::Type, std::basic_string_view<char, std::char_traits<char> >)
# 2  facebook::velox::VeloxRuntimeError::VeloxRuntimeError(char const*, unsigned long, char const*, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, bool, std::basic_string_view<char, std::char_traits<char> >)
# 3  void facebook::velox::detail::veloxCheckFail<facebook::velox::VeloxRuntimeError, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>(facebook::velox::detail::VeloxCheckFailArgs const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
# 4  facebook::velox::AlignedBuffer::checkEndGuardImpl() const
# 5  facebook::velox::Buffer::checkEndGuard() const
# 6  facebook::velox::Buffer::setSize(unsigned long)
# 7  facebook::velox::parquet::ListColumnReader::setLengthsFromRepDefs(facebook::velox::parquet::PageReader&)
# 8  facebook::velox::parquet::(anonymous namespace)::readLeafRepDefs(facebook::velox::dwio::common::SelectiveColumnReader*, int, bool)
# 9  facebook::velox::parquet::ensureRepDefs(facebook::velox::dwio::common::SelectiveColumnReader&, int)
# 10 facebook::velox::parquet::ListColumnReader::read(int, folly::Range<int const*>, unsigned long const*)
# 11 facebook::velox::dwio::common::SelectiveStructColumnReaderBase::read(int, folly::Range<int const*>, unsigned long const*)
# 12 facebook::velox::parquet::StructColumnReader::read(int, folly::Range<int const*>, unsigned long const*)
# 13 facebook::velox::dwio::common::SelectiveStructColumnReaderBase::next(unsigned long, std::shared_ptr<facebook::velox::BaseVector>&, facebook::velox::dwio::common::Mutation const*)
# 14 facebook::velox::parquet::ParquetRowReader::next(unsigned long, std::shared_ptr<facebook::velox::BaseVector>&, facebook::velox::dwio::common::Mutation const*)
# 15 main
# 16 __libc_start_main
# 17 _start

*** Aborted at 1700178836 (Unix time, try 'date -d @1700178836') ***
*** Signal 6 (SIGABRT) (0x3e47003e3230) received by PID 4076080 (pthread TID 0x7f3236df4a80) (linux TID 4076080) (maybe from PID 4076080, UID 15943) (code: -6), stack trace: ***
    @ 0000000001d4b6c7 folly::symbolizer::(anonymous namespace)::innerSignalHandler(int, siginfo_t*, void*)
                       /home/lniu/code/velox_new/velox/folly/_build/../folly/experimental/symbolizer/SignalHandler.cpp:449
    @ 0000000001d4b7a8 folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*)
                       /home/lniu/code/velox_new/velox/folly/_build/../folly/experimental/symbolizer/SignalHandler.cpp:470
    @ 0000000000000000 (unknown)
    @ 000000000004300b gsignal
    @ 0000000000022858 abort
    @ 0000000000000000 (unknown)
    @ 0000000000000000 (unknown)
    @ 00000000000aa3f6 std::terminate()
    @ 00000000000aa6a8 __cxa_throw
    @ 0000000001beb458 __cxa_throw
                       /home/lniu/code/velox_new/velox/folly/_build/../folly/experimental/exception_tracer/ExceptionTracerLib.cpp:108
    @ 0000000001b5c9fc void facebook::velox::detail::veloxCheckFail<facebook::velox::VeloxRuntimeError, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>(facebook::velox::detail::VeloxCheckFailArgs const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
                       /home/lniu/code/velox_new/velox/_build/debug/../.././velox/common/base/Exceptions.h:85
                       -> /home/lniu/code/velox_new/velox/_build/debug/../../velox/common/base/Exceptions.cpp
    @ 0000000000f9d7a0 facebook::velox::AlignedBuffer::checkEndGuardImpl() const
                       /home/lniu/code/velox_new/velox/_build/debug/../.././velox/buffer/Buffer.h:506
                       -> /home/lniu/code/velox_new/velox/_build/debug/../../velox/dwio/parquet/reader/ParquetColumnReader.cpp
    @ 0000000000f9d308 facebook::velox::Buffer::checkEndGuard() const
                       /home/lniu/code/velox_new/velox/_build/debug/../.././velox/buffer/Buffer.h:190
                       -> /home/lniu/code/velox_new/velox/_build/debug/../../velox/dwio/parquet/reader/ParquetColumnReader.cpp
    @ 0000000000f9d1b7 facebook::velox::Buffer::setSize(unsigned long)
                       /home/lniu/code/velox_new/velox/_build/debug/../.././velox/buffer/Buffer.h:123
                       -> /home/lniu/code/velox_new/velox/_build/debug/../../velox/dwio/parquet/reader/ParquetColumnReader.cpp
    @ 00000000015ea651 facebook::velox::parquet::ListColumnReader::setLengthsFromRepDefs(facebook::velox::parquet::PageReader&)
                       /home/lniu/code/velox_new/velox/_build/debug/../../velox/dwio/parquet/reader/RepeatedColumnReader.cpp:283
    @ 00000000015e8f5b facebook::velox::parquet::(anonymous namespace)::readLeafRepDefs(facebook::velox::dwio::common::SelectiveColumnReader*, int, bool)
                       /home/lniu/code/velox_new/velox/_build/debug/../../velox/dwio/parquet/reader/RepeatedColumnReader.cpp:43
    @ 00000000015e94ef facebook::velox::parquet::ensureRepDefs(facebook::velox::dwio::common::SelectiveColumnReader&, int)
                       /home/lniu/code/velox_new/velox/_build/debug/../../velox/dwio/parquet/reader/RepeatedColumnReader.cpp:106
    @ 00000000015ea75b facebook::velox::parquet::ListColumnReader::read(int, folly::Range<int const*>, unsigned long const*)
                       /home/lniu/code/velox_new/velox/_build/debug/../../velox/dwio/parquet/reader/RepeatedColumnReader.cpp:292
    @ 00000000018d6214 facebook::velox::dwio::common::SelectiveStructColumnReaderBase::read(int, folly::Range<int const*>, unsigned long const*)
                       /home/lniu/code/velox_new/velox/_build/debug/../../velox/dwio/common/SelectiveStructColumnReader.cpp:167
    @ 00000000015fcfd7 facebook::velox::parquet::StructColumnReader::read(int, folly::Range<int const*>, unsigned long const*)
                       /home/lniu/code/velox_new/velox/_build/debug/../../velox/dwio/parquet/reader/StructColumnReader.cpp:96
    @ 00000000018d5a8b facebook::velox::dwio::common::SelectiveStructColumnReaderBase::next(unsigned long, std::shared_ptr<facebook::velox::BaseVector>&, facebook::velox::dwio::common::Mutation const*)
                       /home/lniu/code/velox_new/velox/_build/debug/../../velox/dwio/common/SelectiveStructColumnReader.cpp:89
    @ 0000000000f7206d facebook::velox::parquet::ParquetRowReader::next(unsigned long, std::shared_ptr<facebook::velox::BaseVector>&, facebook::velox::dwio::common::Mutation const*)
                       /home/lniu/code/velox_new/velox/_build/debug/../../velox/dwio/parquet/reader/ParquetReader.cpp:736
    @ 0000000000f55863 main
                       /home/lniu/code/velox_new/velox/_build/debug/../../velox/dwio/parquet/tests/reader/ParquetReaderExample.cpp:78
    @ 0000000000024082 __libc_start_main
    @ 0000000000f52b7d _start
Aborted
qqibrow commented 10 months ago

The test file: https://www.dropbox.com/scl/fi/m70zs4yh4e1xi9ogyluy4/issue7617.parquet?rlkey=678thsiwpwz7nord5qf1bj0wr&dl=0

qqibrow commented 10 months ago

just found that reducing Max number of rows to read from 1000 to 500 can mitigate the error:

 while (rowReader->next(1000, result)) {    // change 1000 to 500 can mitigate the error 
    for (vector_size_t i = 0; i < result->size(); i++) {
      std::cout << result->toString(i) << std::endl;
    }
  }
yingsu00 commented 9 months ago

@qqibrow Will you be able to create a draft PR to add this test in ParquetReaderTest? You can mark it as disabled and add a comment mentioning this issue until the bug is fixed. Is this urgent? If not I think it's better to wait for my refactor in which the null handling would be simplified.

makagonov commented 9 months ago

Just FYI, I couldn't repro the issue:

makagonov@makagonov-xps:~/presto/presto-native-execution$ cmake-build-debug/velox/velox/dwio/parquet/tests/reader/velox_scan_parquet ~/Downloads/issue7617.parquet ./out.txt
number of rows: 30000
velox type: ROW<test:ARRAY<INTEGER>>
qqibrow commented 9 months ago

@makagonov did you change the batch size to 1000?

qqibrow commented 9 months ago

@makagonov build with #7642 . run the command:

/home/lniu/code/velox_new/velox/_build/debug/velox/dwio/parquet/tests/reader/velox_dwio_print_parquet --file_path=/home/lniu/issue7617.parquet --batch_size=1000

will reproduce the issue. I just tried. default batch size (500) works fine and cannot surface the issue.

yingsu00 commented 2 months ago

@qqibrow Is this still not fixed?