[CI][C++] arrow-dataset-dataset-writer-test in the ASAN UBSAN job failed

[kou]

Describe the bug, including details regarding any error messages, version, and platform.

Since https://github.com/apache/arrow/commit/7df1cdd0fe364f86eca15bfc482210994ba8e2f0 but the change isn't related to this failure because it doesn't touch the dataset module.

https://github.com/apache/arrow/actions/runs/6919998294/job/18823986112#step:6:3047

```text 53/92 Test #56: arrow-dataset-dataset-writer-test ............***Failed 1.88 sec Running arrow-dataset-dataset-writer-test, redirecting output into /build/cpp/build/test-logs/arrow-dataset-dataset-writer-test.txt (attempt 1/1) Running main() from ./googletest/src/gtest_main.cc [==========] Running 20 tests from 1 test suite. [----------] Global test environment set-up. [----------] 20 tests from DatasetWriterTestFixture [ RUN ] DatasetWriterTestFixture.Basic [ OK ] DatasetWriterTestFixture.Basic (13 ms) [ RUN ] DatasetWriterTestFixture.BasicFilePrefix [ OK ] DatasetWriterTestFixture.BasicFilePrefix (3 ms) [ RUN ] DatasetWriterTestFixture.BasicFileDirectoryPrefix [ OK ] DatasetWriterTestFixture.BasicFileDirectoryPrefix (3 ms) [ RUN ] DatasetWriterTestFixture.DirectoryCreateFails [ OK ] DatasetWriterTestFixture.DirectoryCreateFails (2 ms) [ RUN ] DatasetWriterTestFixture.MaxRowsOneWrite [ OK ] DatasetWriterTestFixture.MaxRowsOneWrite (12 ms) [ RUN ] DatasetWriterTestFixture.MaxRowsOneWriteWithFunctor [ OK ] DatasetWriterTestFixture.MaxRowsOneWriteWithFunctor (8 ms) [ RUN ] DatasetWriterTestFixture.MaxRowsOneWriteWithBrokenFunctor [ OK ] DatasetWriterTestFixture.MaxRowsOneWriteWithBrokenFunctor (2 ms) [ RUN ] DatasetWriterTestFixture.MaxRowsManyWrites [ OK ] DatasetWriterTestFixture.MaxRowsManyWrites (11 ms) [ RUN ] DatasetWriterTestFixture.MinRowGroup [ OK ] DatasetWriterTestFixture.MinRowGroup (9 ms) [ RUN ] DatasetWriterTestFixture.MaxRowGroup [ OK ] DatasetWriterTestFixture.MaxRowGroup (8 ms) [ RUN ] DatasetWriterTestFixture.MinAndMaxRowGroup [ OK ] DatasetWriterTestFixture.MinAndMaxRowGroup (8 ms) [ RUN ] DatasetWriterTestFixture.MinRowGroupBackpressure [ OK ] DatasetWriterTestFixture.MinRowGroupBackpressure (34 ms) [ RUN ] DatasetWriterTestFixture.ConcurrentWritesSameFile [ OK ] DatasetWriterTestFixture.ConcurrentWritesSameFile (13 ms) [ RUN ] DatasetWriterTestFixture.ConcurrentWritesDifferentFiles [ OK ] DatasetWriterTestFixture.ConcurrentWritesDifferentFiles (18 ms) [ RUN ] DatasetWriterTestFixture.MaxOpenFiles [ OK ] DatasetWriterTestFixture.MaxOpenFiles (12 ms) [ RUN ] DatasetWriterTestFixture.NoExistingDirectory [ OK ] DatasetWriterTestFixture.NoExistingDirectory (4 ms) [ RUN ] DatasetWriterTestFixture.DeleteExistingData [ OK ] DatasetWriterTestFixture.DeleteExistingData (4 ms) [ RUN ] DatasetWriterTestFixture.PartitionedDeleteExistingData [ OK ] DatasetWriterTestFixture.PartitionedDeleteExistingData (4 ms) [ RUN ] DatasetWriterTestFixture.LeaveExistingData [ OK ] DatasetWriterTestFixture.LeaveExistingData (4 ms) [ RUN ] DatasetWriterTestFixture.ErrOnExistingData [ OK ] DatasetWriterTestFixture.ErrOnExistingData (1 ms) [----------] 20 tests from DatasetWriterTestFixture (181 ms total) [----------] Global test environment tear-down [==========] 20 tests from 1 test suite ran. (182 ms total) [ PASSED ] 20 tests. ================================================================= ==9995==ERROR: LeakSanitizer: detected memory leaks #15 0x7fa79e9a497f in arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()::operator()() const /arrow/cpp/src/arrow/dataset/dataset_writer.cc:149:13 #16 0x7fa79e9a459b in std::enable_if<((!(std::is_void::value)) && (!(is_future::value))) && ((!(arrow::Future::is_empty)) || (std::is_same::value)), void>::type arrow::detail::ContinueFuture::operator(), std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&, arrow::Status, arrow::Future >(arrow::Future, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&) const /arrow/cpp/src/arrow/util/future.h:150:23 #17 0x7fa79e9a4271 in void std::__invoke_impl&, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&>(std::__invoke_other, arrow::detail::ContinueFuture&, arrow::Future&, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 #18 0x7fa79e9a3d75 in std::__invoke_result&, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&>::type std::__invoke&, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&>(arrow::detail::ContinueFuture&, arrow::Future&, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #19 0x7fa79e9a3b5f in void std::_Bind, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'())>::__call(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:420:11 #20 0x7fa79e9a3738 in void std::_Bind, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'())>::operator()() /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:503:17 #21 0x7fa79e9a3606 in arrow::internal::FnOnce::FnImpl, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'())> >::invoke() /arrow/cpp/src/arrow/util/functional.h:152:42 #22 0x7fa7843c42e6 in arrow::internal::FnOnce::operator()() && /arrow/cpp/src/arrow/util/functional.h:140:17 #23 0x7fa7843f14b7 in arrow::internal::ThreadPool::SpawnReal(arrow::internal::TaskHints, arrow::internal::FnOnce, arrow::StopToken, arrow::internal::FnOnce&&)::$_0::operator()() /arrow/cpp/src/arrow/util/thread_pool.cc:634:9 #24 0x7fa7843f12b6 in arrow::internal::FnOnce::FnImpl, arrow::StopToken, arrow::internal::FnOnce&&)::$_0>::invoke() /arrow/cpp/src/arrow/util/functional.h:152:42 #25 0x7fa7843c42e6 in arrow::internal::FnOnce::operator()() && /arrow/cpp/src/arrow/util/functional.h:140:17 #26 0x7fa7843ea9c4 in arrow::internal::WorkerLoop(std::shared_ptr, std::_List_iterator) /arrow/cpp/src/arrow/util/thread_pool.cc:457:11 #27 0x7fa7843e946b in arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_5::operator()() const /arrow/cpp/src/arrow/util/thread_pool.cc:618:7 #28 0x7fa7843e9293 in void std::__invoke_impl(std::__invoke_other, arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_5&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 #29 0x7fa7843e9083 in std::__invoke_result::type std::__invoke(arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_5&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 Indirect leak of 17 byte(s) in 1 object(s) allocated from: #0 0x55e00f5a4f1d in operator new(unsigned long) (/build/cpp/debug/arrow-dataset-dataset-writer-test+0x398f1d) (BuildId: 176377c2697ecb6a7274ce334f7231a363068218) #1 0x7fa7990820e2 in void std::__cxx11::basic_string, std::allocator >::_M_construct(char*, char*, std::forward_iterator_tag) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.tcc:219:14 #2 0x7fa78bd03cb4 in arrow::fs::internal::MockFileSystem::Impl::OpenOutputStream(std::__cxx11::basic_string, std::allocator > const&, bool, std::shared_ptr const&) /arrow/cpp/src/arrow/filesystem/mockfs.cc:398:44 #3 0x7fa78bd02c58 in arrow::fs::internal::MockFileSystem::OpenOutputStream(std::__cxx11::basic_string, std::allocator > const&, std::shared_ptr const&) /arrow/cpp/src/arrow/filesystem/mockfs.cc:735:17 #4 0x7fa78bb45b27 in arrow::fs::FileSystem::OpenOutputStream(std::__cxx11::basic_string, std::allocator > const&) /arrow/cpp/src/arrow/filesystem/filesystem.cc:250:10 #5 0x7fa79e9a51d6 in arrow::dataset::internal::(anonymous namespace)::OpenWriter(arrow::dataset::FileSystemDatasetWriteOptions const&, std::shared_ptr, std::__cxx11::basic_string, std::allocator > const&) /arrow/cpp/src/arrow/dataset/dataset_writer.cc:127:3 #6 0x7fa79e9a497f in arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()::operator()() const /arrow/cpp/src/arrow/dataset/dataset_writer.cc:149:13 #7 0x7fa79e9a459b in std::enable_if<((!(std::is_void::value)) && (!(is_future::value))) && ((!(arrow::Future::is_empty)) || (std::is_same::value)), void>::type arrow::detail::ContinueFuture::operator(), std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&, arrow::Status, arrow::Future >(arrow::Future, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&) const /arrow/cpp/src/arrow/util/future.h:150:23 #8 0x7fa79e9a4271 in void std::__invoke_impl&, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&>(std::__invoke_other, arrow::detail::ContinueFuture&, arrow::Future&, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 #9 0x7fa79e9a3d75 in std::__invoke_result&, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&>::type std::__invoke&, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&>(arrow::detail::ContinueFuture&, arrow::Future&, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #10 0x7fa79e9a3b5f in void std::_Bind, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'())>::__call(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:420:11 #11 0x7fa79e9a3738 in void std::_Bind, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'())>::operator()() /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:503:17 #12 0x7fa79e9a3606 in arrow::internal::FnOnce::FnImpl, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'())> >::invoke() /arrow/cpp/src/arrow/util/functional.h:152:42 #13 0x7fa7843c42e6 in arrow::internal::FnOnce::operator()() && /arrow/cpp/src/arrow/util/functional.h:140:17 #14 0x7fa7843f14b7 in arrow::internal::ThreadPool::SpawnReal(arrow::internal::TaskHints, arrow::internal::FnOnce, arrow::StopToken, arrow::internal::FnOnce&&)::$_0::operator()() /arrow/cpp/src/arrow/util/thread_pool.cc:634:9 #15 0x7fa7843f12b6 in arrow::internal::FnOnce::FnImpl, arrow::StopToken, arrow::internal::FnOnce&&)::$_0>::invoke() /arrow/cpp/src/arrow/util/functional.h:152:42 #16 0x7fa7843c42e6 in arrow::internal::FnOnce::operator()() && /arrow/cpp/src/arrow/util/functional.h:140:17 #17 0x7fa7843ea9c4 in arrow::internal::WorkerLoop(std::shared_ptr, std::_List_iterator) /arrow/cpp/src/arrow/util/thread_pool.cc:457:11 #18 0x7fa7843e946b in arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_5::operator()() const /arrow/cpp/src/arrow/util/thread_pool.cc:618:7 #19 0x7fa7843e9293 in void std::__invoke_impl(std::__invoke_other, arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_5&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 #20 0x7fa7843e9083 in std::__invoke_result::type std::__invoke(arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_5&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #21 0x7fa7843e8fd1 in void std::thread::_Invoker >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 #22 0x7fa7843e8ea0 in std::thread::_Invoker >::operator()() /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 #23 0x7fa7843e8373 in std::thread::_State_impl > >::_M_run() /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 #5 0x7fa77805e252 in Indirect leak of 16 byte(s) in 1 object(s) allocated from: #0 0x55e00f5a4f1d in operator new(unsigned long) (/build/cpp/debug/arrow-dataset-dataset-writer-test+0x398f1d) (BuildId: 176377c2697ecb6a7274ce334f7231a363068218) #1 0x55e00f6689a0 in __gnu_cxx::new_allocator >::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:127:27 #2 0x55e00f66882c in std::allocator_traits > >::allocate(std::allocator >&, unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:464:20 #3 0x55e00f6684c4 in std::_Vector_base, std::allocator > >::_M_allocate(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:346:20 #4 0x55e00f667a2e in void std::vector, std::allocator > >::_M_range_initialize const*>(std::shared_ptr const*, std::shared_ptr const*, std::forward_iterator_tag) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1582:14 #5 0x55e00f5b9097 in std::vector, std::allocator > >::vector(std::initializer_list >, std::allocator > const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:629:2 #6 0x55e00f7677ab in arrow::dataset::internal::DatasetWriterTestFixture::SetUp() /arrow/cpp/src/arrow/dataset/dataset_writer_test.cc:73:22 #7 0x7fa7994c98ee in void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) (/usr/local/lib/libarrow_testing.so.1500+0xd608ee) (BuildId: 4068fe9148ba0adf20466726aadbd4ac18ef4795) SUMMARY: AddressSanitizer: 3603 byte(s) leaked in 43 allocation(s). Test had memory leaks. Editing XML sed: can't read /build/cpp/build/test-logs/arrow-dataset-dataset-writer-test.xml: No such file or directory mv: cannot stat '/build/cpp/build/test-logs/arrow-dataset-dataset-writer-test.xml.bak': No such file or directory /build/cpp/src/arrow/dataset ```

Component(s)

C++, Continuous Integration

[mapleFU]

I believe dataset-writer has many bugs orz... I'll try to take a look

[mapleFU]

Could you re-produce this fail? I'm compiling this on My MacOS but I'm not able to reproduce it...

But I guess the cause is from DatasetWriterTestFixture.ErrOnExistingData didn't graceful shutdown, I'll dive into it later.

[kou]

I could reproduce this by UBUNTU=22.04 archery docker run ubuntu-cpp-sanitizer. Could you try it?

[mapleFU]

Do you using debug build or release build? I use RelWithDebInfo (-O2) with asan/ubsan, it doesn't trigger the leak

[kou]

UBUNTU=22.04 archery docker run ubuntu-cpp-sanitizer doesn't change the default. (It just changes the Ubuntu version.) So build type is debug like the CI job: https://github.com/apache/arrow/actions/runs/6919998294/job/18823986112#step:6:486

-- Build Type: DEBUG

If you can't reproduce this on your local machine, I'll fix this later.