Open thisisnic opened 1 year ago
I tried again, and it read in fine without head()
. When I tried again with head()
then collect()
it read in the data successfully and then segfaulted immediately after.
> open_dataset("/data/nyc-taxi/year=2016/month=11/part-0.parquet") %>% head() %>% collect()
# A tibble: 6 × 22
vendor_name pickup_datetime dropoff_datetime passenger_count
<chr> <dttm> <dttm> <int>
1 VTS 2016-11-10 20:14:06 2016-11-10 20:19:37 1
2 VTS 2016-11-10 20:14:06 2016-11-10 20:43:31 1
3 VTS 2016-11-10 20:14:06 2016-11-10 20:17:24 1
4 VTS 2016-11-10 20:14:06 2016-11-10 20:20:12 1
5 CMT 2016-11-10 20:14:07 2016-11-10 20:20:23 1
6 CMT 2016-11-10 20:14:07 2016-11-10 21:13:19 2
# ℹ 18 more variables: trip_distance <dbl>, pickup_longitude <dbl>,
# pickup_latitude <dbl>, rate_code <chr>, store_and_fwd <chr>,
# dropoff_longitude <dbl>, dropoff_latitude <dbl>, payment_type <chr>,
# fare_amount <dbl>, extra <dbl>, mta_tax <dbl>, tip_amount <dbl>,
# tolls_amount <dbl>, total_amount <dbl>, improvement_surcharge <dbl>,
# congestion_surcharge <dbl>, pickup_location_id <int>,
# dropoff_location_id <int>
>
Thread 11 "R" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffbffff640 (LWP 480578)]
0x0000000000000000 in ?? ()
I've now tested this with 13.0.0 and get the same result.
Would you mind provide the backtrace or the reproduce way in C++ or Python? Since I'm not so familiar with R
I'm afraid I have no idea how to do it in C++ or Python. The backtrace is here though:
Thread 8 "R" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffddbfe640 (LWP 490914)]
0x00007fffefafb182 in arrow::internal::Executor::Submit<parquet::arrow::(anonymous namespace)::FileReaderImpl::DecodeRowGroups(std::shared_ptr<parquet::arrow::(anonymous namespace)::FileReaderImpl>, const std::vector<int>&, const std::vector<int>&, arrow::internal::Executor*)::<lambda(size_t, std::shared_ptr<parquet::arrow::ColumnReaderImpl>)>&, long unsigned int&, std::shared_ptr<parquet::arrow::ColumnReaderImpl> >(arrow::internal::TaskHints, arrow::StopToken, struct {...} &) (this=0x2db800000008, hints=..., stop_token=..., func=...) at /home/nic2/arrow/cpp/src/arrow/util/thread_pool.h:159
159 ARROW_RETURN_NOT_OK(SpawnReal(hints, std::move(task), std::move(stop_token),
(gdb) bt
#0 0x00007fffefafb182 in arrow::internal::Executor::Submit<parquet::arrow::(anonymous namespace)::FileReaderImpl::DecodeRowGroups(std::shared_ptr<parquet::arrow::(anonymous namespace)::FileReaderImpl>, const std::vector<int>&, const std::vector<int>&, arrow::internal::Executor*)::<lambda(size_t, std::shared_ptr<parquet::arrow::ColumnReaderImpl>)>&, long unsigned int&, std::shared_ptr<parquet::arrow::ColumnReaderImpl> >(arrow::internal::TaskHints, arrow::StopToken, struct {...} &) (this=0x2db800000008, hints=..., stop_token=..., func=...) at /home/nic2/arrow/cpp/src/arrow/util/thread_pool.h:159
#1 0x00007fffefaf9ec9 in arrow::internal::Executor::Submit<parquet::arrow::(anonymous namespace)::FileReaderImpl::DecodeRowGroups(std::shared_ptr<parquet::arrow::(anonymous namespace)::FileReaderImpl>, const std::vector<int>&, const std::vector<int>&, arrow::internal::Executor*)::<lambda(size_t, std::shared_ptr<parquet::arrow::ColumnReaderImpl>)>&, long unsigned int&, std::shared_ptr<parquet::arrow::ColumnReaderImpl> >(struct {...} &) (
this=0x2db800000008, func=...) at /home/nic2/arrow/cpp/src/arrow/util/thread_pool.h:186
#2 0x00007fffefaf8b9b in arrow::internal::ParallelForAsync<parquet::arrow::(anonymous namespace)::FileReaderImpl::DecodeRowGroups(std::shared_ptr<parquet::arrow::(anonymous namespace)::FileReaderImpl>, const std::vector<int>&, const std::vector<int>&, arrow::internal::Executor*)::<lambda(size_t, std::shared_ptr<parquet::arrow::ColumnReaderImpl>)>&, std::shared_ptr<parquet::arrow::ColumnReaderImpl> >(std::vector<std::shared_ptr<parquet::arrow::ColumnReaderImpl>, std::allocator<std::shared_ptr<parquet::arrow::ColumnReaderImpl> > >, struct {...} &, arrow::internal::Executor *) (inputs=std::vector of length 22, capacity 22 = {...}, func=..., executor=0x2db800000008)
at /home/nic2/arrow/cpp/src/arrow/util/parallel.h:56
#3 0x00007fffefaf6c82 in arrow::internal::OptionalParallelForAsync<parquet::arrow::(anonymous namespace)::FileReaderImpl::DecodeRowGroups(std::shared_ptr<parquet::arrow::(anonymous namespace)::FileReaderImpl>, const std::vector<int>&, const std::vector<int>&, arrow::internal::Executor*)::<lambda(size_t, std::shared_ptr<parquet::arrow::ColumnReaderImpl>)>&, std::shared_ptr<parquet::arrow::ColumnReaderImpl> >(bool, std::vector<std::shared_ptr<parquet::arrow::ColumnReaderImpl>, std::allocator<std::shared_ptr<parquet::arrow::ColumnReaderImpl> > >, struct {...} &, arrow::internal::Executor *) (use_threads=true, inputs=std::vector of length 0, capacity 0, func=...,
executor=0x2db800000008) at /home/nic2/arrow/cpp/src/arrow/util/parallel.h:91
#4 0x00007fffefaf35fd in parquet::arrow::(anonymous namespace)::FileReaderImpl::DecodeRowGroups (this=0x7fffd4cf3330,
self=std::shared_ptr<parquet::arrow::(anonymous namespace)::FileReaderImpl> (use count 48, weak count 0) = {...}, row_groups=std::vector of length 1, capacity 1 = {...},
column_indices=std::vector of length 22, capacity 22 = {...}, cpu_executor=0x2db800000008) at /home/nic2/arrow/cpp/src/parquet/arrow/reader.cc:1280
#5 0x00007fffefaf1fbd in parquet::arrow::RowGroupGenerator::ReadOneRowGroup (cpu_executor=0x2db800000008, self=std::shared_ptr<parquet::arrow::(anonymous namespace)::FileReaderImpl> (use count 48, weak count 0) = {...},
row_group=354, column_indices=std::vector of length 22, capacity 22 = {...}) at /home/nic2/arrow/cpp/src/parquet/arrow/reader.cc:1166
#6 0x00007fffefb06978 in parquet::arrow::RowGroupGenerator::FetchNext()::{lambda()#1}::operator()() const (__closure=0x7fffc4153b18) at /home/nic2/arrow/cpp/src/parquet/arrow/reader.cc:1139
#7 0x00007fffefb452e6 in arrow::detail::ContinueFuture::operator()<parquet::arrow::RowGroupGenerator::FetchNext()::{lambda()#1}, , arrow::Future<std::function<arrow::Future<std::shared_ptr<arrow::RecordBatch> > ()> >, arrow::Future<std::function<arrow::Future<std::shared_ptr<arrow::RecordBatch> > ()> > >(arrow::Future<std::function<arrow::Future<std::shared_ptr<arrow::RecordBatch> > ()> >, parquet::arrow::RowGroupGenerator::FetchNext()::{lambda()#1}&&) const (this=0x7fffddbfd71f, next=..., f=...) at /home/nic2/arrow/cpp/src/arrow/util/future.h:178
#8 0x00007fffefb44443 in arrow::detail::ContinueFuture::IgnoringArgsIf<parquet::arrow::RowGroupGenerator::FetchNext()::{lambda()#1}, arrow::Future<std::function<arrow::Future<std::shared_ptr<arrow::RecordBatch> > ()> >, arrow::internal::Empty const&>(std::integral_constant<bool, true>, arrow::Future<std::function<arrow::Future<std::shared_ptr<arrow::RecordBatch> > ()> >&&, parquet::arrow::RowGroupGenerator::FetchNext()::{lambda()#1}&&, arrow::internal::Empty const&) const (this=0x7fffddbfd71f, next=..., f=...) at /home/nic2/arrow/cpp/src/arrow/util/future.h:188
#9 0x00007fffefb4326a in arrow::Future<arrow::internal::Empty>::ThenOnComplete<parquet::arrow::RowGroupGenerator::FetchNext()::{lambda()#1}, arrow::Future<arrow::internal::Empty>::PassthruOnFailure<parquet::arrow::RowGroupGenerator::FetchNext()::{lambda()#1}> >::operator()(arrow::Result<arrow::internal::Empty> const&) && (this=0x7fffc4153b18, result=...) at /home/nic2/arrow/cpp/src/arrow/util/future.h:545
#10 0x00007fffefb41b0b in arrow::Future<arrow::internal::Empty>::WrapResultyOnComplete::Callback<arrow::Future<arrow::internal::Empty>::ThenOnComplete<parquet::arrow::RowGroupGenerator::FetchNext()::{lambda()#1}, arrow::Future<arrow::internal::Empty>::PassthruOnFailure<parquet::arrow::RowGroupGenerator::FetchNext()::{lambda()#1}> > >::operator()(arrow::FutureImpl const&) && (this=0x7fffc4153b18, impl=...)
at /home/nic2/arrow/cpp/src/arrow/util/future.h:442
#11 0x00007fffefb413d1 in arrow::internal::FnOnce<void (arrow::FutureImpl const&)>::FnImpl<arrow::Future<arrow::internal::Empty>::WrapResultyOnComplete::Callback<arrow::Future<arrow::internal::Empty>::ThenOnComplete<parquet::arrow::RowGroupGenerator::FetchNext()::{lambda()#1}, arrow::Future<arrow::internal::Empty>::PassthruOnFailure<parquet::arrow::RowGroupGenerator::FetchNext()::{lambda()#1}> > > >::invoke(arrow::FutureImpl const&) (
this=0x7fffc4153b10, a#0=...) at /home/nic2/arrow/cpp/src/arrow/util/functional.h:152
#12 0x00007fffec14af78 in arrow::internal::FnOnce<void (arrow::FutureImpl const&)>::operator()(arrow::FutureImpl const&) && (this=0x7fffc4151f40, a#0=...) at /home/nic2/arrow/cpp/src/arrow/util/functional.h:140
#13 0x00007fffec14a5bd in arrow::ConcreteFutureImpl::RunOrScheduleCallback (self=std::shared_ptr<arrow::FutureImpl> (use count 2, weak count 1) = {...}, callback_record=..., in_add_callback=false)
at /home/nic2/arrow/cpp/src/arrow/util/future.cc:109
#14 0x00007fffec14a916 in arrow::ConcreteFutureImpl::DoMarkFinishedOrFailed (this=0x7fffd4e4d4b0, state=arrow::FutureState::SUCCESS) at /home/nic2/arrow/cpp/src/arrow/util/future.cc:147
#15 0x00007fffec149d39 in arrow::ConcreteFutureImpl::DoMarkFinished (this=0x7fffd4e4d4b0) at /home/nic2/arrow/cpp/src/arrow/util/future.cc:38
#16 0x00007fffec147dd0 in arrow::FutureImpl::MarkFinished (this=0x7fffd4e4d4b0) at /home/nic2/arrow/cpp/src/arrow/util/future.cc:193
#17 0x00007ffff102c506 in arrow::Future<arrow::internal::Empty>::DoMarkFinished (this=0x7fffc4141618, res=...) at /home/nic2/arrow/cpp/src/arrow/util/future.h:658
#18 0x00007ffff101a09c in arrow::Future<arrow::internal::Empty>::MarkFinished<arrow::internal::Empty, void> (this=0x7fffc4141618, s=...) at /home/nic2/arrow/cpp/src/arrow/util/future.h:409
#19 0x00007fffefb15d68 in arrow::internal::Executor::DoTransfer<arrow::internal::Empty, arrow::Future<arrow::internal::Empty>, arrow::Status>(arrow::Future<arrow::internal::Empty>, bool)::{lambda(arrow::Status const&)#1}::operator()(arrow::Status const&) (__closure=0x7fffc4141618, result=...) at /home/nic2/arrow/cpp/src/arrow/util/thread_pool.h:231
#20 0x00007fffefb419dd in arrow::Future<arrow::internal::Empty>::WrapStatusyOnComplete::Callback<arrow::internal::Executor::DoTransfer<arrow::internal::Empty, arrow::Future<arrow::internal::Empty>, arrow::Status>(arrow::Future<arrow::internal::Empty>, bool)::{lambda(arrow::Status const&)#1}>::operator()(arrow::FutureImpl const&) && (this=0x7fffc4141618, impl=...) at /home/nic2/arrow/cpp/src/arrow/util/future.h:455
#21 0x00007fffefb41195 in arrow::internal::FnOnce<void (arrow::FutureImpl const&)>::FnImpl<arrow::Future<arrow::internal::Empty>::WrapStatusyOnComplete::Callback<arrow::internal::Executor::DoTransfer<arrow::internal::Empty, arrow::Future<arrow::internal::Empty>, arrow::Status>(arrow::Future<arrow::internal::Empty>, bool)::{lambda(arrow::Status const&)#1}> >::invoke(arrow::FutureImpl const&) (this=0x7fffc4141610, a#0=...)
at /home/nic2/arrow/cpp/src/arrow/util/functional.h:152
#22 0x00007fffec14af78 in arrow::internal::FnOnce<void (arrow::FutureImpl const&)>::operator()(arrow::FutureImpl const&) && (this=0x7fffc4158048, a#0=...) at /home/nic2/arrow/cpp/src/arrow/util/functional.h:140
#23 0x00007fffec14a335 in arrow::ConcreteFutureImpl::RunOrScheduleCallback(std::shared_ptr<arrow::FutureImpl> const&, arrow::FutureImpl::CallbackRecord&&, bool)::{lambda()#1}::operator()() (__closure=0x7fffc4158038)
at /home/nic2/arrow/cpp/src/arrow/util/future.cc:105
#24 0x00007fffec15065e in arrow::internal::FnOnce<void ()>::FnImpl<arrow::ConcreteFutureImpl::RunOrScheduleCallback(std::shared_ptr<arrow::FutureImpl> const&, arrow::FutureImpl::CallbackRecord&&, bool)::{lambda()#1}>::invoke() (this=0x7fffc4158030) at /home/nic2/arrow/cpp/src/arrow/util/functional.h:152
The input file is at this URL: "https://voltrondata-labs-datasets.s3.us-east-2.amazonaws.com/nyc-taxi/year=2016/month=11/part-0.parquet"
I tried downloading it again to ensure it wasn't a bad download, but got the same error.
I suspect this is related to https://github.com/apache/arrow/issues/31486 .
I suspect this is related to #31486 .
Thanks @westonpace! Would you mind expanding a bit? I read that issue but didn't really understand it - any idea what the likely source of the issue is?
Describe the bug, including details regarding any error messages, version, and platform.
I'm using the Arrow R package version 12.0.1.1 and am getting segfault when trying to read a Parquet file. Here's the output with the debugger attached:
If I read in the file via
read_parquet()
, I don't have a problem and it loads fine. Happy to supply the file if necessary, though wasn't sure it's possible/desirable to attach a 150Mb file to an issue ticket.Component(s)
C++