Closed jennifgcrl closed 1 month ago
Hey @jennifgcrl, could you share a full traceback? Also, what does your cluster look like (type and number of nodes), and how did you choose between read_parquet
and read_parquet_bulk
?
Hi there,
I am seeing the same issue and it is a major blocker.
I am using read_parquet
for 512 or more parquet chunks at a time and read_binary_files for millions of files. In each case all files are in a common root prefix on S3. I am using about 1000 CPUs, but sometimes more. More CPUs seems to cause the error more often.
Sometimes its error getting information for key, sometimes it's libcurl was given bad argument, sometimes it's timeout like above, however, it's always caused by transient network errors that need to be retried.
I see some fixes for similar problems have been applied in the past, but there are still many gaps.
Some previous related works:
The closest fix is this one: https://github.com/ray-project/ray/pull/42027 while it says it fixed for reads, inspecting the codebase in version 2.9.3 it looks like it's only actually fixed for writes.
The error I am seeing is AWS Error NETWORK_CONNECTION
which is transient and must be retried. It is on the actual read tasks, as well as metadata tasks. Some gaps are here:
"AWS Error NETWORK_CONNECTION"
A bit of custom hackery into the codebase has improved the reliability greatly for me, although my solutions are too messy to PR. These code references should hopefully be helpful enough to indicate gaps in the current retry strategies though.
Here's an example stacktrace with some stuff ***'d out:
Read progress 0: 100%|█████████▉| 3711/3712 [00:39<00:00, 49.63it/s]Traceback (most recent call last):
File "/tmp/ray/session_2024-03-05_11-09-55_961367_273/runtime_resources/working_dir_files/_ray_pkg_ddef4fad01df9951/src/my_file.py", line 277, in <module>
***
File "/tmp/ray/session_2024-03-05_11-09-55_961367_273/runtime_resources/working_dir_files/_ray_pkg_ddef4fad01df9951/src/my_file.py", line 273, in ***
File "/tmp/ray/session_2024-03-05_11-09-55_961367_273/runtime_resources/working_dir_files/_ray_pkg_ddef4fad01df9951/src/my_file.py", line 152, in ***
print('size of prev_batches:', prev_batches.count())
File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/data/dataset.py", line 2606, in count
[get_num_rows.remote(block) for block in self.get_internal_block_refs()]
File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/data/dataset.py", line 4779, in get_internal_block_refs
blocks = self._plan.execute().get_blocks()
File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/data/_internal/lazy_block_list.py", line 293, in get_blocks
blocks, _ = self._get_blocks_with_metadata()
File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/data/_internal/lazy_block_list.py", line 327, in _get_blocks_with_metadata
meta = ray.get(refs_list.pop(-1))
File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
return fn(*args, **kwargs)
File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/_private/worker.py", line 2624, in get
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(OSError): [36mray::_execute_read_task_split()[39m (pid=25144, ip=10.83.226.216)
File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/data/_internal/lazy_block_list.py", line 637, in _execute_read_task_split
for block in blocks:
File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/data/datasource/datasource.py", line 237, in __call__
yield from result
File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/data/_internal/execution/operators/map_transformer.py", line 430, in __call__
for block in blocks:
File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/data/_internal/execution/operators/map_transformer.py", line 371, in __call__
for data in iter:
File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/data/datasource/parquet_datasource.py", line 508, in _read_fragments
for batch in batches:
File "pyarrow/_dataset.pyx", line 3414, in _iterator
File "pyarrow/_dataset.pyx", line 3032, in pyarrow._dataset.TaggedRecordBatchIterator.__next__
File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: When reading information for key '***_000000.parquet' in bucket 'MY_BUCKET': AWS Error NETWORK_CONNECTION during HeadObject operation: curlCode: 28, Timeout was reached
@bveeramani May I know when we can expect a fix on this, Im facing the same issue when im trying to read around 42k parquet files from s3 directory
We are seeing similar issues when reading ~1M images from a S3 bucket with read_images
, which causes e2e training to end abruptly.
- ReadImage->Map(wnid_to_index): 4 active, 360 queued, [cpu: 4.0, objects: 673.3MB]: 67%|██████▋ | 1463/2188 [50:34<23:33, 1.95s/it]
- ReadImage->Map(wnid_to_index): 4 active, 360 queued, [cpu: 4.0, objects: 673.3MB]: 100%|██████████| 1463/1463 [50:34<00:00, 1.95s/it]
Traceback (most recent call last):
ray.exceptions.RayTaskError(OSError): [36mray::ReadImage->Map(wnid_to_index)()[39m (pid=257328, ip=10.0.34.58)
for b_out in map_transformer.apply_transform(iter(blocks), ctx):
File "/home/ubuntu/miniconda3/envs/ray-gpu/lib/python3.10/site-packages/ray/data/_internal/execution/operators/map_transformer.py", line 392, in __call__
for data in iter:
File "/home/ubuntu/miniconda3/envs/ray-gpu/lib/python3.10/site-packages/ray/data/_internal/execution/operators/map_transformer.py", line 134, in _udf_timed_iter
output = next(input)
File "/home/ubuntu/miniconda3/envs/ray-gpu/lib/python3.10/site-packages/ray/data/_internal/execution/operators/map_transformer.py", line 216, in __call__
yield from self._row_fn(input, ctx)
File "/home/ubuntu/miniconda3/envs/ray-gpu/lib/python3.10/site-packages/ray/data/_internal/planner/plan_udf_map_op.py", line 239, in transform_fn
for row in rows:
File "/home/ubuntu/miniconda3/envs/ray-gpu/lib/python3.10/site-packages/ray/data/_internal/execution/operators/map_transformer.py", line 269, in __call__
for block in blocks:
File "/home/ubuntu/miniconda3/envs/ray-gpu/lib/python3.10/site-packages/ray/data/_internal/execution/operators/map_transformer.py", line 253, in __call__
yield from self._block_fn(input, ctx)
File "/home/ubuntu/miniconda3/envs/ray-gpu/lib/python3.10/site-packages/ray/data/_internal/planner/plan_read_op.py", line 91, in do_read
yield from call_with_retry(
File "/home/ubuntu/miniconda3/envs/ray-gpu/lib/python3.10/site-packages/ray/data/datasource/datasource.py", line 164, in __call__
yield from result
File "/home/ubuntu/miniconda3/envs/ray-gpu/lib/python3.10/site-packages/ray/data/datasource/file_based_datasource.py", line 258, in read_task_fn
yield from make_async_gen(
File "/home/ubuntu/miniconda3/envs/ray-gpu/lib/python3.10/site-packages/ray/data/_internal/util.py", line 941, in make_async_gen
raise next_item
File "/home/ubuntu/miniconda3/envs/ray-gpu/lib/python3.10/site-packages/ray/data/_internal/util.py", line 918, in execute_computation
for item in fn(thread_safe_generator):
File "/home/ubuntu/miniconda3/envs/ray-gpu/lib/python3.10/site-packages/ray/data/datasource/file_based_datasource.py", line 232, in read_files
with _open_file_with_retry(
File "/home/ubuntu/miniconda3/envs/ray-gpu/lib/python3.10/site-packages/ray/data/datasource/file_based_datasource.py", line 522, in _open_file_with_retry
return call_with_retry(
File "/home/ubuntu/miniconda3/envs/ray-gpu/lib/python3.10/site-packages/ray/data/_internal/util.py", line 995, in call_with_retry
raise e from None
File "/home/ubuntu/miniconda3/envs/ray-gpu/lib/python3.10/site-packages/ray/data/_internal/util.py", line 982, in call_with_retry
return f()
File "/home/ubuntu/miniconda3/envs/ray-gpu/lib/python3.10/site-packages/ray/data/datasource/file_based_datasource.py", line 234, in <lambda>
lambda: open_input_source(fs, read_path, **open_stream_args),
File "/home/ubuntu/miniconda3/envs/ray-gpu/lib/python3.10/site-packages/ray/data/datasource/file_based_datasource.py", line 344, in _open_input_source
file = filesystem.open_input_stream(path, buffer_size=buffer_size, **open_args)
File "pyarrow/_fs.pyx", line 822, in pyarrow._fs.FileSystem.open_input_stream
File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
OSError: When reading information for key 'IMAGE_KEY' in bucket 'MY_BUCKET': AWS Error NETWORK_CONNECTION during HeadObject operation: curlCode: 28, Timeout was reached
Is there any update on fixing this? Also running into this and it is of high severity for me as well
@murthyn - this is high priority for us but we're a little swamped; it's funded as part of our planning through May. Balaji will provide more details as he has them.
re-reading this - @meltzerpete do you think you'd be game to contribute a PR; we can pair with someone to help shepherd through on the Anyscale side.
Your breakdown of the problem and whereabouts to solve is quite spectacular :) cc @c21
hi @jennifgcrl , looks like this is a transient error. For ray 2.10+, we now support automatic retry of these kind of errors. Could you try again with the latest Ray?
tag @murthyn @ronyw7 @Sri-nidhi @meltzerpete as well see @raulchen above^
tag @murthyn @ronyw7 @Sri-nidhi @meltzerpete as well see @raulchen above^
Thanks for the updates. Hopefully this has been solved. I'm unavailable until Tuesday, but will test this then. Failing that happy to try and support with PR if there's someone to pair with.
@raulchen @anyscalesam
I'm pretty sure I tried ray 2.10 when it was first released and found the issue persisted, however, I have tested today with ray 20.0.0 and can confirm the issue appears to be solved.
I am seeing no transient errors at all, thanks so much!
Ah in fact perhaps I spoke too soon 😬
I am seeing this:
failed to write s3://*****: ray::ReadParquet() (pid=1061231, ip=10.83.226.188)
for b_out in map_transformer.apply_transform(iter(blocks), ctx):
File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/data/_internal/execution/operators/map_transformer.py", line 392, in __call__
for data in iter:
File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/data/_internal/execution/operators/map_transformer.py", line 253, in __call__
yield from self._block_fn(input, ctx)
File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/data/_internal/planner/plan_read_op.py", line 91, in do_read
yield from call_with_retry(
File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/data/datasource/datasource.py", line 163, in __call__
yield from result
File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/data/datasource/parquet_datasource.py", line 549, in _read_fragments
for batch in batches:
File "pyarrow/_dataset.pyx", line 3769, in _iterator
File "pyarrow/_dataset.pyx", line 3387, in pyarrow._dataset.TaggedRecordBatchIterator.__next__
File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
OSError: When reading information for key '*****' in bucket '*****': AWS Error NETWORK_CONNECTION during HeadObject operation: curlCode: 28, Timeout was reached
I'm seeing this several times, but this looks like the only place it's erroring. Before there were many.
Is there any update ? I'm facing the same issue...
Hi Ray Community - TPM @ Anyscale here. We had a longgg discussion about this today and are currently thinking of the following to help mitigate this broad problem:
Next steps is we want to chat with y'all to make sure we get the requirements right for fixing this; can folks reach out to me on Ray Slack (if you're not on you can join on ray.io)
My username is "Sam (Ray Team)"
Still seems to happening on latest with a webdataset datasource:
Traceback (most recent call last):
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/air/execution/_internal/event_manager.py", line 110, in resolve_future
result = ray.get(future)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
return fn(*args, **kwargs)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/_private/worker.py", line 2659, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/_private/worker.py", line 871, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(OSError): [36mray::_Inner.train()[39m (pid=444169, ip=10.233.84.84, actor_id=dbde2580c35ca6be6e1eceb904000000, repr=TorchTrainer)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/tune/trainable/trainable.py", line 331, in train
raise skipped from exception_cause(skipped)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/train/_internal/utils.py", line 53, in check_for_failure
ray.get(object_ref)
ray.exceptions.RayTaskError(OSError): [36mray::_RayTrainWorker__execute.get_next()[39m (pid=445843, ip=10.233.84.84, actor_id=4b94fe3916cc6aa7126f7bba04000000, repr=<ray.train._internal.worker_group.RayTrainWorker object at 0x7f5acc472950>)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/train/_internal/worker_group.py", line 33, in __execute
raise skipped from exception_cause(skipped)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/train/_internal/utils.py", line 169, in discard_return_wrapper
train_func(*args, **kwargs)
File "/tmp/ray/session_2024-07-31_04-25-04_745822_8880/runtime_resources/working_dir_files/_ray_pkg_6a7ed0f237d82b05/examples/brushnet/train_brushnet_sdxl_ray.py", line 297, in train_func_per_worker
trainer.fit(model, train_dataloaders=train_dataloader, val_dataloaders=val_dataloader)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 543, in fit
call._call_and_handle_interrupt(
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 43, in _call_and_handle_interrupt
return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/lightning/pytorch/strategies/launchers/subprocess_script.py", line 105, in launch
return function(*args, **kwargs)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 579, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 986, in _run
results = self._run_stage()
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1030, in _run_stage
self.fit_loop.run()
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/lightning/pytorch/loops/fit_loop.py", line 205, in run
self.advance()
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/lightning/pytorch/loops/fit_loop.py", line 363, in advance
self.epoch_loop.run(self._data_fetcher)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/lightning/pytorch/loops/training_epoch_loop.py", line 140, in run
self.advance(data_fetcher)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/lightning/pytorch/loops/training_epoch_loop.py", line 212, in advance
batch, _, __ = next(data_fetcher)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 127, in __next__
self.batches.append(super().__next__())
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 60, in __next__
batch = next(self.iterator)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 341, in __next__
out = next(self._iterator)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 78, in __next__
out[i] = next(self.iterators[i])
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/iterator.py", line 178, in _create_iterator
for batch in iterator:
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/block_batching/iter_batches.py", line 178, in iter_batches
next_batch = next(async_batch_iter)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/util.py", line 932, in make_async_gen
raise next_item
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/util.py", line 909, in execute_computation
for item in fn(thread_safe_generator):
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/block_batching/iter_batches.py", line 167, in _async_iter_batches
yield from extract_data_from_batch(batch_iter)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/block_batching/util.py", line 211, in extract_data_from_batch
for batch in batch_iter:
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/block_batching/iter_batches.py", line 313, in restore_original_order
for batch in batch_iter:
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/block_batching/util.py", line 204, in finalize_batches
for batch in batch_iter:
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/util.py", line 932, in make_async_gen
raise next_item
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/util.py", line 909, in execute_computation
for item in fn(thread_safe_generator):
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/block_batching/iter_batches.py", line 220, in threadpool_computations_format_collate
yield from formatted_batch_iter
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/block_batching/util.py", line 180, in collate
for batch in batch_iter:
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/block_batching/util.py", line 159, in format_batches
for batch in block_iter:
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/util.py", line 889, in __next__
return next(self.it)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/block_batching/util.py", line 118, in blocks_to_batches
for block in block_iter:
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/block_batching/util.py", line 55, in resolve_block_refs
for block_ref in block_ref_iter:
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/block_batching/iter_batches.py", line 288, in prefetch_batches_locally
next_ref_bundle = next(ref_bundles)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/util.py", line 889, in __next__
return next(self.it)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/iterator/stream_split_iterator.py", line 84, in gen_blocks
] = ray.get(future)
ray.exceptions.RayTaskError(OSError): [36mray::SplitCoordinator.get()[39m (pid=446810, ip=10.233.84.84, actor_id=77cf38306952dea44e4bae7004000000, repr=<ray.data._internal.iterator.stream_split_iterator.SplitCoordinator object at 0x7fbc25dbe5c0>)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/iterator/stream_split_iterator.py", line 224, in get
next_bundle = self._output_iterator.get_next(output_split_idx)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/execution/legacy_compat.py", line 76, in get_next
bundle = self._base_iterator.get_next(output_split_idx)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/execution/streaming_executor.py", line 145, in get_next
item = self._outer._output_node.get_output_blocking(
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/execution/streaming_executor_state.py", line 284, in get_output_blocking
raise self._exception
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/execution/streaming_executor.py", line 145, in get_next
item = self._outer._output_node.get_output_blocking(
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/execution/streaming_executor_state.py", line 284, in get_output_blocking
raise self._exception
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/execution/streaming_executor.py", line 145, in get_next
item = self._outer._output_node.get_output_blocking(
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/execution/streaming_executor_state.py", line 284, in get_output_blocking
raise self._exception
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/execution/streaming_executor.py", line 222, in run
continue_sched = self._scheduling_loop_step(self._topology)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/execution/streaming_executor.py", line 277, in _scheduling_loop_step
num_errored_blocks = process_completed_tasks(
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/execution/streaming_executor_state.py", line 457, in process_completed_tasks
raise e from None
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/execution/streaming_executor_state.py", line 424, in process_completed_tasks
bytes_read = task.on_data_ready(
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/execution/interfaces/physical_operator.py", line 105, in on_data_ready
raise ex from None
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/execution/interfaces/physical_operator.py", line 101, in on_data_ready
ray.get(block_ref)
ray.exceptions.RayTaskError(OSError): [36mray::ReadWebDataset->SplitBlocks(2)()[39m (pid=447163, ip=10.233.84.84)
for b_out in map_transformer.apply_transform(iter(blocks), ctx):
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/execution/operators/map_transformer.py", line 451, in __call__
for block in blocks:
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/execution/operators/map_transformer.py", line 392, in __call__
for data in iter:
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/execution/operators/map_transformer.py", line 253, in __call__
yield from self._block_fn(input, ctx)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/planner/plan_read_op.py", line 101, in do_read
yield from call_with_retry(
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/datasource/datasource.py", line 197, in __call__
yield from result
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/datasource/file_based_datasource.py", line 253, in read_task_fn
yield from read_files(read_paths)
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/datasource/file_based_datasource.py", line 219, in read_files
for block in read_stream(f, read_path):
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/datasource/webdataset_datasource.py", line 352, in _read_stream
for sample in samples:
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/datasource/webdataset_datasource.py", line 148, in _group_by_keys
for filesample in data:
File "/home/ubuntu/Brushnet/.venv/lib/python3.10/site-packages/ray/data/_internal/datasource/webdataset_datasource.py", line 121, in _tar_file_iterator
data = stream.extractfile(tarinfo).read()
File "/usr/lib/python3.10/tarfile.py", line 689, in read
b = self.fileobj.read(length)
File "/usr/lib/python3.10/tarfile.py", line 526, in read
buf = self._read(size)
File "/usr/lib/python3.10/tarfile.py", line 534, in _read
return self.__read(size)
File "/usr/lib/python3.10/tarfile.py", line 564, in __read
buf = self.fileobj.read(self.bufsize)
File "pyarrow/io.pxi", line 422, in pyarrow.lib.NativeFile.read
File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
OSError: AWS Error NETWORK_CONNECTION during GetObject operation: curlCode: 28, Timeout was reached
TY for the PR @BitPhinix; let's follow up there.
What happened + What you expected to happen
ray.data.read_parquet_bulk crashes on parquet_base_datasource's pq.read_table with
This error should be retried
Versions / Dependencies
ray==2.9.2
Reproduction script
Issue Severity
High: It blocks me from completing my task.