search

pathwaycom / pathway

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
https://pathway.com
Other
2.84k stars 98 forks source link

[Bug]: csv connector should retry, not fail #60

Open izulin opened 2 weeks ago

izulin commented 2 weeks ago

Steps to reproduce

csv connector under faulty network fails, instead of retrying

Relevant log output

E0614 11:16:48.882455 140007979890240 mod.rs:311] There had been an error processing the row read result: CSV error: record 550622 (line: 550623, byte: 76923319): found record with 1 fields, but the previous record has 9 fields
E0614 11:16:48.925295 140007979890240 mod.rs:311] There had been an error processing the row read result: failed to perform S3 operation GetObject reason: Io(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" })
thread 'pathway:send-3' panicked at external/timely-dataflow/communication/src/allocator/zero_copy/bytes_exchange.rs:96:48:
MergeQueue poisoned.

What did you expect to happen?

retry reading data

Version

0.13.0

Docker Versions (if used)

No response

OS

Linux

On which CPU architecture did you run Pathway?

None