Can handle larger files by buffering and flushing to S3.
Memory stays low and can handle large files by processing longer.
Timeouts are still an issue, a more optimal approach is to pre split work (even virtually with byte offsets) and have predictable data size. That's a more involved change that will likely not be worked on right now, but this code is still useful.
Can handle larger files by buffering and flushing to S3.
Memory stays low and can handle large files by processing longer.
Timeouts are still an issue, a more optimal approach is to pre split work (even virtually with byte offsets) and have predictable data size. That's a more involved change that will likely not be worked on right now, but this code is still useful.
Related: #155