This PR is based on the docker python library's ability to stream out the output from a "docker run" line by line.
We take 100 output lines each time, create an arrow table, and translate the table to record batches.
The number 100 is currently hard-coded.
This PR is based on the docker python library's ability to stream out the output from a "docker run" line by line. We take 100 output lines each time, create an arrow table, and translate the table to record batches. The number 100 is currently hard-coded.