elastic / stream2es

Stream data into ES (Wikipedia, Twitter, stdin, or other ESes)
355 stars 60 forks source link

stdin should block and wait for new inputs till stdin is closed. #57

Closed ikedam closed 8 years ago

ikedam commented 8 years ago

StdinStreamRunner considers it reaches EOF when the stdin is not ready. This results stream2es finishes before the previous program finishes the stream, especially when the program is slow.

This change blocks the runner while reading stdin, and considers reaching EOF only after stdin is closed.

Steps to reproduce problems:

  1. Create a script testoutput.sh:

    #!/bin/sh
    
    for i in $(seq 10); do
       TIMESTAMP=`date +%Y-%m-%dT%H:%M:%SZ`
       printf '{"@timestamp": "%s", "duration": 10}\n' "${TIMESTAMP}"
       sleep 1 # demonstrating a really slow program.
    done
  2. run stream2es: $ ./testoutput.sh |./stream2es stdin

Result (actual, unexpected): stream2es finishes before processing all the outputs from the script like this (only 3 lines are processed):

2015-12-08T23:10:54.503+0900 INFO  00:00.172 17.4d/s 1.5K/s (0.0mb) indexed 3 streamed 3 errors 0
2015-12-08T23:10:54.503+0900 INFO  done

Expected result (result after this change): all 10 lines are processed:

2015-12-08T23:12:11.330+0900 INFO  00:08.099 1.2d/s 0.1K/s (0.0mb) indexed 10 streamed 10 errors 0
2015-12-08T23:12:11.345+0900 INFO  done