I discovered an issue where sandbox decoders continue to process messages/packs after the sandbox should have been terminated due to an error. For example, if a pack fails because the injected message was too large, several more packs, regardless of size, also fail with an “output too large” error until Heka exits. As far as I can tell this issue doesn't happen if you catch all your errors with pcall.
Here is an example of how I reproduced the issue with dev as of this afternoon (08b95b0). It first sets the output_limit on the sandbox to 1024 bytes and then processes a file via a logstream input. The log file has a bunch of 11 byte lines with a 2049 byte line in the middle of it. Every line before the big line is processed correctly. The big line and about 30 lines afterwards all fail with an "output_limit exceeded" error despite those lines being well under the output_limit size.
It should also be noted that some of the lines after the big line are "correctly" injected into the router due to send_decode_failures being set to true. In my example there were 4 messages that were injected into the router. The rest get dropped. In other test runs sometimes it would be 1-2 messages and sometimes none would get injected.
I originally noticed the issue in a Linux Docker container. I then reproduced it on my Macbook. I'm using Go 1.5. Let me know if you need anymore details.
I discovered an issue where sandbox decoders continue to process messages/packs after the sandbox should have been terminated due to an error. For example, if a pack fails because the injected message was too large, several more packs, regardless of size, also fail with an “output too large” error until Heka exits. As far as I can tell this issue doesn't happen if you catch all your errors with
pcall.Here is an example of how I reproduced the issue with dev as of this afternoon (08b95b0). It first sets the
output_limiton the sandbox to 1024 bytes and then processes a file via a logstream input. The log file has a bunch of 11 byte lines with a 2049 byte line in the middle of it. Every line before the big line is processed correctly. The big line and about 30 lines afterwards all fail with an "output_limit exceeded" error despite those lines being well under theoutput_limitsize.https://gist.github.com/williamsandrew/437c490ba89c01261674
It should also be noted that some of the lines after the big line are "correctly" injected into the router due to
send_decode_failuresbeing set to true. In my example there were 4 messages that were injected into the router. The rest get dropped. In other test runs sometimes it would be 1-2 messages and sometimes none would get injected.I originally noticed the issue in a Linux Docker container. I then reproduced it on my Macbook. I'm using Go 1.5. Let me know if you need anymore details.