input benthos_lab option multipart:false doesn't appear to make a difference

[bkcsfi]

Hi,

I have a text file that I want to process each line individually in a pipeline. I expected input type benthos_lab to work similar to 'file' input, in that by default each individual input line would be treated as a single message.

That doesn't appear to be the case as I had to nest my pipeline in a for_each to get it to work properly.

I tried adding "multipart: false" but the results are the same.

I then ran the same config directly under benthos, after changing the input type to 'file'.

In my test directly on benthos, changing the input file to multipart: true caused it to behave like input benthos_lab does (e.g. incorrectly for my use-case)

That makes me think benthos_lab input defaults to multipart: true, and cannot be changed to multipart: false.

[Jeffail]

Hey @bkcsfi, right now you have two options:

• Input data is read as batches, where each line is a message of a batch, and you can insert blank lines between messages to split the batches
• Input data is read as a single message (containing all the content including line breaks)

It sounds like what you want can either be solved by inserting blank lines between each line (which is lame) or I can add a third option where each line is a single message batch, similar to the file and stdin inputs.

I'll mark this ticket as an enhancement for implementing the third option, but obviously I won't get mad if you just decide to add blank lines.

[bkcsfi]

Hi, thanks for the quick response.

I've watched all your videos and was using the labs version to setup a pipeline that can handle both plain text and json as input similar to your "intro to labs" video. It was confusing because the settings were set to "each line is a message".

Would it be clearer if this option was named "each line is a message in a multi-part batch" vs what I was really looking for which is "each line is a message in a single part batch".

Perhaps the phrasing is wrong, I'm still learning..

[Jeffail]

Yeah I like that idea, I've I changed the names to try and make it clearer. We now have three options:

• each line is a message of a batch
• each line is a single message batch
• single message

Implemented: https://github.com/benthosdev/benthos-lab/commit/d7893fe38363032fc5b69ff62a8b52d32368d616 Released: https://github.com/benthosdev/benthos-lab/releases/tag/v0.7.1

Currently upgrading the production deployment.