search

MasterOfBinary / gobatch

Batch processing library for Golang.
MIT License
25 stars 6 forks source link

How do I get values out of a batch? #5

Open FrankDMartinez opened 3 years ago

FrankDMartinez commented 3 years ago

Hi.

Suppose I have a slice of string objects and I want to batch process them, pacing the result of each function call into a second slice. How would I do that?

Thanks in advance.

MasterOfBinary commented 3 years ago

One possibility is to have the Processor implementation contain the slice, and the Process method can append to the slice. Keep in mind, though, that Process can be called concurrently so you'll need to have the append operation protected with some kind of synchronization primitives.

Originally I had planned to build support for chaining together multiple processors, by having the processor output the new values which would be input to the next processor. However, it's not implemented right now and I didn't have a plan for retrieving the values directly. This would be a good addition if I can come up with a clean way to do so and have some time to implement it.

batazor commented 2 years ago

@MasterOfBinary Hello. Any idea how to return the result of the Process function can somehow use ps.Output <- lastItem

I am reading messages from kafka and I need to do a commit/ask if the data was processed successfully