It seems like we can "just" remove the json encode instruction, update related tests and voila. But that's a breaking change of course, consumers need to update their pipeline to remove the following json decode stage.
What I'm not sure, is if it's still necessary to allow json encoding, as all following stages use a native map format anyway. If we think we must maintain a json-encoded option, there's several possible approaches I can see so far:
add a json-encode option as part of the ingest-collector stage params
create a new json-encode stage (oddly, this stage should be a "transform" stage and not an "encode" stage, because transform has input & output, whereas "encode" has no output (??) )
jotak
commented
2 years ago
As commented here https://github.com/netobserv/flowlogs-pipeline/blob/main/pkg/pipeline/ingest/ingest_collector.go#L168-L169, it seems like there's an easy perf improvement we can do by avoiding json encoding during flow ingestion, especially as it kinda forces users to use json decoding as the following stage.
It seems like we can "just" remove the json encode instruction, update related tests and voila. But that's a breaking change of course, consumers need to update their pipeline to remove the following json decode stage.
What I'm not sure, is if it's still necessary to allow json encoding, as all following stages use a native map format anyway. If we think we must maintain a json-encoded option, there's several possible approaches I can see so far:
ping @eranra @KalmanMeth @ronensc