jf-tech
commented
2 years ago Thanks for the suggestion!
Currently the parser internally generates and transforms into, for each ingested record, a generic map[string]interface{} for most the cases, then marshals it into a json []byte and returns to caller. That's what the Ingester.Read() interface method is about: https://github.com/jf-tech/omniparser/blob/master/schemahandler/schemaHandler.go#L58.
On one hand, we thought json is the ubiquitous format used in golang and it's virtually marshalable/unmarshalable into/from any other formats, hence the choice of it as the return format from Ingester.Read(). In your case, it would be a simple use of jsonpb (https://pkg.go.dev/github.com/golang/protobuf/jsonpb) to convert json output from omniparser into protobuf.
On the other hand, we understand there might be cases when perf is critical, and the extra map[string]interface{} -> json -> jsonpb step is undesired and one might want to convert the internally generated generic object map[string]interface{} directly into target/custom format. We, however, simply don't know if there is enough need for such a perf optimization.
What's your opinion? Do you think your case requires this perf enhancement by avoiding the json->pb conversion on your caller side?
Hi everyone, Are there plans to add mappings to additional output formats, or some documentation around how to output to custom formats?
current usecase would include ProtoBuf-sourced Go struct population from, say, EDI sources. Ideally without requiring a JSON intermediary, but for providing value reusing JSON output towards custom formats might help other consumers adopt omniparser and build precedence for different output options.