Omniparser is a native Golang ETL parser that ingests input data of various formats (CSV, txt, fixed length/width, XML, EDI/X12/EDIFACT, JSON, and custom formats) in streaming fashion and transforms data into desired JSON output based on a schema written in JSON.
Min Golang Version: 1.16
Omniparser is publicly available under MIT License. Individual and corporate sponsorships are welcome and gratefully appreciated, and will be listed in the SPONSORS page. Company-level sponsors get additional benefits and supports granted in the COMPANY LICENSE.
Docs:
transform_declarations
.custom_func
, Specially javascript
: An in depth look of how custom_func
is used, specially the all mighty javascript
(and javascript_with_context
).References:
Examples:
In the example folders above you will find pairs of input files and their schema files. Then in the
.snapshots
sub directory, you'll find their corresponding output files.
~~Use The Playground (may need to wait for a few seconds for instance to wake up) for trying out schemas and inputs, yours or existing samples, to see how ingestion and transform work.~~
As for now (2023/03/14), all of our previous free docker hosting solutions went away and we haven't found another one yet. For now please clone the repo and use ./cli.sh
as described in the Getting Started page.
csv2
file format that supersedes the original csv
format with support of hierarchical and nested records.fixedlength2
file format that supersedes the original fixed-length
format with support of hierarchical and nested envelopes.Transform.RawRecord()
for caller of omniparser to access the raw ingested record.custom_parse
in favor of custom_func
(custom_parse
is still usable for
back-compatibility, it is just removed from all public docs and samples).NonValidatingReader
EDI segment reader.omni.2.1
due a number of incompatible schema changes:
'result_type'
-> 'type'
'ignore_error_and_return_empty_str
-> 'ignore_error'
'keep_leading_trailing_space'
-> 'no_trim'
times.SmartParse
.transform
cmd or long-running http server
mode).javascript
engine integration as a custom_func.