search

jf-tech / omniparser

omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.
MIT License
931 stars 68 forks source link

Intro `*Node` cache and recycling #86

Closed jf-tech closed 3 years ago

jf-tech commented 3 years ago

As a result, adding Release(idr.Node) to FormatReader so that Ingester will free up idr.Node allocations for recycling

Updated all benchmarks.

Not much degradation so far. The reason the json sample bench mark didn't improve much is because there isn't much nodes allocated per read. However typically in the CSV parsing/transform scenarios, files are usually very long and node allocation caching saving would start to show significantly.

resolves #83

jf-tech commented 3 years ago

all right i gave up. this PR it is. no checks that's fine. already ran tests many times locally.

codecov[bot] commented 3 years ago

Codecov Report

Merging #86 into master will not change coverage. The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff            @@
##            master       #86   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           35        35           
  Lines         1552      1587   +35     
=========================================
+ Hits          1552      1587   +35
Impacted Files Coverage Δ
customfuncs/javascript.go 100.00% <100.00%> (ø)
handlers/omni/v2/fileformat/csv/reader.go 100.00% <100.00%> (ø)
handlers/omni/v2/fileformat/json/reader.go 100.00% <100.00%> (ø)
handlers/omni/v2/fileformat/xml/reader.go 100.00% <100.00%> (ø)
handlers/omni/v2/ingester.go 100.00% <100.00%> (ø)
handlers/omni/v2/transform/parse.go 100.00% <100.00%> (ø)
idr/jsonreader.go 100.00% <100.00%> (ø)
idr/node.go 100.00% <100.00%> (ø)
idr/xmlreader.go 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update d44a0be...a6b0b22. Read the comment docs.