search

jf-tech / omniparser

omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.
MIT License
931 stars 68 forks source link

Introduce `idr.Node` caching and recycling #84

Closed jf-tech closed 3 years ago

jf-tech commented 3 years ago

As a result, adding Release(*idr.Node) to FormatReader so that Ingester will free up *idr.Node allocations for recycling

Updated all benchmarks.

No degradation so far. The reason the json sample bench mark didn't improve much is because there isn't much nodes allocated per read. However typically in the CSV parsing/transform scenarios, files are usually very long and node allocation caching saving would start to show significantly.

resolves #83

codecov[bot] commented 3 years ago

Codecov Report

Merging #84 into master will decrease coverage by 0.37%. The diff coverage is 88.46%.

Impacted file tree graph

@@             Coverage Diff             @@
##            master      #84      +/-   ##
===========================================
- Coverage   100.00%   99.62%   -0.38%     
===========================================
  Files           35       35              
  Lines         1552     1587      +35     
===========================================
+ Hits          1552     1581      +29     
- Misses           0        6       +6
Impacted Files Coverage Δ
idr/jsonreader.go 97.39% <40.00%> (-2.61%) :arrow_down:
idr/xmlreader.go 96.51% <40.00%> (-3.49%) :arrow_down:
customfuncs/javascript.go 100.00% <100.00%> (ø)
handlers/omni/v2/fileformat/csv/reader.go 100.00% <100.00%> (ø)
handlers/omni/v2/fileformat/json/reader.go 100.00% <100.00%> (ø)
handlers/omni/v2/fileformat/xml/reader.go 100.00% <100.00%> (ø)
handlers/omni/v2/ingester.go 100.00% <100.00%> (ø)
handlers/omni/v2/transform/parse.go 100.00% <100.00%> (ø)
idr/node.go 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update d44a0be...7feffd4. Read the comment docs.

jf-tech commented 3 years ago

somehow the last commit 

commit 601656371a3abd8e8810c56d1ca64fbde284439e (HEAD -> nodecache, origin/nodecache)
Author: jf-tech <jf.tech.llc@gmail.com>
Date:   Sat Oct 10 10:38:09 2020 +1300

    fix cov

refuses to show up.

seems github sync issue. will close and recreate the PR.