algorand / conduit

Algorand's data pipeline framework.
MIT License
37 stars 26 forks source link

file-plugins: enabling messagepack format #142

Closed tzaffi closed 1 year ago

tzaffi commented 1 year ago

File Plugins: Add Messagepack Format and Default to msgp.gz

Based on the extension of the file pattern provided in a file_reader or file_writer plugin, choose:

  1. the Encoding Format (based on .msgp or .json)
  2. whether compression is used (based on .gz)

When no pattern is provided, it defaults to *.msgp.gz

Test Plan

This is all tested in CI including a new integration test conduit/plugins/importers/filereader/fileReadWrite_test.go

codecov[bot] commented 1 year ago

Codecov Report

Merging #142 (6c7f4c8) into master (442791a) will increase coverage by 3.65%. Report is 56 commits behind head on master. The diff coverage is 80.61%.

@@            Coverage Diff             @@
##           master     #142      +/-   ##
==========================================
+ Coverage   67.66%   71.31%   +3.65%     
==========================================
  Files          32       37       +5     
  Lines        1976     2761     +785     
==========================================
+ Hits         1337     1969     +632     
- Misses        570      690     +120     
- Partials       69      102      +33     
Files Changed Coverage Δ
conduit/data/block_export_data.go 100.00% <ø> (+92.30%) :arrow_up:
conduit/metrics/metrics.go 100.00% <ø> (ø)
conduit/pipeline/metadata.go 69.11% <ø> (ø)
conduit/plugins/config.go 100.00% <ø> (ø)
conduit/plugins/importers/algod/metrics.go 100.00% <ø> (ø)
...gins/processors/filterprocessor/fields/searcher.go 77.50% <ø> (ø)
...ins/processors/filterprocessor/filter_processor.go 83.82% <ø> (+3.54%) :arrow_up:
...plugins/processors/filterprocessor/gen/generate.go 34.28% <ø> (ø)
conduit/plugins/processors/noop/noop_processor.go 64.70% <ø> (+6.81%) :arrow_up:
pkg/cli/internal/list/list.go 20.75% <ø> (ø)
... and 21 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more