Update tools and benchmark datasets to read compressed json

f5 / otel-arrow-adapter

Adapter used to convert OTEL batches to/from OTEL Arrow batches in both directions.

Apache License 2.0

39 stars 6 forks source link

Closed moh-osman3 closed 1 year ago

moh-osman3 commented 1 year ago

This PR

updates the [logs|traces|metrics] datasets to read files that are in a compressed json format. Files are read line by line. This logic is similar to how the otel contrib filereceiver component will read compressed json.
updates the [logs|traces|metrics] test data generating tools to write compressed json format. Telemetry is written one at a time followed by a newline. This logic is similar to how the otel contrib fileexporter will write compressed json.

lquerel commented 1 year ago

@moh-osman3 there is an issue in the build process preventing us to merge this PR. Could you fix it and I will merge this PR. Thanks.

moh-osman3 commented 1 year ago

@moh-osman3 there is an issue in the build process preventing us to merge this PR. Could you fix it and I will merge this PR. Thanks.

@lquerel Ahh forgot to update one of the test files with the updated function signature. I think it should be fixed now. Thank you!

lquerel commented 1 year ago

@moh-osman3 thanks for this PR.