search

ViaQ / watches-cli

:watch: CLI tool to pull statistics from Elasticsearch
Apache License 2.0
1 stars 1 forks source link

Figure out data streaming with jq #18

Closed lukas-vlcek closed 7 years ago

lukas-vlcek commented 7 years ago

There is tutorial showing how to turn JSON output into CVS format using jq library. We need to find out if and how we can use jq streaming to handle huge data.

Also see https://github.com/stedolan/jq/wiki/Cookbook#processing-a-huge-json-array-of-items

richm commented 7 years ago

Don't we want watches to output csv directly? Do we want users of watches to pipe the output to jq if they want csv output?

lukas-vlcek commented 7 years ago

This depends. Right now we know the "user" is pbench. The question is what format pbench can consume (and it is not fully clear to me ATM, it seems it could consume even JSON directly?). So if users can consume JSON then we are ready to go. If users need CSV then the question is if we need to implement it or shop elsewhere, I do not know how much resources jq need to process huge data but it seem to provide a lot of interesting functionality.

lukas-vlcek commented 7 years ago

Closing for now. We can re-open later if needed. The first integration is for pbench and pbench post-processing scripts can perfectly work with any collection tool output format including JSON. See related ticket https://github.com/distributed-system-analysis/pbench/issues/396.