tolbertam / sstable-tools

Tools for parsing, creating and doing other fun stuff with sstables
Apache License 2.0
162 stars 31 forks source link

#19 remove cql create and add compact option #23

Closed clohfink closed 8 years ago

clohfink commented 8 years ago
tolbertam commented 8 years ago

looks good to me. I liked having the key names, but I agree that it's not really too important (users should know their schema) and I agree with your rationale for simplicity regarding a json2sstable implementation.

devdazed commented 8 years ago

would it be possible to get an example output of the compact option?

clohfink commented 8 years ago
[
{"partition" : {"key" : ["e","f"]},"rows" : [{"type" : "row","clustering" : ["g","h"],"liveness_info" : {"tstamp" : 1451574763525324},"cells" : [{"name" : "value","value" : "2"}]}]},
{"partition" : {"key" : ["a","b"]},"rows" : [{"type" : "row","clustering" : ["c","d"],"liveness_info" : {"tstamp" : 1451574763506421},"cells" : [{"name" : "value","value" : "1"}]}]}
]
devdazed commented 8 years ago

it's not a huge issue, but from the perspective of apps that will use this for streaming it would be best if the beginning [ and ending ] and the trailing , per line were removed. it would mean that each line can be ingested and JSON parsed without having to determine if the line only contains one char and have to rstrip(',') the row. what do you think?

clohfink commented 8 years ago

I would like to keep this as valid json for people who will use normal or incremental parsers (which will still be required when checking for wide partitions or ton of tombstones anyway). We could maybe build a different tool

java -jar sstable-tools.jar dump [sstable]

and come up with different format, a non-hierarchical, all state contained in each object outputted one that just translates the raw binary to something more easily parseable. but I think the "to json" tool should output json. Since doing per line parsing anyway shouldn't be bad to remove a trailing ','.

clohfink commented 8 years ago

I think I have an idea that may work for you devdazed. Will take a little so stay tuned

devdazed commented 8 years ago

i can live with that, like i said, it's not difficult to strip those things out, was just a very minor annoyance.

tolbertam commented 8 years ago

Going to go ahead and merge this as there are also some other good changes in here that'd be nice to have on master. We can follow up with another PR for more changes to compact mode or add a dump command, thanks!