search

tolbertam / sstable-tools

Tools for parsing, creating and doing other fun stuff with sstables
Apache License 2.0
162 stars 31 forks source link

Offline Select 'cqlsh'-style console mode #35

Closed tolbertam closed 8 years ago

tolbertam commented 8 years ago

Initial commit of a 'cqlsh'-like mode for selecting data from sstable(s).

Via jline2, supports history maintained between sessions.

Note: History doesn't work with IntelliJ, IntelliJ also repeats whatever you input, better to use a terminal window to test this out.

Usage:

Offline CQL Shell for Apache Cassandra 3.x
Options:
  -f <arg> Execute commands from FILE, then exit.
  -s <arg> Schema file to use.

Example usage and output:

$ java -jar target/sstable-tools-3.0.0-SNAPSHOT.jar cqlsh ma-1-big-Data.db -s schema.cql
cqlsh> select * from sstable where ticker in ('YHOO') and date > '2016-02-01';
 ┌─────────┬─────────────────────┬────────────┬─────────┬─────────┬─────────┬─────────┬─────────┐
 │ticker   │date                 │adj_close   │close    │high     │low      │open     │volume   │
 ╞═════════╪═════════════════════╪════════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
 │YHOO     │2016-02-19 00:00-0600│30.040001   │30.040001│30.23    │29.700001│30.190001│20706100 │
 ├─────────┼─────────────────────┼────────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
 │YHOO     │2016-02-18 00:00-0600│29.42       │29.42    │30.139999│29.389999│29.559999│15259400 │
 ├─────────┼─────────────────────┼────────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
 │YHOO     │2016-02-17 00:00-0600│29.370001   │29.370001│29.66    │29.059999│29.469999│12891600 │
 ├─────────┼─────────────────────┼────────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
 │YHOO     │2016-02-16 00:00-0600│29.280001   │29.280001│29.440001│27.940001│27.98    │20127300 │
 ├─────────┼─────────────────────┼────────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
 │YHOO     │2016-02-12 00:00-0600│27.040001   │27.040001│27.32    │26.719999│27.120001│12986200 │
 ├─────────┼─────────────────────┼────────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
 │YHOO     │2016-02-11 00:00-0600│26.76       │26.76    │26.969999│26.15    │26.459999│11154900 │
 ├─────────┼─────────────────────┼────────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
 │YHOO     │2016-02-10 00:00-0600│27.10       │27.10    │27.809999│26.84    │27.110001│8933600  │
 ├─────────┼─────────────────────┼────────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
 │YHOO     │2016-02-09 00:00-0600│26.82       │26.82    │27.690001│26.51    │26.639999│13919100 │
 ├─────────┼─────────────────────┼────────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
 │YHOO     │2016-02-08 00:00-0600│27.049999   │27.049999│27.969999│26.48    │27.610001│24473600 │
 ├─────────┼─────────────────────┼────────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
 │YHOO     │2016-02-05 00:00-0600│27.969999   │27.969999│29.139999│27.73    │29.059999│16077500 │
 ├─────────┼─────────────────────┼────────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
 │YHOO     │2016-02-04 00:00-0600│29.15       │29.15    │29.23    │27.709999│27.91    │28517000 │
 ├─────────┼─────────────────────┼────────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
 │YHOO     │2016-02-03 00:00-0600│27.68       │27.68    │28.610001│26.57    │28.450001│55527600 │
 ├─────────┼─────────────────────┼────────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
 │YHOO     │2016-02-02 00:00-0600│29.059999   │29.059999│30.23    │28.129999│29.32    │34022500 │
 └─────────┴─────────────────────┴────────────┴─────────┴─────────┴─────────┴─────────┴─────────┘
cqlsh> exit;
$

TODO

  1. Find out why we are leaking refs in SSTableReader (LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@6b25a2cc) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1601791084:/Users/atolbert/Documents/Projects/sstable-tools/ma-1-big was not released before the reference was garbage collected)
  2. Add 'USE' command to switch active sstable. (Done, @clohfink)
  3. Log number of rows. (Done, @tolbertam)
  4. 'help' command support (Done, @clohfink)
  5. Row paging. (Done, @tolbertam)
  6. Tab completion (Done, @clohfink)
  7. Allow select * from 'sstablepath' to override active sstable for particular query. (Done, @clohfink)
  8. Allow 'USE' on a directory to involve all sstables or multiple ones? (Done, @clohfink)
  9. Command to import schema from file (Done, @tolbertam)
  10. Persist state (used sstable(s), active schema, paging) (Done, @tolbertam)
  11. Implement paging for dump command. (Done, @tolbertam)
  12. Progress bars for describe on large table (Done, @clohfink)
tolbertam commented 8 years ago

Nice work :+1:, I'm going to give paging a go tonight. Do we have a command for importing a schema from file in the cli? I know we can provide at when starting but was wondering if we could otherwise. I can add that if we don't have it.