Job output should be able to be filtered, sorted, and field-filtered - Githubissues

chronicle-app / chronicle-etl

📜 A CLI toolkit for extracting and working with your digital history

https://chronicle.app/

MIT License

149 stars 3 forks source link

Job output should be able to be filtered, sorted, and field-filtered #8

Open hyfen opened 4 years ago

hyfen commented 4 years ago

In line with CLI conventions, we should provide these options:

filter: only output records that match a set of conditions
sort: sort records by a given column (ascending, descending)
fields: only output values from a given set of fields

There are a few ways this could be implemented

global --filter, --sort, --fields flags for chronicle-etl
passing as options for the Loader through --loader-opts
treating them as multiple transformers that run before getting to the Loader (requires completion of #6)

We'd also make Loaders determine if output can be incremental or has to happen when the job is complete (for instance, if a sort option is used)