CS Tools v1.4.0 Planning

[boonhapus]

This issue will be used to track progress against all planned items for v1.4.0 .. the goal date for this release is prior to the start of the holidays (11/20 12/23).

Additional feedback can be provided in the comments below.

Documentation

• [ ] present an overview + architecture page prior to introducing the tutorial
• [x] reflect minimum version SW 7.2.1 8.4.1
• [ ] reflect minimum version python 3.7
• [ ] tool docs pages are now programmatically generated
• [ ] fullpath cron example w/ which

New Released Tools

swarm perform automated testing against your thoughtspot cluster

• [x] strategies for LoadTest , ValidityCheck , DataParity
• [x] results can be dumped to a syncer

bulk-tag use the power of Search to tag your objects

• [ ] search - target a datasource which returns guids, leverage those in tagging
• [ ] from-tabular - use the power of syncers to return guids to tag
• [ ] single - quickly tag individual objects

Modified Tools

archiver

• [x] add a user & group filter to archiver identify (#22)

searchable

• [x] v1 of the ThoughtSpot SpotApp
• [ ] add Formula Text to Column Metadata
• [x] implement a more efficient algorithm for finding SharingAccess
• [x] migrate to public apis
• [ ] allow selective extraction for tables (eg. --tables TS_USER would only dump a single table) (#27)
• [ ] leverage intermediate data storage for low-memory environments
  • [ ] this will be one of: CSV, parquet, or SQLite

scriptability

• [x] (maybe) replay authorship and sharing after upload
• [ ] (maybe) implement delta migrations (eg. export any content created/modified after X date)

security-sharing

• [x] rename to bulk-sharing
• [x] rename command run to cls-ui
• [x] rework share in two commands
  • [x] from-tabular takes a form similar to bulk-deleter from-tabular
    • [x] leverages syncers and the SharingAccess table format
  • [x] single takes a form similar to bulk-deleter single

user-management

• [x] explore more efficient content scan algorithm
• [x] add dataflow jobs to transfer-ownership content scan (#34)
  • [x] triggered by option --include-dataflow-jobs
  • [x] this check only happens for ts.platform.deployment == "software"
• [x] platforms with excess users/groups trigger callosum time on sync
  • [x] need a way for intelligent batching

Syncer Protocol

• [x] filepath-based syncers should happy-path defaults (just the filepath as the directive)
• [x] database syncers: defer Table creation until a CSTools command is run
• [x] CSV syncer: allow datetime specification (based on link)
• [ ] implement a filter predicate capability
  • [ ] database syncers translate into SQL WHERE
  • [ ] non-database syncers will run in the python layer
• [ ] add support for the Parquet format ([link](https://aseigneurin.github.io/2017/03/14/incrementally-loaded-parquet-files.html) and link)
• [x] add support for the JSON format
• [x] handle no-data-returned by attempted to dump [all syncers]

Internals, Deprecations, UX

bootstrapper

• [x] venv handling code in parent-lib
• [x] explore adding offline support for the M1 chip (#29)
• [x] add a remote install option for those who have outside internet access

core

• [x] add a cs_tools upgrade --check-version / cs_tools upgrade {local | remote} command
• [x] rework and formalize the dependency injection system
• [x] cs_tools config show --config needs better default colors
• [x] cs_tools config show --config should include syncer config details as well (#28) - [] explore pip-tools over poetry
  • [] if successful, rework dependencies to leverage pip-tools workflow instead of poetry
• [x] bump minimum python version to 3.7
  • [x] add a GitHub issue which shows how to install a separate python version on the ThoughtSpot cluster
  • [x] bump dependencies to include new typer==0.0.6 for deep rich integration
  • [x] rework Tool & Command usage with deep rich integration (link and link)
  • [x] add epilog for tools which link to the docs pages and syncer
• [x] expose and document programmatic access to class CSTool API (#33 & #36)

Bugfix

• [x] venv handling code in parent-lib
• [x] #42 subtype -> subtypes
• [x] user-management transfer-ownership now checks objects validity and ownership
• [x] bring more clarity to the warning message on usage of private API calls (#38)
• [x] add ?callosumTimeout=600 to the super private periscope API call
• [x] add required --options to pip-based activities in bootstrapper and syncer requirements installs
  • [x] --trusted-host files.pythonhosted.org
  • [x] --trusted-host pypi.org
  • [x] --trusted-host pypi.python.org

[boonhapus]

for linkthrough: https://docs.thoughtspot.com/software/latest/python-upgrade

[boonhapus]

V1.4.0 is released 😄