Clearly scope problems to solve with v1

[sungchun12]

The main goal is to invoke the playful and productive look and feel of hot reloading. When a frontend engineer updates colors or buttons in their javascript, their website instantly reflects those changes. It builds momentum and gives directional correctness. No extra commands to run. Just staying in flow.

When a user opens up a dbt project they should go through simple steps:

1. run fst start
2. Any file they touch with SQL models should be detected for modifications and run the below with extra signals that the program is doing exactly as expected
3. dbt build -s <model name>
4. duckdb preview query table

Considerations:

• set the preview limit row setting: ex: 5 rows, 10 rows, 100, etc. any value that's an integer
• If no tests exist, prompt the user if they want to generate and rerun with tests. Maybe make this a config flag to disable upfront. enabled by default
• save results to a markdown file in a flow_state/ subfolder so it can constantly reload results. example: flow_state/new_file.md, flow_state/customers.md

[sungchun12]

Get fst start to work without pointing to a file path. Think through whether or not to maintain file specific flow states like fst start --file-path <file path>

Edit: out of scope, not that useful to keep this to a single file at a time.

[sungchun12]

add manual prompt for user to create tests or not have fst start option to NOT generate test files at all when running

[sungchun12]

save results to an overwrite markdown file

[sungchun12]

Future Considerations:

• save hot reload results to a separate duckdb database and track individual and team productivity over time

Optimize performance:

Continuously analyze and improve the tool's performance to minimize latency during hot-reloading and ensure a seamless user experience. Incremental builds and partial previews:

Implement a mechanism to only rebuild and preview the affected parts of a model when a change is made, reducing the time spent on rebuilding the entire model. Integration with popular code editors:

Develop plugins or extensions for popular code editors like Visual Studio Code, Sublime Text, and Atom, allowing developers to access the hot-reloading tool directly within their preferred environment. Enhanced error reporting and debugging:

Implement a robust error reporting system that provides helpful and actionable feedback, guiding developers to quickly identify and fix issues in their models. Add debugging features that allow users to step through their SQL code and visualize the intermediate results, making it easier to understand and troubleshoot complex queries. Collaboration features:

Develop features that enable multiple developers to work on the same dbt project simultaneously, with real-time updates and conflict resolution mechanisms. Version control integration:

Integrate the tool with popular version control systems like Git, allowing developers to track and manage their changes more effectively. Support for custom plugins and extensions:

Create a framework that allows developers to create their own plugins and extensions for the hot-reloading tool, enabling them to tailor the tool to their specific needs. Continuous integration and deployment (CI/CD) support:

Develop features that help users automate their dbt development workflow, including automatic testing, build validation, and deployment to production environments. Integration with data visualization tools:

Allow users to quickly visualize their data using popular data visualization tools like Tableau, Power BI, or Looker, directly within the hot-reloading interface. Regular updates and maintenance:

Provide ongoing support, maintenance, and feature updates for the hot-reloading tool based on user feedback, industry trends, and the evolution of dbt best practices.

[sungchun12]

Future: use sqlglot to transpile sql across duckdb and other databases like snowflake, bigquery, etc.