Persistant storage

[tsvetkovv]

As the project evolves and grows, we are encountering limitations with our current app state storage approach, which relies on a file-based system. To address these limitations and enhance the scalability, performance, and reliability of the application, I propose integrating a Postgres database as the new storage solution for the app state.

Background

Currently, the app state is stored in a file, which has served us well during the initial stages of development. However, as the user base and data volume increase, we foresee potential challenges and drawbacks with this approach, including:

• Performance bottlenecks: Accessing and updating the file-based app state might lead to performance degradation, especially with concurrent user interactions and large datasets.

• Data integrity and security: The file-based system lacks built-in mechanisms to ensure data consistency and protection against data loss or corruption.

• Limited scalability: As the app gains popularity, the file-based approach may not scale efficiently to handle the growing user base and higher data demands.

Proposal

To address these concerns and future-proof our application, I propose adopting a Postgres database to manage the app state. Postgres is a robust and widely-used relational database that offers several benefits, including:

• Performance optimization: Postgres is designed to handle complex queries and large datasets efficiently, providing faster response times and reducing latency.

• Data integrity and transactions: Postgres ensures data integrity by supporting ACID (Atomicity, Consistency, Isolation, Durability) properties, allowing us to maintain data consistency even during concurrent operations.

• Scalability and concurrency: With Postgres, we can easily scale our database to accommodate increased user traffic and concurrent connections, ensuring a smooth user experience.

• Data backup and recovery: Postgres offers reliable mechanisms for data backup and restoration, minimizing the risk of data loss.

Tasks

• [ ] Database setup: Install and configure Postgres on the server to establish a connection with the application.
• [ ] Application code changes: Modify the application code to interact with the Postgres database using appropriate queries and transactions.
• [ ] Database schema design: Define the schema to represent the app state data and its relationships.
• [ ] Data migration: Develop a script to migrate the existing app state data from the file-based system to the Postgres database.
• [ ] Error handling: Implement error handling mechanisms to handle database-related exceptions gracefully.

Acceptance Criteria:

• The Postgres database is successfully integrated into the application for storing and managing the app state.
• The app state migration from the file-based system to the Postgres database is verified and validated.
• Comprehensive testing demonstrates that the application functions as expected with the new database setup.
• The team reviews and approves the changes, ensuring there are no regressions or critical issues introduced.

I already started implementation. Let me know your thoughts before I roll out a huge pull request.

[d-manevich]

I'm worried about the complexity of infrastructural support for such a solution We use a small amount of data and it is not critical for users We only need to store data for online statistics for the last month But for the sake of such a problem, I consider it redundant to create and maintain a complex infrastructure, with database support, backups, etc. Apart from the increasing complexity of transferring the bot between servers

[d-manevich]

As an intermediate solution, I propose to make the use of databases optional, and I propose to consider no code solutions, such as firebase, etc.

[A77AY]

I will try to transfer to vercel serverless

[A77AY]

there is also a KV database