[Scoping] Move posthog to govt servers

[nikhil241]

@karntrehan As of now, I was able to find only these optimizations.

• Persistent Volumes: Specified absolute paths for data volumes to ensure data preservation across container restarts.
• Resource Management: Increased PostgreSQL max connections to 2000 and configured Redis in append-only mode to support higher concurrency and ensure data persistence
• Service Configuration:Utilized custom Dockerfiles and configuration files for ClickHouse to fine-tune settings for better performance and security.
• Worker and Web Services: Defined separate worker and web services with custom commands and environment variables to improve scalability and fault tolerance.
• Error Handling: Ensured all services restart on failure to enhance resilience and uptime
• Removed Unnecessary Services: Streamlined setup by removing services like maildev to reduce resource usage and potential failure points.

[nikhil241]

Migration Process:

• Backup Existing Data
• Set Up New Environment
• Transfer Data
• Update Environment Variables
• Deploy the New Setup
• Verify the Setup
• Update DNS/Access Points
• Test Thoroughly

[nikhil241]

@karntrehan I've found some additional optimizations done here according to this thread: https://posthog.com/questions/custom-events-in-posthog-are-being-processed-with-a-delay-of-2-days

Increased Kafka partitions and scaled plugin server instances, along with Gunicorn configuration changes for the web service, to remove delays and eliminate queue lags.

Impact on Migration Process:

• Scale Kafka Partitions: Before migration, adjust Kafka topic partitions to handle higher throughput.

• Scale Plugin Server Instances: Increase the number of plugin server instances during setup.

• Update Gunicorn Configurations: Apply necessary Gunicorn configuration changes to the web service to manage peak loads effectively.

[karntrehan]

Migration to be picked next sprint. Effort - 1 day. Ticket to be created by @nikhil241