Logging at Zerodha;;At Zerodha, we run a multitude of internal and public-facing services that generate copious amounts of logs. While developers use these logs to debug or troubleshoot incidents, some services also emit logs that must be persisted for prolonged periods to comply with numerous regulatory requirements. In this post, I will delve into our experiences with the ELK stack, why it didn&rsquot fit our needs and our migration to ClickHouse.Why ELK wasn’t the right fit for us In 2018, we adopted the ELK stack as our de facto stack for storing application logs.;;-;;https://zerodha.tech/blog/logging-at-zerodha/;;zerodha;;2023-3-23;;;User disengagement;;Over time, I have come to realise that the term user &ldquoengagement” in software, more often than not, is a thinly veiled proxy for user entrapment, whether intentional on the part of software designers or not. The trend of shallow user numbers and “engagement” metrics taking centre stage in business valuations has created perverse incentives at a massive scale for software to indulge in unethical practices of “engaging” users at any cost, with no respect for the limited and fast eroding levels of cognitive resources.;;-;;https://zerodha.tech/blog/user-disengagement/;;zerodha;;2022-5-6;;;From Native to React Native to Flutter;;At Zerodha, the first mobile version our flagship trading platform Kite was written as a native Android app in 2015. After building a cross-platform version in React Native in 2017, we finally settled for a full rewrite in Flutter in 2018, a choice that has paid off really well for us. There were several factors and trade-offs that prompted these rewrites.This post covers our journey and experiences with each of the frameworks, and why we finally bet on Flutter even when it was bleeding edge alpha technology.;;-;;https://zerodha.tech/blog/from-native-to-react-native-to-flutter/;;zerodha;;2022-1-17;;;Scaling with common sense #2: Being future ready.;;Over the last year, owing to the unexpected rally in capital markets, Zerodha&rsquos customer base has more than tripled, significantly increasing the number of concurrent users on our platforms along with the traffic and load they generate on numerous systems in the background. For context, in January 2020, we were handling 2+ million retail trades daily. In April 2020, it had risen to 7+ million. Today, it goes up to 12+ million.;;-;;https://zerodha.tech/blog/being-future-ready-with-common-sense/;;zerodha;;2021-7-28;;;Working with PostgreSQL;;This post is in the context of the large, data heavy PostgreSQL instances that store historical transactional data and reports, the databases that power Console and its large scale number crunching and reporting. It talks about how we self-host, tune, and manage all our DB instances on bare EC2 instances. For high availability and backups, we use simple failover replicas and for backups, AWS disk snapshots.The Console DBs store hundreds of billions of rows of different kinds of financial and transactional data, currently close to 20 TB, across four sharded nodes.;;-;;https://zerodha.tech/blog/working-with-postgresql/;;zerodha;;2021-4-22;;;A lesson in creating and using niche business DSLs at scale;;At Zerodha, we process millions of trades in real-time, where each trade comes into the system as concurrent high throughput HTTP requests. Each trade increases the latency for subsequent orders in the queue that are under processing at the same time at our OMS (Order Management System). When a single order comes through to the OMS, it goes through a bunch of computationally intensive validations and adds to the latency. To reduce the latency of orders, we decided to offload some of these business validations from the OMS into an external component called Veto, which pre-validates incoming orders based on custom dynamic rules set by our Risk Management team.;;-;;https://zerodha.tech/blog/a-lesson-in-niche-business-dsls-at-scale/;;zerodha;;2020-11-24;;;Alar: The making of an open source dictionary;;ನಮಸ್ಕಾರ (Namaskāra)! This is not a post on fintech, or even technology for that matter. This is the story of a product of tenacity, selflessness, and passion a product that will transcend and outlive most technology we know of. This is the story of a massive dictionary that will become the window to a language spoken by tens of millions of people for generations to come, a resource its author has donated to posterity.;;-;;https://zerodha.tech/blog/alar-the-making-of-an-open-source-dictionary/;;zerodha;;2020-9-22;;;Scaling with common sense;;&ldquoScaling” is not a specific technique but an entire spectrum that stretches all the way from “Google scale” to the K8s cluster Karan runs on a Raspberry Pi plugged into a 12V shaver outlet in his bathroom to encrypt his DNS queries for peak privacy, while he asks Alexa to dim the lights and play his favorite non-mainstream indie music.It is a collection of practices unique and intimate to every organisation, and is the product of an infinite number of variables; the numerous domain-specific problems, the structure of the organisation, the nature of the people involved and their biases, countless engineering decisions and trade-offs, technical debt and history, ad infinitum.;;-;;https://zerodha.tech/blog/scaling-with-common-sense/;;zerodha;;2020-6-14;;;Infrastructure monitoring with Prometheus at Zerodha;;On any given day, we handle around 15% of daily retail trading volume across all stock exchanges in India. Billions of requests generated in the process are handled by a suite of systems we have built in-house. Also, we are very particular on self-hosting as many dependencies as possible, everything from CRMs to large databases, Kafka clusters, mail servers etc.To aid these primary systems, there are a large number of ancillary workloads that run, covering everything from real-time trades, document processing, KYC, and account opening, legal and compliance, complex, large scale P&L and number crunching, and a wide range of backoffice workloads.;;-;;https://zerodha.tech/blog/infra-monitoring-at-zerodha/;;zerodha;;2020-4-27;;;Hello, World!;;Zerodha, now India&rsquos largest stock broker, bootstrapped and profitable, turns ten years old this year. The Zerodha tech team turns seven years old. The tech team has remained largely elusive over the course of our existence. While we have pondered starting a tech blog for more than half a decade, we have often found ourselves too busy building the fundamental blocks underlying a stock brokerage. We have also been stalled by a sense of unpreparedness to talk to the world about our very unconventional setup.;;-;;https://zerodha.tech/blog/hello-world/;;zerodha;;2020-4-6;;;Projects;;listmonk ★ 10896High performance, self-hosted, newsletter and mailing list manager with a modern dashboard. Single binary app.koanf ★ 1890Simple, extremely lightweight, extensible, configuration management library for Go. Support for JSON, TOML, YAML, env, command line, file, S3 etc. Alternative to viper.csvtotable ★ 1054Simple command-line utility to convert CSV files to searchable and sortable HTML table.otpgateway ★ 350;;-;;https://zerodha.tech/projects/;;zerodha;;2001-1-1
Logging at Zerodha;;At Zerodha, we run a multitude of internal and public-facing services that generate copious amounts of logs. While developers use these logs to debug or troubleshoot incidents, some services also emit logs that must be persisted for prolonged periods to comply with numerous regulatory requirements. In this post, I will delve into our experiences with the ELK stack, why it didn&rsquot fit our needs and our migration to ClickHouse.Why ELK wasn’t the right fit for us In 2018, we adopted the ELK stack as our de facto stack for storing application logs.;;-;;https://zerodha.tech/blog/logging-at-zerodha/;;zerodha;;2023-3-23;;;User disengagement;;Over time, I have come to realise that the term user &ldquoengagement” in software, more often than not, is a thinly veiled proxy for user entrapment, whether intentional on the part of software designers or not. The trend of shallow user numbers and “engagement” metrics taking centre stage in business valuations has created perverse incentives at a massive scale for software to indulge in unethical practices of “engaging” users at any cost, with no respect for the limited and fast eroding levels of cognitive resources.;;-;;https://zerodha.tech/blog/user-disengagement/;;zerodha;;2022-5-6;;;From Native to React Native to Flutter;;At Zerodha, the first mobile version our flagship trading platform Kite was written as a native Android app in 2015. After building a cross-platform version in React Native in 2017, we finally settled for a full rewrite in Flutter in 2018, a choice that has paid off really well for us. There were several factors and trade-offs that prompted these rewrites.This post covers our journey and experiences with each of the frameworks, and why we finally bet on Flutter even when it was bleeding edge alpha technology.;;-;;https://zerodha.tech/blog/from-native-to-react-native-to-flutter/;;zerodha;;2022-1-17;;;Scaling with common sense #2: Being future ready.;;Over the last year, owing to the unexpected rally in capital markets, Zerodha&rsquos customer base has more than tripled, significantly increasing the number of concurrent users on our platforms along with the traffic and load they generate on numerous systems in the background. For context, in January 2020, we were handling 2+ million retail trades daily. In April 2020, it had risen to 7+ million. Today, it goes up to 12+ million.;;-;;https://zerodha.tech/blog/being-future-ready-with-common-sense/;;zerodha;;2021-7-28;;;Working with PostgreSQL;;This post is in the context of the large, data heavy PostgreSQL instances that store historical transactional data and reports, the databases that power Console and its large scale number crunching and reporting. It talks about how we self-host, tune, and manage all our DB instances on bare EC2 instances. For high availability and backups, we use simple failover replicas and for backups, AWS disk snapshots.The Console DBs store hundreds of billions of rows of different kinds of financial and transactional data, currently close to 20 TB, across four sharded nodes.;;-;;https://zerodha.tech/blog/working-with-postgresql/;;zerodha;;2021-4-22;;;A lesson in creating and using niche business DSLs at scale;;At Zerodha, we process millions of trades in real-time, where each trade comes into the system as concurrent high throughput HTTP requests. Each trade increases the latency for subsequent orders in the queue that are under processing at the same time at our OMS (Order Management System). When a single order comes through to the OMS, it goes through a bunch of computationally intensive validations and adds to the latency. To reduce the latency of orders, we decided to offload some of these business validations from the OMS into an external component called Veto, which pre-validates incoming orders based on custom dynamic rules set by our Risk Management team.;;-;;https://zerodha.tech/blog/a-lesson-in-niche-business-dsls-at-scale/;;zerodha;;2020-11-24;;;Alar: The making of an open source dictionary;;ನಮಸ್ಕಾರ (Namaskāra)! This is not a post on fintech, or even technology for that matter. This is the story of a product of tenacity, selflessness, and passion a product that will transcend and outlive most technology we know of. This is the story of a massive dictionary that will become the window to a language spoken by tens of millions of people for generations to come, a resource its author has donated to posterity.;;-;;https://zerodha.tech/blog/alar-the-making-of-an-open-source-dictionary/;;zerodha;;2020-9-22;;;Scaling with common sense;;&ldquoScaling” is not a specific technique but an entire spectrum that stretches all the way from “Google scale” to the K8s cluster Karan runs on a Raspberry Pi plugged into a 12V shaver outlet in his bathroom to encrypt his DNS queries for peak privacy, while he asks Alexa to dim the lights and play his favorite non-mainstream indie music.It is a collection of practices unique and intimate to every organisation, and is the product of an infinite number of variables; the numerous domain-specific problems, the structure of the organisation, the nature of the people involved and their biases, countless engineering decisions and trade-offs, technical debt and history, ad infinitum.;;-;;https://zerodha.tech/blog/scaling-with-common-sense/;;zerodha;;2020-6-14;;;Infrastructure monitoring with Prometheus at Zerodha;;On any given day, we handle around 15% of daily retail trading volume across all stock exchanges in India. Billions of requests generated in the process are handled by a suite of systems we have built in-house. Also, we are very particular on self-hosting as many dependencies as possible, everything from CRMs to large databases, Kafka clusters, mail servers etc.To aid these primary systems, there are a large number of ancillary workloads that run, covering everything from real-time trades, document processing, KYC, and account opening, legal and compliance, complex, large scale P&L and number crunching, and a wide range of backoffice workloads.;;-;;https://zerodha.tech/blog/infra-monitoring-at-zerodha/;;zerodha;;2020-4-27;;;Hello, World!;;Zerodha, now India&rsquos largest stock broker, bootstrapped and profitable, turns ten years old this year. The Zerodha tech team turns seven years old. The tech team has remained largely elusive over the course of our existence. While we have pondered starting a tech blog for more than half a decade, we have often found ourselves too busy building the fundamental blocks underlying a stock brokerage. We have also been stalled by a sense of unpreparedness to talk to the world about our very unconventional setup.;;-;;https://zerodha.tech/blog/hello-world/;;zerodha;;2020-4-6;;;Projects;;listmonk ★ 10896High performance, self-hosted, newsletter and mailing list manager with a modern dashboard. Single binary app.koanf ★ 1890Simple, extremely lightweight, extensible, configuration management library for Go. Support for JSON, TOML, YAML, env, command line, file, S3 etc. Alternative to viper.csvtotable ★ 1054Simple command-line utility to convert CSV files to searchable and sortable HTML table.otpgateway ★ 350;;-;;https://zerodha.tech/projects/;;zerodha;;2001-1-1