Week4 - Lessons from Building Observability Tools at Netflix

[kymr]

Title

• Lessons from Building Observability Tools at Netflix

Summary

This article introduce why Netflix build centralized observability tools and how. As Netflix business grows, scalability is very critical. Also because they adapted mircoservices architecture, it is getting harder and harder to troubleshoot issues and find a root cause when error happens. Because of that, they built many observerability tools to keep most of engineers going on innovating rapidly and bringing to their customers. For scalable logs, they built Mantis which is a real-time streaming processing platform. They also built a tool for distributed request tracing. It serves insights on microservices environment by providing additional contextual data based on constituent traces. They have Atlas for monitoring metrics and analyzing it. They use Cassandra, Elasticsearch and Hive as data persistent storages. Finally, they provide user interfaces which is customized views for each group what they want to see.

Reference

• https://medium.com/netflix-techblog/lessons-from-building-observability-tools-at-netflix-7cfafed6ab17

[kymr]

Words

• Our mission at Netflix is to deliver joy
• presented with a delightful experience.
• We are constantly innovatiing
• at a rapid pace
• in pursuit of this mission
• Our innovations span
• We tackle that challenge
• analyze those measurement
• to derive meaningful insights
• At some point in business growth
• we learned that storing raw application logs won't scale
• To address scalability
• we needed a wat to gain insight into
• but is not sufficient to
• Augmenting the request trace
• Besides
• we've learned how to define actionable alertin
• beyond just threshold alerting
• Our observability tools need to access various persisted data types
• Choosing which kind of database to store
• how each particular data type is written and retrieved
• Data presentation requirements vary widely
• deliver views tailored to a user's profile
• We started our tooling efforts with providing visibility into device
• instead of having to use separate data specific tools
• Providing visibility into logs is valuable
• at some point in our business growth
• because the increasing volume of log data caused our storage cost to balloon
• Besides reducing our storage retention time period
• we addressed scalability by implementing
• Instead of saving all logs to persistent storage
• Mantis enables our users to stream logs
• A query that retrieves a sample of playback start events
• Once a user obtains an initial set of samples
• they can iteratively refine their queries to narrow down the specific set of samples
• perhaps the root cause of an issue
• The key takeaway is that
• in terms of cost
• An architecture that leverages
• provides the ability to quickly and iteratively identify
• the relevant subset of logs
• we needed insight into
• we embarked on implementing
• as a way to address this need
• Since most inter-process communication uses HTTP and gRPC
• with the trend for newer services to use gRPC to benefit from its binary protocol
• in which a single request results in calling a second tier of servers
• The smaller squares beneath a server indicate individual operations
• What is unique in our approach is that
• all the requests relevant to a given playback session
• Continuing with the above example
• might be why a given session did or did not receive 4K video
• Our goal is to increase
• This analysis does some basic clustering
• we display this informaation along with the request trace
• we've noticed that other similar devices have had the same error
• helpful in finding the root cause
• In summary
• the key learning from our effort are that
• trying multiple request traces into a logical concept
• that may involve multiple systems
• we are able to take this a step further
• observability also involves analysis of metrics
• Because having users examine many logs
• which enables our users to quickly see macro level error trends
• if a given metric exceeds a defined threshold
• a user can define metrics derived from matching logs
• by comparing the current trend with a baseline trend
• We are also working on correlating metrics for related microservices.
• For observability data that is always retrieved by
• When data needs to be queried by one or more fields
• We use Elasticsearch since multiple fields within a given record can be easilty indexed
• such as up to the last week
• since most of our users troubleshoot recent issues
• have their own advantages and disadvantages in terms of cost,, latency, and queryability
• In contrast
• since Elasticsearch is primarily optimized for free text search
• its indexing overhead during writes will demand
• as writes rate increases
• indexing time bacame long enough that either
• but we defined data retrieval for the three unique keys
• Regarding storage and computing cost
• can be kept in a single file
• Elasticserach is most likely the next more expensive option
• it encourages replicating each record to more that one replica
• in a view customized for
• whereas other users want to see
• via persisted user preferences
• Loading those logs for a customer account increases the paging loading time
• Examining a given log type may require domain expertise that not all users may have
• required to effectively diagnose problems
• by joining identifires with the data they refer to
• that surface relevant information are critical
• have empowered many teams
• quickly troubleshoot issues across various facets
• As Netflix evolves and grows
• we want to continue to provide our engineers with the ability to innovate rapidly
• some of our ongoing efforts