System design training

[jiaqing-tee]

System Design Training

Playlist: https://www.youtube.com/playlist?list=PLLTou95Km32QtvOHQpZqOggppmXN-qj2Z

[jiaqing-tee]

Latency Numbers

Title: Latency Numbers Programmer Should Know: Crash Course System Design #1 Link: https://youtu.be/FqR5vESuKe0

Takeaways:

Latency numbers for 2020s

• 1ns: Accessing CPU registers, CPU clock cycle
• 1 - 10ns: L1 / L2 cache access, branch mispredict
• 10 - 100ns: L3 cache access, Apple M1 referencing main memory
• 100 - 1000ns: System call, MD5 hash
• 1 - 10microsec: Context switching, memory copy of 64KB
• 10 - 100microsec: Processing a http request, sequential reading, read an 8K page on SSD
• 100 - 1000microsec: SSD write latency, intra-zone networking round trip, memcache / redis get operation
• 1 - 10ms: Intra-zone networking latency, HDD seek time
• 10 - 100ms: Network round trip between US East and West Coast or US East Coast and Europe, read 1GB from main memory
• 100 - 1000ms: bcrypt a password, TLS handshake, network round trip between US-west and Singapore, sequential read 1GB from SSD
• 1s: Transfer 1GB over the network within the same region

[jiaqing-tee]

CI / CD

Title: CI/CD In 5 Minutes | Is It Worth The Hassle: Crash Course System Design #2 Link: https://youtu.be/42UP1fxi2SY

Takeaways:

• From code commit to deployment
• Eliminates manual human intervention
• Building, testing, and deploying

Continuous Integration (CI)

• Practice of using automation to enable teams to merge code into shared repository early and often
• Commit triggers automated workflow on CI server to make sure commit is safe to merge into main branch
• Relies on good tests
• Uses source code management system (Github)

Continuous Deployment (CD)

• Practice not as common, practice on basic types of systems
• Systems usually stateless like API or web server tiers
• Good production monitoring --> systems deployed continuously with minimal risks
• Stateless --> rollbacks usually quick and harmless
• Wrap new features with feature flags --> quickly shut off broken feature without full rollback
• Canary development --> deploys new production to subset of power users and staff who are interested with new features to catch problems before wide deployment

[jiaqing-tee]

REST API

Title: What Is REST API? Examples And How To Use It: Crash Course System Design #3 Link: https://youtu.be/-mN3VyJuCjM

Takeaways:

• REST --> REpresentational State Transfer
• Resources group by noun not verb (use products, not getAllProducts)
• Huge amount of data --> use pagination scheme (limit and offset)
• Use REST API versioning

[jiaqing-tee]

DNS

Title: Everything You Need to Know About DNS: Crash Course System Design #4 Link: https://youtu.be/27r4Bzuj5NQ

Takeaways:

• Gets IP address with domain name
• DNS Resolver: ISP / Cloudflare (1.1.1.1) / Google (8.8.8.8)
• Authoritative nameservers: www.google.com / wikipedia.org
• Top Level Domain (TLD) nameservers: .com / .org / .sg
• Root nameservers: .

DNS query flow

• Check browser cache
• Check OS call cache
• DNS resolver check cache
• Ask root nameserver . which respond with TLD nameservers
• Ask TLD nameserver whcih respond authoritative nameservers
• DNS resolver receives IP address
• OS receives IP address
• Browser receives IP address

DNS propagation

DNS propagation slow due to TTL on DNS record

Solution:

1. Shorten TTL (to maybe 60s) --> Allow actual record update to take effect
2. Leave server running on old IP address for a while. Decommission after traffic dies down.

[jiaqing-tee]

Memory & Storage Systems

Title: 10+ Key Memory & Storage Systems: Crash Course System Design #5 Link: https://youtu.be/lX4CrbXMsNQ

Takeaways

Memory

RAM

• Random Access Memory (RAM): Stores memory temporarily while computer is running
• Static Random Access Memory (SRAM): Used in high speed application like CPU cache
• Dynamic Random Access Memory (DRAM): Need to constantly refreshed to retain data (high maintenance)

• ROM

• Read-Only Memory (ROM): Retains data even when power is off (eg. firmware and BIOS)
• Firmware: Software in ROM that controls how hardware devices communicate with each other
• Basic Input / Output (BIOS): First software that run when computer power up

Storage

• Hard Disk Drive (HDD): Data on spinning magnetic disk --> large storage capacities at low price
• Solid State Drives (SSD): Use NAND-based flash memory --> faster data access, reduced power consumption, increased durability, high price
• Non-Volatile Memory Express (NVMe): High performance interface for SSDs that connects directly to CPU via PCIe lanes --> allow lower latency, faster data transfer rates than SATA-based SSDs
• Flash Drive / USB Drive / Thumbdrive: Plug and play devices with any USB port to transfer files between computer
• SD Card: Found in cameras and smartphones with different size (SD, micro SD, mini SD)

[jiaqing-tee]

Password Storage in Database

Title: System Design: How to store passwords in the database? Link: https://youtu.be/zt8Cocdy15c

Takeaways

• Don't store password in plain text
• Use modern hashing algorithm
• MD5 and SHA-1 are fast but less secure
• Salt the password: unique randomly generated string added to each password (makes each hash unique)

[jiaqing-tee]

Full-Stack "Burger"

Title: The Most Beloved Burger for Developers Link: https://youtu.be/7swoLEqABhQ

Takeaways

Scenario: Early stage startup, limited resources, prefer maximum flexibility

• Containerization: Nothing --> Max flexibility will go serverless functions
• Architectural patterns: Serverless --> Scaling flexibility, pay for resources used, enhance developer productivity
• CI/CD Tools: Github Actions --> No need CI/Cd at early stage
• APIs: REST --> Standard between client and server
• Version Control System (VCS): Github --> Sufficient
• Caching: CDN and Browser cache --> Most important to improve user experience
• Frameworks: Next.js (frontend) and Express.js (backend)
• Programming Language: Typescript
• Testing: Unit testing
• Database: Firebase for simple data, Aurora Serverless (complicated data model)

[jiaqing-tee]

Open Systems Interconnect (OSI) Model

Title: What is OSI Model | Real World Examples Link: https://youtu.be/0y6FtKsg6J4

Takeaways

• Theoretical framework for networking, splits network communication into 7 abstraction layer

1. Physical layer: Transmit raw bits of data across physical connection
2. Data link layer: Organize raw bits into frames and ensure frames are delivered to correct destination
3. Network layer: Route data frames across different networks
4. Transport layer: Handles end-to-end communication between 2 nodes
5. Session layer
6. Presentation layer
7. Application layer

[jiaqing-tee]

CAP Theorem

Title: CAP Theorem Simplified Link: https://youtu.be/BHqjEjzAicA

Takeaways

Concept in computer science explaining trade-offs between:

• Consistency: All nodes have consistent view of data

• Availability: Ability to respond to users at all times

• Partition tolerance: Ability to continue operating even if there is a network partition

• Network partition happens when nodes in a distributed system are unable to communicate with each other due to network failures.

• If network partition happened, need to choose between Consistency or Availability

[jiaqing-tee]

Microservices

Title: What Are Microservices Really All About? (And When Not To Use It) Link: https://youtu.be/lTAcCNbJ7KE

Takeaways

• Microservices are loosely coupled
• Can be independently deployed
• More flexibility to scale individual microservices independently
• Each service handles a dedicates function inside a large-scale application
• Breaking up database: database cannot maintain foreign key relationships and enforce referential integrity. Maintaining data integrity moved to application layer

API gateway

• API gateway handles incoming requests and routes them to relevant microservices
• API gateway relies on identity provider service to handle authentication and authorization of request coming through API gateway
• API gateway consults service registry & discovery to locate service for routing incoming request

Communication

• Microservices talk to 1 another over a combination of remote procedure calls (RPC) / event streaming / message brokers
• RPC (eg gRPC) provides faster response. Larger blast radius / impact when service goes down.
• Event streaming provides better isolation between services but take longer to process

When to use microservices

• For large teams, enables team independence
• Each domain independently maintained by dedicated team
• Each service could be independently designed, deployed, and scaled

[jiaqing-tee]

Proxy & Reverse Proxy

Title: Proxy vs Reverse Proxy (Real-world Examples) Link: https://youtu.be/4NB0NDtOwIQ

Takeaway

Forward proxy

• Server that sits between a client machines and internet
• Protects client's online identity
• Used to bypass browsing restrictions
• Used to block access to certain content (by applying filtering rules)
• Transport proxy works with layer 4 switches to redirect traffic to proxy automatically

Reverse proxy

• Sits between internet and the web servers
• Intercepts requests from clients and talks to web server on behalf of clients
• Used to protect a website: Website IP addresses hidden behind reverse proxy and not revealed to clients, harder to target DDoS attack
• Used for load balancing
• Reverse proxy are placed around the world (eg cloudflare): closer to users
• Caches static content
• Handles SSL encryption: SSL handshake is computationally expensive

[jiaqing-tee]

API Gateway

Title: What is API Gateway? Link: https://youtu.be/6ULyxuHKxg8

Takeaways

• Single point of entry to clients of application
• Sits between clients and backend services of application
• Error handling: Should track errors
• Circuit break: Provide circuit-breaking functionality to protect services from overloading
• Logging, monitoring, analytics services: Operational observability

Functions

• Authentication and security policy enforcement
• Load balancing and circuit breaking
• Protocol translation and service discovery
• Monitoring, logging, analytics, and billing
• Caching

API gateway flow

1. HTTP request: Client sends a request to API gateway
2. Parameter validation: Validates HTTP request
3. Allow-list / Deny-list: Checks caller's IP address and other HTTP headers, and perform basic rate limit checks against IP address and HTTP headers
4. Authentication / Authorization: Passes request to identity provider for authentication and authorization
5. Rate limit: Higher level rate-limit applied against authenticated session. Over limit, request rejected
6. Dynamic routing
7. Service discovery: Locates appropriate backend service and handle request path matching
8. Protocol Conversion: Transform request into appropriate protocol and sends transformed request to backend service (eg protocol: gRPC)

[jiaqing-tee]

Cloud Native

Title: But What Is Cloud Native Really All About? Link: https://youtu.be/p-88GN1WVs8

Takeaways

• No clear definitions, means different things to different people
• Blueprint for building web-scale applications on cloud that are more available and scalable
• Promise increased agility to ship new features without compromising availability

Cloud computing

• Running applications on computing resources managed by cloud providers
• Fast to provision computing resources, scaling is effortless
• Running application on cloud not equal to Cloud Native

Cloud Native Pillars

1. Microservices: Break functionalities of large application into smaller microservices
2. Containers: Applications packaged in containers. Container orchestration platform: Kubernetes - Oversees and controls where container run, detect and repair failures, and balances load between microservices
3. DevOps: Emphasizes collaboration, communication, and automation between development and operations team to deliver cloud-native applications quickly and reliably. Use CI/CD to automate software development and deployment process
4. Cloud Native Open Standards: Critical components standardized and best practices are widely available.

[jiaqing-tee]

Content Delivery Network (CDN)

Title: What Is A CDN? How Does It Work? Link: https://youtu.be/RI9np1LWzqw

Takeaway

• Originally developed to speed up delivery of static HTML content for users worldwide
• Now used whenever HTTP traffic is served
• Brings content closer to user: Improves performance of web service
• Servers deployed at hundreds of locations worldwide
• Server locations called Point of Presence (PoP): commonly called edge server
• Edge server acts as reverse proxy with huge content cache
• Reduce load and bandwidth requirements of origin server cluster
• Modern CDN transform static content into more optimized formats
• TLS connections terminate at edge server --> reduces latency for user to establish encrypted TCP connection
• Different CDN use different technologies to direct user's request to closest PoP

2 Common CDN technologies to direct request

1. DNS-based routing
   • PoP has its own IP address
   • When user look up IP address for CDN, DNS returns closest PoP IP address
2. Anycast
   • All PoPs share same IP address
   • Request comes into Anycast network, network sends request to PoP closest to requester

CDN Benefits

1. Security
   • Provide effective DDoS protection
   • Effective with CDN built on Anycast network: Diffuse attack traffic over huge number of servers
2. Availability
   • By nature, highly distributed
   • Able to withstand hardware failures

[jiaqing-tee]

Cache Systems

Title: Cache Systems Every Developer Should Know Link: https://youtu.be/dGAgxozNWFE

Takeaways

• Used to enhance system performance and reduce response time
• Web browsers: Cache HTTP responses to enable faster retrieval of data
• CDN: Improve delivery of static content - images, videos, other web assets
• Load balancer: Can cache resources to reduce load on backend servers
• Messaging infrastructure: Can cache massive messages on disk, allow consumers to retrieve messages at their own pace (eg Kafka)
• Distributed caches: Store key value pairs in memory which provides high read / write performance compared to traditional databases (eg Redis)
• Full text search engines: Index data for document search and log search which provide quick and efficient access to specific data (eg Elastic search)
• Relational database:
  • Data written to a write-ahead log (WAL) before indexed in a B-tree
  • Buffer pool: Used to cache query results
  • Materialized views: Precompute query results for faster performance
  • Transaction log: Records all transactions and updates to database
  • Replication log: Tracks replication state in database cluster

Computer hardware cache

1. L1 cache
   • Smallest and fastest cache, typically integrated into CPU
   • Stores frequently accessed data and instructions
2. L2 cache
   • Larger but slower than L1 cache
   • Typically located on CPU or separate chip
3. L3 cache
   • Larger and slower than L2 cache
   • Often shared between multiple CPU cores
4. Translation Lookaside Buffer (TLB)
   • Stores recently used virtual to physical address translations
   • Used by CPU to translate virtual memory addresses to physical memory address
   • Reduce time needed to access data from memory

Operating System

1. Page cache
   • Managed by OS and resides in main memory
   • Store recently used disk blocks in memory
2. Inode cache
   • Used to speed up file system operations by reducing disk accesses required to access files and directories
3. File system cache

[jiaqing-tee]

Redis

Title: Top 5 Redis Use Cases Link: https://youtu.be/a4yX7RUgTxI

Takeaway

• In memory data structure store, most commonly used as cache
• Supports many data structures: strings, hashes, lists, sets, sorted sets
• Known for speed
• Versatile

Use Cases for Redis

1. Cache

   • Cache object to speed up web applications
   • Stores frequently request data in memory
   • Allow web servers to return frequently accessed data quickly
   • Reduce load on database and improves response time for application
   • Cache is distributed among cluster of Redis servers
   • Sharding: Common technique to distribute caching load evenly across cluster
   • When using Redis as distributed cache, to consider
     • Setting correct TTL
     • Handling thundering herd on cold start

2. Session

   • Share session data among stateless servers
   • Session data stored in Redis with unique session ID returned to client as cookie
   • When request is made, session ID is included in the request
   • Stateless web server retrieves session data from Redis using session ID
   • Session data stored will be lost if Redis server restarts!
   • Redis provides persistence options like snapshots and Append-Only File (AOF) which allow session data to be saved to disk and reloaded into memory in the event of restart
   • Such options takes too long to load on restart to be practical
   • As such, replication is used instead
   • Data replicated to a backup instance
   • Event of crash of main instance, backup is used to take over traffic

3. Distributed Lock

   • Used when multiple nodes in application need to coordinate access to shared resource
   • Used with its atomic commands like SET if Not eXists (SETNX): allow caller to set key if not exist
   • Simple implementation but not completely fault tolerant
   • Many production libraries providing high quality distributed lock implementation
   • Eg
     • Client 1 tries to acquire lock by setting key with unique value and time out using SETNX command
     • If key not set
       • SETNX command returns 1: indicates lock acquired by Client 1
       • Client 1 finishes work and release lock by deleting key
     • If key set
       • SETNX command returns 0: indicates lock is held by another client
       • Client 1 waits and retries the SETNX operations until lock is released by other client

4. Rate Limiter

   • By using increment command on some counters and setting expiration times on those counters
   • Basic rate limiting algorithm
     • Request IP or user ID used as key
     • Number of request for key incremented using INCR command in Redis
     • Current count compared to allowed rate limit
       • Count within rate limit: Request processed
       • Count not within rate limit: Request rejected
     • Keys set to expire after specific time window which resets the counts for next time window
     • Sophisticated rate limiters like leaky bucket algorithm can be implemented

5. Rank / Leaderboard

   • Fundamental data structure: Sorted sets - collection of unique elements with score associated and elements are sorted by score
   • Allows for quick retrieval of elements by score in logarithmic time

[jiaqing-tee]

Kafka

Title: System Design: Why is Kafka fast? Link: https://youtu.be/UNUz1-msbOM

Takeaways

• Optimized for high throughput AKA ability to move lot of data efficiently
• Design to move large records in short duration

Design Decision for Kafka Performance

1. Sequential I/O
   • Misconception: Disk access is slow compared to memory access. Its dependent on data access patterns
   • Access patterns
     1. Random
        • HDD takes time to physically move arm to different location of magnetic disk - makes access slow
     2. Sequential
        • SSD doesn't need arm to jump - much faster to read and write blocks of data
   • Uses sequential access pattern: Append-Only Log as primary data structure
   • Append-only log adds new data to end of file
2. Zero copy principle
   • Move lots of data, important to eliminate excess copy when moving pages of data between disk and network
   • Uses zero copy principle
   • Modern unix OS are optimized to transfer data from disk to network without copying data excessively
   • Zero copy not used
     1. Data loaded from disk to OS cache
     2. Data copied from OS cache into Kafka application
     3. Data copied from Kafka to socket buffer
     4. Data copied from socket buffer to Network Interface Card (NIC) buffer
     5. Data is sent over network to consumer
        • Uses 4 copies and 2 system calls
   • Zero copy used
     1. Data loaded from disk to OS cache
     2. Kafka use system call sendfile() to tell OS to directly copy data from OS cache to NIC buffer
        • Only copy from OS cache into NIC buffer
        • Modern NIC copy with Direct Memory Access (DMA)
        • When DMA used, CPU not involved --> more efficient

[jiaqing-tee]

GraphQL

Title: What Is GraphQL? REST vs. GraphQL Link: https://youtu.be/yWzKJPw_VzM

Takeaways

• Query language API developed by Meta
• Provides schema of data in API --> gives client power to ask for exactly what is needed
• Sit between clients and backend services
• Able to aggregate multiple resource requests into single query
• Supports mutations(addNode) and subscriptions (newNode)
• Mutations: applying data modifications to resources
• Subscriptions: clients to receive notifications on data modification

GraphQL vs REST

• Both send HTTP requests and receive HTTP responses
• REST centers around resources identified in URL, might need further REST calls
• GraphQL define properties and property data type
• GraphQL can specify exact resources and fields needed in 1 query

GraphQL Drawbacks

• REST don't need special libraries
• GraphQL requires heavier tooling support on both client and server side
• Requires upfront investment, might not be worth for simple CRUD APIs
• Difficult to cache. Possible but it's extra work and not trivial!

[jiaqing-tee]

Single Sign-on (SSO)

Title: What Is Single Sign-on (SSO)? How It Works Link: https://youtu.be/O1cRJWYF-g4?list=PLLTou95Km32QtvOHQpZqOggppmXN-qj2Z

Takeaway

• Authentication scheme which enable user to securely access multiple application and services using single ID
• Able to access many apps without having to log in each time
• Built on concept called federated identity which enables sharing of identity information across trusted but independent systems

2 Common Protocols

1. Security Assertion Markup Language (SAML)
   • XML based open stand for exchanging identity information between services
2. OpenID Connect
   • Uses JSON Web Token (JWT) to share identity information between services

SSO Login Flow (SAML)

1. Service provider server detects user is from work domain, returns a SAML authentication request to the browser
2. Browser redirects user to Identity Provider specified in SAML authentication request
3. Identity provider shows login page for users to enter login credentials
4. After authentication, identity provider generates a SAML response to return to browser AKA SAML assertion - cryptograhically signed XML document containing information about user and what user can access
5. Browser forwards signed SAML assertion to service provider
6. Service provider validates assertion is signed by identity provider (done using public key cryptography)
7. Service provider returns the protected resource to browser based on what user is allowed to access specified in SAML assertion

Attempts to access another service provider

1. New service provider server detects user is from work domain, returns a SAML authentication request to the browser
2. Browser redirects user to Identity Provider specified in SAML authentication request
3. Since user already logged in with Identity Provider, skips login process
4. Generate SAML assertion for new service provider indicating what user can access
5. SAML assertion returned to browser and forwarded to new service provider
6. New service provider validates the signed assertion and grants user access accordingly

OpenID Flow

• Similar to SAML but instead of passing signed XML documents, OpenID Connect passes around JWT - signed JSON document

Which Protocols to use?

• Both implementation are secured
• Depends on which protocol is easier to integrate with the application
• New web application integrating with popular OpenID Connect platforms like Google, Facebook, and Github - Use OpenID Connect

[jiaqing-tee]

SQL Queries Optimization

Title: Secret To Optimizing SQL Queries - Understand The SQL Execution Order Link: https://youtu.be/BHwzDmr6d7s

Takeaways

• Using index on join columns significantly improve performance of join operation
• Use Search ARGument ABLE (SARGABLE) query: use indexes for faster execution
  • Avoid using functions or calculations on indexed columns in WHERE clause
  • Use direct comparisons where possible, instead of wrapping the column in a function
  • If we need to use a function on a column, create a computer column or function-based index
• Optimizing SELECT clauses by using covering indexes which includes all the columns needed in the query found in SELECT, WHERE, JOIN clauses
  • Enables database engine to retrieve data directly from the index which speeds up query execution
  • Covering indexes can reduce the need for covering I/O and minimize operation overhead
• When using ORDER BY, consider using smaller result set with filtering and pagination
• Avoid sorting the entire dataset, use appropriate indexes to speed up sorting