More Systems Design & WPL Championship 2021

[r002]

Wednesday

Clarifying questions to ask about a feature request/new project spec:

0) What Plane of Abstraction are we working on? What's the altitude? Org chart paradigm?

• Are you the CTO scoping out a new imitative?
• Are you a Product Owner at the dept level?
• Are you the tech lead already on a specific team?
• What org chart paradigm are we operating in?

1) Understand more clearly what is being built. Project definition.

• Audience: Who is using it? Internal or external? Scale? Number of users? Availability requirements? Frequency of access?
  • Do the Amazon/Bezos "Write the press release first exercise."
• Security/privacy? Regulatory concerns?
• If it breaks, what are the consequences?

2) Are there technology constraints from on-high?

• 100% greenfield or are we appending to a legacy system? What's our purview?
• Does the company already have contracts with certain providers? Eg. We must use Avaya or Cisco?
• Are we limited to technology choices by what the rest of our stack looks like? Eg. We're already using MySQL everywhere so we need to stick to relational and can't use NoSQL?

3) Personnel concerns

• What does our team know? What do they want to learn? Do we actually have people who can and/or want to code/learn how to code this? If we ask them to work on this, are they excited about it? Or will they quit halfway through? Morale issues?

4) Political concerns

• What stakeholders have what mandates? Understanding the lay of the land. Eg. If we push on platform X, and over in department Y, they're pushing their new product on the roadmap on platform Z for the same quarter we're shooting for, how will the optics look? Must be aware of the overall political landscape and which direction the winds of change/power are blowing. Which coalition is ascendant? Which is on its way out?
• Technology matters, but coding and building a solution in a large org is categorically different than building some solo-weekend warrior project or hobby coding on your own for fun. In my humble experience, there usually exist other factors equal, if not more, important than the technology. One needs to at least be aware of the landscape.
  

Other/Fun:

Last night I was having a ton of trouble sleeping 😕 so I ended up wandering over to the ClassicTetris Twitch channel. This past weekend, they held the World Puzzle League Championships in Columbus! Omg, so epic! Sadly, I totally missed it. 😣 The final championship stream was amazing though! One of my dreams is to one day compete in this! 🥳🎉🎈

• https://www.twitch.tv/videos/1118341246?t=05h38m10s
  

References:

• https://www.cultofmac.com/102917/apple-ms-google-etc-imagined-as-fun-org-charts/

[r002]

Technology Notes

• https://github.com/donnemartin/system-design-primer
• https://deniseyu.io/2020/05/21/talking-about-software-at-scale.html
• https://neo4j.com/blog/acid-vs-base-consistency-models-explained/
• https://genius.com/albums/Heroku/Routergate
• https://redditblog.com/2017/04/13/how-we-built-rplace/
  

Data

• System of Record vs Source of Insight
  • https://stackoverflow.blog/2021/08/06/podcast-364-mark-porter-mongodb-database/
• NoSQL vs Relational
  • In 2021, at least, according to Mark Porter (CTO of Mongo)-- NoSQL is the absolute best. It can supposedly do everything that RDMS systems can. You can supposedly just enable schema enforcement and toggle different availability modes, including ACID transactions.
  • Having personally never used Mongo though, my general understanding is this. It might be outdated in 2021 though:
  • MySQL and Postgres is great when you've got a firm idea on the data schema. Classic examples are your usual Customer or Widget object. You know the fields. They're (to your best knowledge at least) not likely to change any time soon. And of ACID-- the one most salient here is immediate consistency. The moment you write to disk, you know that data is consistent (the same) everywhere.
    • Classic use case: Bank transactions-- anything low frequency; high importance
    • Cons: Slower, expensive, fail over over entire DBs take forever.
  • NoSQL, historically at least, has always been the goto for BASE-- eventual consistency. The classic use case here is a Tweet. You don't really care if everyone on the planet here immediately sees that same tweet.
    • Historically, you can go schema-less. You have no idea what you're building. The product is highly organic.
  • I've never used NoSQL for anything professionally so I can only write on what I've read and what I've read. But in my experience, I've really liked using NoSQL because I've been working on tiny/solo projects, my "requirements" are evolving all the time, and the DX dev experience is just fast. Like, it's incredibly easy for me to prototype.
  • Back when I used Postgres for my Heroku apps, I always had a bunch of ActiveRecord insert/update statements ready to go. Or I had to fight with that awful PgAdmin php web GUI. Using Firebase, OTOH, has been an absolute delight from a developer standpoint though.
  • NoSQL (Firebase, at least) downsides: Environments can't be easily split between QA/PROD. Import/export also hacky.
  • With NoSQL, I --for my own small projects-- end up writing tons of meta tables keeping track of various aggregations and metrics that SQL used to give me for free if I wrote clever queries! 😑
    

Queues

• Pro: A way of managing load in asynchronous fashion. Alternative would be requests all slamming on a server/set of servers all at the same time which is not what we want.
• Cons: There's complexity about jobs stuck in the queue, how to flush the queue, if the queue is accurate. Etc. I haven't work with them personally, but I know at a high-level, entire products and companies have been built around this. To my knowledge, this is a solved problem but you do need to configure it correctly.
  

Health Checks

• At my previous job, we used keepalive "heartbeat" checks for instrumentation.
  

Bad Actors

• If we are designing an API, and client don't exponentially back off, we need to log their origins to block them outright. Or do that "Abnormal activity detected/CAPTCHA check" if it's a GUI endpoint for human interaction.
  • https://redditblog.com/2017/04/13/how-we-built-rplace/
    

Logging

• Good instrumentation-- gauges on the usual suspects: CPU, memory, non-200 response codes.
• Solid run books--when things start redlining, does L1 support actually know what to do?
• Is your PaaS actually doing what you think it's doing?
  • https://genius.com/albums/Heroku/Routergate