Most engineers who work on large projects will have noticed that the time you need to accomplish anything in a large codebase is incredibly high compared to the same task performed in a small project.
There are a number of potential reasons for slow velocity, but I think technical debt is the most underestimated one.
It’s safe to say that tech debt is the black hole of engineering time and a huge source of developers’ unhappiness. Let’s dive deeper into this phenomenon and define some of its aspects, but more importantly — define what can be done about it.
What is technical debt?
Technical debt (also known as design debt or code debt) is a concept in software development that reflects the implied cost of additional rework caused by choosing an easy solution now instead of using a better approach that would take longer.
More on Wikipedia
Is it always bad?
There is only one good reason for keeping technical debt — when a codebase doesn’t need maintenance. (That may be the case for one-off projects that are completed and never touched again, or legacy projects that are unlikely to be revived or updated.)
In all other cases, debt is tragic for all participants and needs to be taken seriously. Legacy code will have a huge impact on the development of further features, bugfixing, developers’ happiness and in the worst case scenario it can lead to a full development stop and a major rewrite. This happened already to many big companies like eBay and many others didn’t survive it.
Let me introduce Sally
Sally is an engineer working in a team as web frontend specialist. Her team has mobile developers, designers, backend developers and a product manager. Her team is one of many other teams, focusing on one particular feature in a big product. Every few weeks she discusses features and improvements and agrees to complete them.
Sometimes, during the sprint, she finds technical limitations, specific to the web platform she works on, where the feature design did not consider them during the design phase. Sometimes she recognizes a UX mistake that crept in during the design phase, which only becomes obvious once you see how the app works in real life.
Sometimes there’s a question that needs to be answered by the colleague who wrote the code, because the code doesn’t explain why it does what it does and the colleague is on vacation.
This time though is different: She discovered that in order to implement the feature properly in the way it is designed, she needs to refactor a number of modules used by other features. Sally is in trouble. She already committed to accomplishing the feature in two weeks and one week is already gone.
Sally knows her manager doesn’t have time to understand the technical side of the problem since he operates on a different level of abstraction. She feels obligated to find a solution that doesn’t require additional weeks of work since refactoring other features is not part of her current sprint and the product manager cannot really value something he doesn’t touch directly. He cares much more about getting the feature implemented on time as estimated.
So what can Sally do? She finds a workaround to implement a feature. She does it in a way that is neither clear nor good. She knows this will clearly create a headache for her co-workers, so she creates a ticket to fix this technical debt later. During a code review, her colleagues understand the problem, since Sally said this is a very urgent feature and she needs to have it done tomorrow — otherwise her product manager will be very unhappy. Her understanding colleagues approve the pull request and the code gets merged.
Sally continues to increase technical debt while product management fails to see the importance of reducing technical debt, since management doesn’t have to deal with code and engineers neglect to explain the problems due to their complexity.
Velocity decrease
Many people think of technical debt as a single big problem — which it sometimes is. Most of the time though, it’s not a single problem, it’s hundreds of small problems.
Readability and clarity of code have a very strong impact on development speed since they make it easier to understand what code is supposed to do — the intent. For this reason, it is harder to anticipate the side effects unclear code may create and it is harder to change and test.
Indeed, the ratio of time spent reading versus writing is well over 10 to 1. We are constantly reading old code as part of the effort to write new code. …[Therefore,] making it easy to read makes it easier to write.
— Robert C. Martin, Clean Code: A Handbook of Agile Software Craftsmanship
Bugs increase
The ease of understanding the code usually correlates with the ease of fixing bugs. If you are not careful with this, you will generate huge amount of bugs even before releasing the feature.
The cleaner the code is, the easier it is to avoid adding bugs.
Tests are very important to have, but they will never cover everything, even if the test coverage reports 100%. The code in an actively maintained project is a moving target, it changes all the time and so do the use cases for the code. Public functions may be used more and more over time, which adds more potential use cases that were not anticipated by the initial author.
Dirt attracts dirt
The more debt there is in the code base, the less desire people have to fix it — and the more debt will be created. Imagine walking on a perfectly clean street and you just finished your last cigarette and are looking to drop the stub. Would you do it on a nice, clean street? Probably not. Now imagine the same scenario, but the street is dirty, trash is everywhere — now it’s easy, right? This is also known as the “broken windows theory”. The same goes for the codebase. Keep it clean, or you’ll eventually find yourself sinking in tech debt.
Always put people first
We say so often “put people first” when you build an organization or society, but what would that mean in software engineering?
I think it means we need to understand what makes engineers unhappy about the code quality. There are many factors for sure, but burning brain cycles to figure out how the code works (because it’s hard to understand or poorly structured) is one of the primary causes. Having tests is also quite important in a large organization, because they help people understand the intent and help to ensure previous use cases don’t break when code is changed.
https://neo4j.com/blog/other-graph-database-technologies/
Code is a graph database
We could look at the codebase like at a graph database. Each dot represents a an isolated system and has a different purpose, performance characteristics, series of dependencies and dependants. Each time we change it, we always need to think of whether any of these characteristics also changed in a way that would require refactoring.
When you’re thinking about adding a component, for example, you need to check whether an existing component could be used, since it would otherwise create a needless duplication.
In another case, you might think you could use an existing component which looks very similar to what you need, but the use case for which it was designed is different — its characteristics have been optimized for a different purpose. In this case, you’d be breaking the separation of concerns, which may lead to numerous other issues. For example, if you need to make a change in that component, you will be adapting it for a second use case and by doing so you will make it hard to maintain.
Another consideration is performance. Something that is built to run once is usually designed differently than something that is written to be run millions of times in a fraction of second.
It’s very hard work to keep this graph clean and correct. It’s a constant challenge that requires an investment of time and smart minds. The bigger the graph, the more it takes to keep it clean and functional.
Code is a dialog
Source code is not just a dialog with a computer, it’s also a dialog with a colleague or your future self. If you write code that’s very clever, you may be asking the next reader to put a lot of brain cycles into understanding it. If you write code that doesn’t do what it says it does, you send the reader through a labyrinth of thoughts trying to justify it. It is important to invest enough time and make code easy to read because the next reader will need much more time to understand it than the author — since the author already spent the time thinking about it while implementing it.
Every time you save your time at a cost of code quality, you take even more time from the next person who needs to maintain it.
Why is tech debt so hard to get rid of?
There are many reasons obviously, but if I had to name just 3, it would be those:
Deadlines
Deadlines are the most popular reason for not doing something right. Quality costs time, everyone knows that. Things left for later are usually not done at all until it hurts. Even the work of putting off dozens of things for later is actually a lot of work since you need to carefully document what needs to be done later. Also don’t confuse external deadlines with your own promises, which are just estimates!
Processes
When processes are optimized for and encourage new features only, every person on the team (even engineers) will start seeing this as the most important thing — even if the software is maintained by many people and technical debt is at least as important. Placing emphasis that way makes everyone question the time spent on anything but new features. Developers shouldn’t need to put in extra effort to get the time to do the right thing.
Values
Without a culture that celebrates the actual quality of engineering work (not just the parts that are visible to the user), engineers can end up with a “quick and dirty” mindset. Why would they do otherwise? Give them a reason. Find ways to analyze quality, track technical debt and the quality of architectural decisions to make them visible and celebrate them!
What can we do?
Here are a few ideas on how to improve and avoid getting mired in debt:
Make sure time for learning, pair programming, experimenting, teaching and refactoring is explicitly defined and considered when you estimate. These things can easily take 50% of your time budget.
Create a culture where bad technical choices are not equal to bad engineers. It is often the complexity and time pressure that creates bad decisions. Often enough, good choices can be only made during refactoring.
Make technical debt visible for everyone. When you write a TODO in code:
– Always create a ticket in the issue tracker.
– Add this ticket number to the TODO item in the code.
– Add the “tech-debt” label to the ticket.
Communicate clearly what solving the debt will bring, but even more clearly what problems will be created if the issue is not fixed now.
Make the complexity of debt visible. Either use story points or make small, homogenous subtasks so that effort can be estimated by counting them.
Make a developer-friendly codebase and reduction of technical debt part of the product’s success metrics. Celebrate the quality of inner workings.
Software is hard. Engineers have to deal with huge levels of complexity and naturally sometimes they fail. My role as an architect requires me to think critically and my ultimate goal is to evangelize technical quality in a way that leads to a reduction of maintenance cost over the long term.
I would like to encourage engineers to always think about the people who need to work with their code. Put them first, not yourself, because engineering is a team effort.
Also, I encourage managers to be sensitive towards the problems engineers sometimes fail to explain well, to create a communication culture where delays and plan changes due to discovered difficulties are normal.
If we make sure code quality and architecture is part of the overall product success story, everything will naturally begin to work better.
https://ebaytech.berlin/a-city-of-technical-debt-58568747b12
Most engineers who work on large projects will have noticed that the time you need to accomplish anything in a large codebase is incredibly high compared to the same task performed in a small project. There are a number of potential reasons for slow velocity, but I think technical debt is the most underestimated one. It’s safe to say that tech debt is the black hole of engineering time and a huge source of developers’ unhappiness. Let’s dive deeper into this phenomenon and define some of its aspects, but more importantly — define what can be done about it. What is technical debt? Technical debt (also known as design debt or code debt) is a concept in software development that reflects the implied cost of additional rework caused by choosing an easy solution now instead of using a better approach that would take longer. More on Wikipedia Is it always bad? There is only one good reason for keeping technical debt — when a codebase doesn’t need maintenance. (That may be the case for one-off projects that are completed and never touched again, or legacy projects that are unlikely to be revived or updated.) In all other cases, debt is tragic for all participants and needs to be taken seriously. Legacy code will have a huge impact on the development of further features, bugfixing, developers’ happiness and in the worst case scenario it can lead to a full development stop and a major rewrite. This happened already to many big companies like eBay and many others didn’t survive it. Let me introduce Sally Sally is an engineer working in a team as web frontend specialist. Her team has mobile developers, designers, backend developers and a product manager. Her team is one of many other teams, focusing on one particular feature in a big product. Every few weeks she discusses features and improvements and agrees to complete them. Sometimes, during the sprint, she finds technical limitations, specific to the web platform she works on, where the feature design did not consider them during the design phase. Sometimes she recognizes a UX mistake that crept in during the design phase, which only becomes obvious once you see how the app works in real life. Sometimes there’s a question that needs to be answered by the colleague who wrote the code, because the code doesn’t explain why it does what it does and the colleague is on vacation. This time though is different: She discovered that in order to implement the feature properly in the way it is designed, she needs to refactor a number of modules used by other features. Sally is in trouble. She already committed to accomplishing the feature in two weeks and one week is already gone. Sally knows her manager doesn’t have time to understand the technical side of the problem since he operates on a different level of abstraction. She feels obligated to find a solution that doesn’t require additional weeks of work since refactoring other features is not part of her current sprint and the product manager cannot really value something he doesn’t touch directly. He cares much more about getting the feature implemented on time as estimated. So what can Sally do? She finds a workaround to implement a feature. She does it in a way that is neither clear nor good. She knows this will clearly create a headache for her co-workers, so she creates a ticket to fix this technical debt later. During a code review, her colleagues understand the problem, since Sally said this is a very urgent feature and she needs to have it done tomorrow — otherwise her product manager will be very unhappy. Her understanding colleagues approve the pull request and the code gets merged. Sally continues to increase technical debt while product management fails to see the importance of reducing technical debt, since management doesn’t have to deal with code and engineers neglect to explain the problems due to their complexity.
https://www.researchgate.net/publication/310461939 Why is it important to reduce the debt?
https://neo4j.com/blog/other-graph-database-technologies/ Code is a graph database We could look at the codebase like at a graph database. Each dot represents a an isolated system and has a different purpose, performance characteristics, series of dependencies and dependants. Each time we change it, we always need to think of whether any of these characteristics also changed in a way that would require refactoring. When you’re thinking about adding a component, for example, you need to check whether an existing component could be used, since it would otherwise create a needless duplication. In another case, you might think you could use an existing component which looks very similar to what you need, but the use case for which it was designed is different — its characteristics have been optimized for a different purpose. In this case, you’d be breaking the separation of concerns, which may lead to numerous other issues. For example, if you need to make a change in that component, you will be adapting it for a second use case and by doing so you will make it hard to maintain. Another consideration is performance. Something that is built to run once is usually designed differently than something that is written to be run millions of times in a fraction of second. It’s very hard work to keep this graph clean and correct. It’s a constant challenge that requires an investment of time and smart minds. The bigger the graph, the more it takes to keep it clean and functional. Code is a dialog Source code is not just a dialog with a computer, it’s also a dialog with a colleague or your future self. If you write code that’s very clever, you may be asking the next reader to put a lot of brain cycles into understanding it. If you write code that doesn’t do what it says it does, you send the reader through a labyrinth of thoughts trying to justify it. It is important to invest enough time and make code easy to read because the next reader will need much more time to understand it than the author — since the author already spent the time thinking about it while implementing it. Every time you save your time at a cost of code quality, you take even more time from the next person who needs to maintain it. Why is tech debt so hard to get rid of? There are many reasons obviously, but if I had to name just 3, it would be those: Deadlines Deadlines are the most popular reason for not doing something right. Quality costs time, everyone knows that. Things left for later are usually not done at all until it hurts. Even the work of putting off dozens of things for later is actually a lot of work since you need to carefully document what needs to be done later. Also don’t confuse external deadlines with your own promises, which are just estimates! Processes When processes are optimized for and encourage new features only, every person on the team (even engineers) will start seeing this as the most important thing — even if the software is maintained by many people and technical debt is at least as important. Placing emphasis that way makes everyone question the time spent on anything but new features. Developers shouldn’t need to put in extra effort to get the time to do the right thing. Values Without a culture that celebrates the actual quality of engineering work (not just the parts that are visible to the user), engineers can end up with a “quick and dirty” mindset. Why would they do otherwise? Give them a reason. Find ways to analyze quality, track technical debt and the quality of architectural decisions to make them visible and celebrate them! What can we do? Here are a few ideas on how to improve and avoid getting mired in debt: Make sure time for learning, pair programming, experimenting, teaching and refactoring is explicitly defined and considered when you estimate. These things can easily take 50% of your time budget. Create a culture where bad technical choices are not equal to bad engineers. It is often the complexity and time pressure that creates bad decisions. Often enough, good choices can be only made during refactoring. Make technical debt visible for everyone. When you write a TODO in code: – Always create a ticket in the issue tracker. – Add this ticket number to the TODO item in the code. – Add the “tech-debt” label to the ticket. Communicate clearly what solving the debt will bring, but even more clearly what problems will be created if the issue is not fixed now. Make the complexity of debt visible. Either use story points or make small, homogenous subtasks so that effort can be estimated by counting them. Make a developer-friendly codebase and reduction of technical debt part of the product’s success metrics. Celebrate the quality of inner workings. Software is hard. Engineers have to deal with huge levels of complexity and naturally sometimes they fail. My role as an architect requires me to think critically and my ultimate goal is to evangelize technical quality in a way that leads to a reduction of maintenance cost over the long term. I would like to encourage engineers to always think about the people who need to work with their code. Put them first, not yourself, because engineering is a team effort. Also, I encourage managers to be sensitive towards the problems engineers sometimes fail to explain well, to create a communication culture where delays and plan changes due to discovered difficulties are normal. If we make sure code quality and architecture is part of the overall product success story, everything will naturally begin to work better.