Modern Software Engineering (64%)

Congrats on adding Modern Software Engineering by David Farley to your bookshelf, I hope you enjoy it! It has an average of unknown/5 stars and 0 ratings on Google Books.

Book details (JSON)

```json { "title": "Modern Software Engineering", "authors": [ "David Farley" ], "publisher": "Addison-Wesley Professional", "publishedDate": "2022-01-19", "description": "Writing for students at all levels of experience, Farley illuminates durable principles at the heart of effective software development. He distills the discipline into two core exercises: first, learning and exploration, and second, managing complexity. For each, he defines principles that can help students improve everything from their mindset to the quality of their code, and describes approaches proven to promote success. Farley's ideas and techniques cohere into a unified, scientific, and foundational approach to solving practical software development problems within realistic economic constraints. This general, durable, and pervasive approach to software engineering can help students solve problems they haven't encountered yet, using today's technologies and tomorrow's. It offers students deeper insight into what they do every day, helping them create better software, faster, with more pleasure and personal fulfillment.", "image": "http://books.google.com/books/content?id=ZKxHzgEACAAJ&printsec=frontcover&img=1&zoom=1&source=gbs_api", "language": "en", "categories": [ "Engineering" ], "pageCount": 256, "isbn10": "0137314914", "isbn13": "9780137314911", "googleBooks": { "id": "ZKxHzgEACAAJ", "preview": "http://books.google.com/books?id=ZKxHzgEACAAJ&dq=intitle:Modern+Software+Engingeering&hl=&cd=1&source=gbs_api", "info": "http://books.google.com/books?id=ZKxHzgEACAAJ&dq=intitle:Modern+Software+Engingeering&hl=&source=gbs_api", "canonical": "https://books.google.com/books/about/Modern_Software_Engineering.html?hl=&id=ZKxHzgEACAAJ" } } ```

When you're finished with reading this book, just close this issue and I'll mark it as completed. Best of luck! 👍

Page 29: Organizations often complain about the quality or efficiency of software development teams. Software development is not an simple task, but there are some generic practices in managing, organizing, and practicing software development. Getting these things right we see higher productivity, less stress and burnout, higher quality design, more resilience in the system, and dramatically fewer bugs in production.

Page 32: It is difficult to learn new ideas and discard old ideas. One reason for this is because we don't measure the performance of software effectively. Most metrics are irrelevant (velocity) or even harmful (test coverage).

Page 33/34: Instead of measuring productivity, we can evaluate the effectiveness of software development teams with 2 attributes:

Stability
- Change Failure Rate: Rate at which a change introduces a defect.
- Recovery Failure Time: How long to recover from a failure.
Throughput
- Lead Time: How long to go from "idea" to "working software".
- Frequency: How often changes are deployed to production.

High performing teams dispel the myth of "you can have speed or quality, but not both". This is not true, speed and quality are correlated. The route to speed is high quality software. The route to high quality software is speed of feedback.

Page 36/37: We need to manage the complexity of the systems we create with:

Working iteratively
Employing fast, high quality feedback
Working incrementally
Being experimental
Being empirical

Otherwise the result is familiar - big-ball-of-mud systems out of control technical debt, crippling bug counts, and organizations afraid to make change.

Page 46: How do we predict how many changes are needed to present value to our users? Most organizations guess at a collection of features that present value. The problem with guessing that a set of changes constitute value because it depends on the assumption you know all of the feature that you need when you start. This is the assumption most traditional organizations have made when transitioning to agile.

Instead, by working iteratively we have a choice. We iterate on the products and steer them, based on good feedback from customers toward higher value outcomes. This is often missed by traditional organizations when they adopt agile.

Page 51: Industry data says 2/3 of the best software companies ideas produce zero or negative value. We don't know what users want, and even users don't know what they want. The most effective approach is to iterate. Accept our ideas may be wrong and work in a way that allows us to try them quickly, cheaply, and efficiently.

Assessing business value is also notoriously difficult. To make progress we must take a chance, make a guess, and take a risk. But because we are very bad at guessing, we must do it in a way that does not destroy us. Proceed in small steps and limit the scope of our guesses and learn from them. Work iteratively.

Page 54: Working iteratively, in small batches, if the idea is invalidated by changing circumstances, or because of a misunderstanding, there is less work lost. Small steps really matter.

Sprints are one incarnation of working in small steps. However, there are more incarnations of working iteratively:

CI is iterative - commit changes frequently to learn and understand if our code still works alongside everyone elses.
TDD is a fine-grained iterative approach. Re-running tests for each small change while coding means the code is correct and each step is safer. At each step, the code can be re-evaluated and the code or the design can be changed easily.
** Not in the book, but another one is pair programming. Pairing is essentially code reviewing in very small batches - each new line of code is reviewed instantaneously. It is easy to catch an issue early and correct it quickly. This is in contrast to traditional code reviews, where someone reviews the whole change when the development is complete. This can lead to a lot of rework if there was a bad assumption or bad design decision by the developer.

I've reached page 63.

** Personal anecdote: In my current organization, we are 100% in the boat of adopting agile, but in the incorrect way described on page 46. We may be iterating quickly, but we are not releasing quickly. If we are not releasing quickly, then we cannot get feedback quickly. So how do we continue iterating if iterating requires feedback to know whether the value we produced is good or bad?

Page 64: [Using TDD] "If we are going to write the test first, we have to be a strange, dumb kind of person to make our own lives more difficult. So we are going to try to do that in way that makes life easier."

We're extremely unlikely to write a test in a way that we can't get the results back from the code we are testing. When we create the test we are also designing the interface to our code.

In TDD there is pressure applied to write code that is more testable:

Modular
Good separation of concerns
High cohesion
Information hiding (abstraction)
Appropriately coupled

Page 66: The independence of microservices is a significant benefit, but also a significant complication. By definition they are independently deployable units of code. That means we don't get to test them together.

Page 69: When your team has an idea to improve its approach, be cleat where you think you are now and where you would prefer to be. Describe a step that you think will take you in the correct direction and decide how you will decide whether you are closer or further away from your target state. Make the step and check and repeat until you are at the target.

Page 79: Never aim to add code for things you don't know are needed now.

If your code is hard to change, it is low quality, whatever it does.

Working incrementally is fundamental from building any complex system. It is an illusion to imagine that such systems spring fully formed from the minds of some experts - they don't.

Page 97: If you write your automated tests after you have written the code the value of the experiment is reduced. An experiment should be based on some kind of hypothesis.

Organize development around a series of iterative experiments that make tiny predictions about the expected behavior of our code that will allow us to incrementally increase the function of our software.

The clearest form of this is TDD.

Page 106: A lot of code that I see is that people don't just do modularity badly, but rather they don't attempt it at all. A lot of code is written as a recipe, meaning a linear sequence of steps collected together in methods.

Page 110-111:

System A -> System B -> System C

Most organizations working on System B jump to the assumption that it is essential to test everything together to be sure the system is safe to use. The whole system is so complex that we lack precision, reproducibility, control, and clear visibility of what any results that we do collect really mean. E.g. it is impossible to measure System A sending a malformed message because the real System A will be sending well-formed messages. The results that we do collect don't tell us much. If a test fails, is that because of our system or one of the others? Complex mega-system testing may lead to simplistic tests, so bugs may be missed.

Page 112: If you have a suite of automated tests to determine if the software is ready to release and those tests don't produce the same result every time, what do those tests really mean? In order to depend on the tests, they need to be deterministic. It is worth the work to achieve repeatable outcomes. This not only affects tests, but the real value is that it will impact the design of the software.

Page 116: If the pipeline says everything is good, there should be no more work to do to make you comfortable with the release -- nothing -- no more integration tests, sign offs, staging tests. This has implications for the sensible scope of a deployment pipeline, if it is releasable it must also be independently deployable. The scope of an effective deployment pipeline is always "independently deployable unit of software".

There are 2 strategies to take this idea to the logical conclusion:

Build, test, and deploy everything in our system together.
Build, test, and deploy part of that system separately.

There is no halfway solution - if we don't trust the results of the pipeline and feel we need to do additional testing/pipelines then the message deployment pipeline generates is unclear. The scope of our evaluation is compromised - when are we done? - when the other tests/pipelines are done?

If we wait for the other pipelines, we know include everyone else's changes and in turn have a monolithic evaluation. The most scalable approach to software development is to distribute it. Reduce coupling and dependencies between teams and their products, independently create, test, and deploy their work with no reference to another team. This is how Amazon grew so quickly.

This idea of building, testing, and deploying independent modules - this is what microservices are [ or should be :( ]. If you are testing your microservices together, they are really microservices. Part of the definition of microservices is independently deployable.

Page 119: If we need small teams to create high quality work, then we need to find ways to seriously limit the coupling between those teams. This is as much of an organizational strategy problem as a technical one. We need modular organizations as well as modular software.

Page 125: Code is a communication tool. The primary goal of code is to communicate ideas to humans. Optimize to reduce thinking rather than e.g. reduce typing.

Page 152: Organizational and Cultural Problems: "The first thing to say is why do we, as software developers, need to ask for permission to do a good job? We are the experts in software development, so we are best placed to understand what works and what doesn't."

Page 153: If I were a chef preparing a meal, I could probably prepare a meal more quickly if I decided not to clean my tools and work area when finished. This might work for 1 or 2 meals, but all the time and I'd get fired. This would give patrons food poisoning. By the 3rd meal, I would be far slower because of the mess that would get in the way of my work. I'd have to clear the work area for every task and struggle with unmaintained tools. A chef would never ask "do I have permission to sharpen my knives?" because as a professional chef, we'd assume those are fundamental to being a professional.

As software professionals, it is our duty to understand what it takes to develop software and own the responsibility for the quality of the code we work on. Software is not a game of short term wins. If you drop testing, avoid refactoring, not continually tweaking your design, you are going slower, not faster.

Page 155: We need to work in small steps that are easy to undo. We need out code to be a habitable space that we can revisit months or years later and still understand.

Page 158: If you have thought, "We may not need this now, but we probably will in the future", you are over-engineering by future-proofing. YAGNI!

Page 160: "You don't write specification after you have completed the work; you need them before you start. So we will write our specifications (tests) before we write the code. Since we don't have the code, our focus is more clearly fixed on making our life easier. Our aim, at this point, is to make it as simple as possible to express the specification (test) as clearly and simply as we can.

Inevitably then, we are, or at least should be, expressing our desires for the behavior that we want, from our code from the perspective of a consumer of it, as clearly and simply as we can. We should not be thinking about the implementation detail that will be required to fulfill that mini-specification at this point.

If we follow this approach, then, by definition, we are abstracting our design. We are defining an interface to our code that makes it easy to express our ideas so that we can write our test case nicely. That means that our code is also easy to use. Writing the specification (test) is an act of design. We are designing how we expect programmers to interact with our code, separate from how the code itself works. All this before we have gotten to the implementation detail of the code This approach, based on abstraction, helps us separate what the code needs to do from how it does it. At this point, we say little or nothing about how we will implement the behavior; that comes later"

I've reached page 162.

Page 164:

The subtlety here, though, and the enormous value that TDD delivers, is that if I have written my abstract specification, focusing on what the code should do and not how it achieves that outcome, then what my test is expressing is my abstraction. So if the test is fragile in the face of change, then my abstraction is fragile in the face of change.

Page 168:

We have represented the transactional nature of the relationship between storing and adding the item to the cart with a success or failure return value. Note we aren't confusing our abstraction by returning implementation-specific errors codes and leaking those into our domain-level abstraction.

Page 172-173:

If we don't take the issues and costs of coupling seriously, then we create big balls of mud in software, and we create organizations that find it impossible to make or release any change into production. Coupling is a big deal!

You don't get better software faster by throwing more people at the problem. [...] developmentally coupled, we could maybe work to coordinate out releases. [...] The overhead of keeping everyone in step rapidly spirals out of control.

Minimize this overhead [...] through continuous integration. The downside of scaling up in this way is that you have to invest heavily in the engineering around repositories, builds, CI, and automated testing to get feedback on changed quickly enough to steer development activities. Most organizations are unable or unwilling to invest enough in the changes necessary to make this work.

Microservices are the most scalable way to build software, but they aren't what most people think they are. The microservice approach is considerably more complex than it looks and requires a fair degree of design sophistication to achieve.

Microservices are as follows:
- Small
- Focused on one task
- Aligned with a bounded context
- Autonomous
- Independently deployable
- Loosely coupled

Page 174:

If we can build a service and deploy it independently of other services, that means we don't care what version those other services are at it means that we don't get to test our service with those other services prior to its release. This ability wins us the freedom to focus on the now simple module in front of us: our service.

Page 179:

Dependency management is an insidious form of developmental coupling. If your service and my service share the use of a library of some kind and you are forced to update your service when I update mine, then our services and our teams are developmentally coupled. This coupling will have a profound impact on our ability to work autonomously and to make progress on the things that matter to us. It may be a problem for you to hold your release until you have changed to consume the new version of the library that my team imposed upon you.

Page 181:

Failure points in synchronous communications
1. There may be a bug in A
2. A may fail to establish a connection to the network
3. The message may be lost in transmission.
4. B may fail to establish a connection to the network.
5. There may be a bug in B.
6. The connection to the network may fail before B can send a response.
7. The response may be lost in transmission.
8. A may lose the connection before it has the response.
9. There may be a bug in A's handling of the response.
At the point that something like a connection problem or a dropped message on the network happens, this technical failure intrudes into the business-level conversation. [...] but l am a believer in treating process boundaries as asynchronous and communicating between distributed services and modules via only asynchronous events. For complex distributed systems this approach significantly reduces the impact or abstraction leaks and reduces the coupling.

Page 183:

I see many large organizations hamstrung by organizational coupling. They find it almost impossible to release any change into production, because over the years they have ignored the costs of coupling, and now making the smallest change involves tens, or hundreds, of people to coordinate their work.

If you want consistency across a large, complex piece of software, you should adopt the coordinated approach. In this you store everything together, build everything together, test everything together, and deploy everything together. This gives you the clearest, most accurate picture but comes at the cost of your needing to be able to do all of these things quickly and efficiently. I generally recommend that you strive to achieve this kind of feedback multiple times per day. This can mean a significant investment in time, effort, and technology to get feedback quickly enough.

Page 184:

The cost [of a microservice approach] is that you give up on coordination, or at least reduce it to the simplest, most generic terms. You can offer centralized guidance, but you can't enforce it, because enforcement will incur coordination costs. Organizations that take microservices seriously consciously loosen control; in fact, a microservices approach makes little or no sense in the absence of that loosening of control.

My take: I feel like our current architects took a very simplistic definition of microservices. I infer they wanted to independently scale services, and that is all. There are no bounded contexts, no team autonomy, few & coordinated releases, technically sliced services, extremely high coupling, release management, lots of process. The other 99% of microservices was ignored.

Page 189:

Adding separate steps into the process, in the form of separate groups of people, does not improve the speed or quality of the feedback that we can collect. This is not criticism of the people involved; all people are too slow, too variable in what they do, and too expensive to rival an automated approach to gathering the feedback that we need.

ryanlevell / books

Modern Software Engineering (64%) #8