nottrobin
commented
5 years ago Thanks for raising this.
The reason for my response on the Python Practices PR is because I saw the role of that PR as defining current practice rather than forming new practice. To build up this set of practices quickly, I'd like to, where possible, separate the two - quickly write documents that define the way we work first, and then discuss changes to those documents to evolve the way we work.
Yes I think we should definitely discuss it in dev catch-up. I'll also provide a bit of background here:
Up until now, we haven't enforced testing in any sense. Now may be the time to change that. I see the reason that we haven't been more focused on testing as having 2 broad reasons:
1. A very small team
We have a very small number of Python devs managing a very large number of projects. Currently we have an all-time high of 4 devs (if you count me), but we've often been as few as 2 devs, managing over 20 production websites, many more supporting codebases, and the CI & deployment systems & development tooling for all those projects.
2. Not an "engineering team"
Until quite recently, it would have been hard to characterise us as an "engineering" team. We were one small part of the Design team, creating a large number of simple websites of usually fairly static content, to support Design and Marketing goals.
Although our websites have always been public and viewed by millions, unlike other "engineering" teams we were never delivering services or software. We weren't really handling customer data, or providing service contracts that clients rely on or distributing tools to be run by others (apart from of course making our stuff open-source).
We also never had the benefit of either a fixed predictable release cycle, or of ever really getting to choose to release new features only when they were ready. We'd very often have tight deadlines imposed from the very top (this is still true - there's a key example from just yesterday which I can elaborate more on in Dev Catch-up).
We needed to move fast, and I would say that strategy wasn't entirely inappropriate - it worked pretty well. We have successfully got sites out the door in very quick turn around, those sites look good and generally perform well, and we certainly haven't experienced an outsize number of outages or security breaches.
Of course, we've always had some tests (e.g. documentation-builder, yaml-responses, assets.ubuntu.com), and I've tried to write them where I've had time, but up to now I've never strictly enforced it.
The changing team
I say "until quite recently" because the type of software our team produces is definitely changing. The most clear example of that is your work @squidsoup on MAAS and related products. This is work that will be distributed as software, and security and reliability are therefore paramount. However, this particular work doesn't involve Python, and so isn't impacted by the Python test policy. I don't know what testing policy you're following for your work, but I'm guessing you're treating tests as mandatory, which is great. Maybe this could also be formalised in practices, in tandem with evolving the Python policy.
The other significant example would be https://snapcraft.io. This is actually neither a public service or distributed software, but it is significantly more complex than our typical project, and it forms a more significant part of the supporting infrastructure for the Linux ecosystem, and handles very sensitive user tokens and data. This project really does need a higher level of focus on reliability and security. It has fairly high test coverage (thanks to @tbille) for this reason, but currently no official policy to enforce it.
I'm not aware of other existing projects which have the significance of these two, but I do believe the number of such sensitive projects that fall to our team to implement will only increase. So we do need to sure up our processes to handle them.
Options going forward
I welcome the call to revisit this policy.
I'm very open to revising, or creating, a policy around testing. I personally have specific opinions and concerns around how we test - I'm aware that occasionally testing policies of other teams have turned out to be unnecessarily obstructive - so I do believe that it's important to have good practices defining what and how to test - e.g. the importance of not testing internals (as some team members will remember - although that particular point could probably have been better articulated). But that doesn't need to get in the way of evolving a policy around requiring tests to be written, only that we should evolve testing guidelines at the same time, which I intend to do.
As I see it, we could tighten up our policy around Python testing in one of two ways:
- Strongly recommend testing for working on most projects, as a default, and then enforce testing any new work for specific projects (e.g. snapcraft.io). This would not be a huge departure from current practice, but would probably lead to a slight increase in testing on snapcraft.io, and hopefully encourage significantly more voluntary testing on other projects, without significantly impacting on delivery when deadlines are tight. Or:
- Use this as an opportunity to switch straight to enforcing testing on all new work. This will clash with our pressure to "move fast" fairly often, but we might want to do it anyway and let those pain points happen and work themselves out. However, this switch in policy would need buy-in from @pmahnke, as it would impact stake-holders.
So let's discuss these options further in Dev Catch-up.
squidsoup
bartaz
In the python standards doc that just landed, we state:
This is a bit at odds with Canonical engineering culture. We should never land code without tests where feasible. If there are technical reasons making it difficult to write tests, we should discuss those and find a solution. Maybe this would be a good discussion for the next dev catchup?