search

nunit / nunit-console

NUnit Console runner and test engine
MIT License
215 stars 151 forks source link

NUnit Console/Engine v4 #752

Open ChrisMaddock opened 4 years ago

ChrisMaddock commented 4 years ago

We've talked casually about several changes we'd like to make in v4 over the years - but there's never yet been an important enough breaking change to justify a new major version. Most of our big projects currently seem to be possible in a backwards-compatible fashion. My view however is that the .NET Core changes are starting to create these, and it might be time to start thinking about the future.

I'd like to start tracking changes we'd like to make in v4 - I've created the v4 milestone to do that. Consider this an open invite to anyone to put in issues against v4! 😄 For now, let's just go with tracking them - in future we might create a v4 branch, but I don't want to introduce the overhead of having to keep two branches in sync until we need to.

For issues under 'v4', I'd like to mainly target them towards breaking changes - new features that could be done in a backwards-compatible way in v3 seem much better not to be delayed, and tracked as normal issues. Ultimately I don't see v4 being a 'big bang' release - more a well-curated group of minor breaking changes, to make our lives easier and allow us to move quicker in future.

Thoughts very welcome. 🙂

ChrisMaddock commented 4 years ago

I'd encourage everyone to create to create an issue-per-change as normal for this, but due to time, here's a quick brain dump of things I see possibly going in to this. I'll create individual issues for these at some later point, with more time.

CharliePoole commented 4 years ago

@ChrisMaddock I'd like to add some things which are more like "decide whether to dos" than "todos". Probably using label is:question (for the team) or is:idea depending on how doubtful we feel about the particular item.

As we discussed, I'd also like to create a GitHub Project with a subset of these items aiming toward re-converging the NUnit and TestCentric engines in V4. I'll create that under the TestCentric project, since it's a goal for that project and it will allow pairing up issues in both projects, where one or both may need to make a change in order to converge.

One comment about not wanting this to be a "big bang"...

I think that's desirable and also possible to a certain extent. The API is the issue. Once you release an API, you are stuck unless we carefully define what changes are allowed. I'll continue in the individual issues we create for this.

CharliePoole commented 4 years ago

@ChrisMaddock Re your list of potential issues... could you clarify a few things? (either here or in creating the individual issues):

CharliePoole commented 4 years ago

@ChrisMaddock Do we have a written definition of "Breaking Change" anywhere? I know we have an unwritten one, which is essentially to try not to break anyone even if they use the code in an unexpected way.

One thing to consider for V4 is to write down a definition and publish it so that we do not have to feel responsible for "breaking" people who rely on implementation details we don't want them to rely on. I wrote a document for the TestCentric GUI, which came out of my experience with NUnit and our changing view of "breaking" over time.

I suspect we might all argue a bit over the details, but I hope that "writing it down" isn't a point of contention. :smile_cat:

ChrisMaddock commented 4 years ago

Sounds good. For v4 - I'd like to enforce public/internal classes as aggressively as possible - so we lose as much of this ambiguity as possible.

Would you like to take the lead on writing something up, and then we'll argue it out? 😇

ChrisMaddock commented 4 years ago

Oh - I only saw the last response at first. On your other points:

ChrisMaddock commented 4 years ago

Been thinking more about this - I'd like to lay out some high-level key objectives for v4, to give us more focus. Here's some of my thoughts, suggestions welcome! I might try and write something up for this, next time I get a little time.

1. Test Anywhere

Since NUnit 3.0 was released, the .NET scene has evolved from being largely Windows focussed, to running across a whole bunch of different platforms. Currently users often need to set up large amounts of boilerplate code to test across these platforms - the next generation engine should provide that functionality directly to the user. The engine should also be suitably extensible, to allow users to run tests on less-common platforms that the project itself isn't resourced to support.

2. Improving the Developer Experience

NUnit has a long, proud history and a big user-base, utilising a range of technologies from the last 20 years of .NET development. This is not without it's cost - we have a large codebase still restricted to building for legacy platforms, and a drawn-out and manual release process, which impacts our ability to move quickly.

For v4, I'd like to see us improve the experience for developers of the engine and console. This would cover things like moving as much code as possible to modern platforms, reducing our reliance on manual testing and release processes, and considering code complexity as a factor against feature value and development.

3. Clarity of supported interfaces, and protection for the future

A big cost of on-going development, particularly in the framework, is handling code which has been exposed to end users, but was never intended to be as supported interface. The team are often in a position of 'guessing' how user's may be using a class to decide on an appropriate change - which delays and confuses external contributions, and often unintentionally breaks end-users' setups.

For v4, I'd like for us to consider locking everything down as far as possible, and only exposing guaranteed interfaces, intended for ongoing support. Docs are great too - but locking things down prevents people making the mistake right from the start.

Thoughts and ideas welcome. 🙂

CharliePoole commented 4 years ago

I like the high level goals. They provide a vision that your team (us) can buy into. I'm missing one thing... How about a statement about backward compatibility? Traditionally, NUnit has been at an extreme of backard compatibility as compared to other projects - and especially to Microsot. We've moved somewhat away from that or at least changed what backward compatibility means to us. Shouldn't we say something that clarifies where we are on that now?

As a straw man, I would say "We will continue to support running tests on all the platforms supported by NUnit 3, although we may no longer support running some platforms in process."

ChrisMaddock commented 4 years ago

It's a good point Charlie, and one I've thought a lot about. Your suggestion motivated me to start writing them down, and it turned into a small essay I'm afraid. 😅 Let's discuss this over at https://github.com/nunit/nunit-console/issues/770.

ChrisMaddock commented 4 years ago

I should add the above is specifically relating to platforms, as that was the particular element you focussed on in your straw-man. 🙂 I'd like to also expand our backwards compatibility statement to cover the API's and functionality we intend on supporting - the engine API of course, and then additional particular console behaviour, any generated xml, packaging, and other things we may see fit.

CharliePoole commented 4 years ago

I liked what you wrote!

WRT more general backwards compatibility, is there an existing statement somewhere? If so, I wasn't aware of it. I wrote something for my own work in the GUI because I think people will be OK - or more OK - with changes if you have given them a statement in the first place.

ChrisMaddock commented 4 years ago

No, not yet, at least not that I know of. It sounds like a sensible thing to document.

CharliePoole commented 2 years ago

I think most of what is covered in this issue has been either completed or documented as an individual issue. I'll review it before closing.