Future direction of TeamCity project

[CharliePoole]

We have been spending a lot of time discussing generalities in this governance repo. I thought it might help if I focused on a few specific issues for a bit. At least that way I think we'll all be talking the same language. 😄

The TeamCity extension is exclusively maintained by JetBrains folks, in particular by @NikolayPianikov. This is as it should be, IMO. It's a commercial product and is not used by all our users. And, of course, we don't really know how it works and so couldn't maintain it if we wanted to.

I would have preferred getting it out of our stable of projects entirely. After discussion with @NikolayPianikov and some other JB folk, I agreed we would keep it for a time. @NikolayPianikov is entirely responsible for maintaining it and releasing it. Our only role at this time is to bundle a version of it with the console runner.

I propose that we set a long-term direction of separating this project from NUnit more completely. I don't think it's fair for us to be giving a special status for a product that is, after all, only one of the commercial and open source choices available to users of NUnit. It also seems like a clear violation of our staement on platform independence in the vision doc.

OTOH, I don't want us to make any sudden drastic moves. I would see something like this happening...

1. A public announcement of our intention, along with the reasons.

2. Removal of the team-city extension from those that are bundled with the console runner.

3a. Encouragement of JB to host the extension elsewhere OR 3b. Making it clear that it's a contributed project - possibly in connection with a new nunit-contrib organization on GitHub that would house all our contributed projects.

In the past, I would have simply decided this. Under our current governance, I think it calls for a vote. Let's do that after discussion and questions.

[rprouse]

Whatever we decide for the Team City adapter, I think we should also apply to the TD.NET adapter contributed and managed by @jcansdale.

I would be fine continuing to host them under the NUnit organization. I think it gives users one place to look for everything NUnit related, even if it is not maintained by us. It also makes it easy for us to transfer misreported issues over to the appropriate repository. I think a separate contrib organization would just add additional work and administration for little benefit.

As a compromise, we could require all contrib projects to clearly state that at the beginning of their README on GitHub. It should also state that the NUnit team is not responsible for maintenance or support. This would be a requirement for being hosted by the organization.

As for removing TC from our bundled extensions, in principle I am for the idea, but in practice I believe that it will just lead to a flood of issues reported to us that TeamCity is broken. The majority of users never see our announcements and have come to expect NUnit to just work in TeamCity.

We could answer/transfer all of those issues, but we already spend more time answering issues and questions than we do coding. I want to find ways to reduce that load, not increase it.

So, my proposal for the eventual vote - Maintain the status-quo, but require contrib projects to identify as such in their readme.md.

[ChrisMaddock]

As a compromise, we could require all contrib projects to clearly state that at the beginning of their README on GitHub. It should also state that the NUnit team is not responsible for maintenance or support. This would be a requirement for being hosted by the organization.

I agree with this - these projects should be clear that they are entirely supported and managed by the relevant companies/people.

---

Regarding removing the TeamCity extension from bundles, I disagree. This would be a significant breaking change for many of our users. I agree this situation doesn't fit our vision - but view this as a 'historic mistake'. I think the best path forwards is to acknowledge that this is why the TeamCity extension is bundled, and clearly state that this is due to historical reasons, and to not offer the same level of integration to any equivalent commercial extension.

I think removing this extension would be something that would belong in NUnit 4.0. (Too soon for that!!) To my mind, the current situation is working well, but it would be interesting to hear Nikolay's thoughts.

[CharliePoole]

We are not at the voting stage, and I don't plan to call for a vote to maintain the status quo... I wouldn't have made an issue if that were the case.

I feel very strongly that distributing the teamcity extension gives an appearance that we are somehow aligned with teamcity and that we recommend it over other tools. We should not be doing that.

Users already mostly get their extensions in one place... nuget.org. We have made it extremely easy to add extensions to an existing console runner using nuget and I expect we can do the same thing on chocolatey.org. The only reason that teamcity has the status it does with us is that we originally wrote it.

What happens if somebody else wants us to host their extension? Are we willing to do the work that is required for every extension? What if it's an extension for a different CI system? Do we then distribute extensions for two competing CI systems with the runner? That makes no sense.

The other four extensions at least provide a sort of core functionality, even though not all people need all of them. I think TeamCity is in a separate category and we should stick to our guns and stop distributing it.

[NikolayPianikov]

Of couse, I like the idea of @ChrisMaddock :) I was always interested in the question: Are we planning any ways for integration with other CI? I mean when user can see realtime feedback from the test runner in the CI. The xml report it is quite slowly feedback

[ChrisMaddock]

What happens if somebody else wants us to host their extension? Are we willing to do the work that is required for every extension? What if it's an extension for a different CI system? Do we then distribute extensions for two competing CI systems with the runner? That makes no sense.

As my comment above, in my opinion, 'no'. In my opinion, we should acknowledge this as a historic decision that we continue to support, would aim to remove in future major versions, and accordingly would not expand to any any other commercial project. @CharliePoole - I think we fundamentally disagree on this one, so I'll leave my opinion there, and wait on the decision of others. 🙂

Are we planning any ways for integration with other CI? I mean when user can see realtime feedback from the test runner in the CI. The xml report it is quite slowly feedback

@NikolayPianikov has potentially inadvertantly just proposed a third solution. We could of course, remove any mention of TeamCity from the extension, and package it as a generic 'real-time-integration' extension. This sounds like a new project that would need a maintainer however... ;-)

[NikolayPianikov]

And one more great argument for TeamCity: we have very good support of NUnit in TeamCity and in Resharper/Rider. Due to this NUnit is very popular among our customers (and we have a lot of them). I think it would be nice to continue this good deed. If you build the same kind relationship with other CI it will be also fine.

[CharliePoole]

@ChrisMaddock Removing in a major release is a decision I can agree with. We're not trying to decide what to do tomorrow, but to set policy for the future.

@NikolayPianikov The XML report is not what NUnit now offers for real-time feedback. Rather, it's the events. That's our standard. As you know, the teamcity listener translates what NUnit provides to what teamcity wants.

@ChrisMaddock If the teamcity format were something already being used by others, then I think it would make sense to adopt it as an NUnit standard. But it isn't. And disguising the teamcity standard as an NUnit standard is even worse in my view than openly supporting teamcity.

As an analogy, consider the "VS" project format. We originally did not support it as a proprietary, commercial format. Other IDEs arose that used the same format then we began to support it.

@NikolayPianikov I understand that you have to say that TC has great NUnit support but have you been following all the issues that we keep getting around TC? It gives me a pretty negative impression.

[rprouse]

I agree with @ChrisMaddock in that it is a historical mistake that we need to live with and learn from. We won't include anything like it in future bundles and will remove it at a major upgrade point. I would remove it now, but I don't want to deal with the flood of issues that we broke TeamCity 😄

I can see how it is a nice feature for TeamCity users, but I would prefer to see it ship with TeamCity rather than NUnit.

[CharliePoole]

So if we want to undo the historical mistake in a major release, do we wait for that release or make it public now? And if so, how do we make publish the information? @rprouse ? @NikolayPianikov ?

[NikolayPianikov]

@NikolayPianikov I understand that you have to say that TC has great NUnit support but have you been following all the issues that we keep getting around TC? It gives me a pretty negative impression.

@CharliePoole could you provide some details about issues related to TeamCity?

[ChrisMaddock]

So if we want to undo the historical mistake in a major release, do we wait for that release or make it public now? And if so, how do we make publish the information? @rprouse ? @NikolayPianikov ?

I was assuming the next major release would be similar to the NUnit 2 -> NUnit 3 release, and come with a page full of breaking changes. As such, I don't see any reason to formally publish anything until that point.

[CharliePoole]

@NikolayPianikov Do you follow the feed of issues and PRs for nunit-console? That's where they all show up. If they seem to be an event-listener issue, we transfer them to you, otherwise not.

@ChrisMaddock I'm of two minds. We had a lot of advance notice of planned changes with NUnit 3. OTOH, nobody ever seemed to pay attention and we got issues from surprised users after the upgrade. However, I still think we should give notice if we want to do it right. But maybe the time is when we actually plan the 4.0 release.

Should we start a folder or something of "Notes toward 4.0?" I previously had a lot of those things as issues in GitHub but they are no longer there. At a much earlier time, I used a Roadmap doc, but that's gone as well. I"m throwing this out as something somebody else may want to try again.

I think we at least need a vote on this to close it and should notify JetBrains through @NikolayPianikov that it will be removed at some point. Here's what I propose we vote on:

"With the release of nunit3-console 4.0, we intend to stop distributing the teamcity-event-listener bundled with with the console as a standard extension. We will continue to support the extension when it is installed by the user."

My use of the word "intend" instead of "will" is to give us a little wiggle-room.

Please suggest any changes to the text and then let's vote.

[ChrisMaddock]

To clarify, I presume we're both expecting 4.0 to be several years away? I was considering this to be on the same timescale as versions 2 -> 3 were.

A 'notes for 4.0' page in the developer docs in the wiki sounds a sensible place.

One point on the wording - I thought we'd already agreed that all support was to be done by JetBrains? By "We will continue to support the extension" - did you just mean that we will continue to support the interfaces it uses? In which case, does that need saying?

[CharliePoole]

Sounds good to me... Anyone else?

On wording, I agree it's ambiguous. Basically, we currently support it by having a --teamcity option in the console. Everything else it uses, including the interface, is generic.

Hows this? "We will continue to support the --teamcity option in the console runner when the extension is installed separately by the user."

[CharliePoole]

On timescale... that's my expectation. If something happened to cause us to do a 4.0 release in the short term, I figured we could just push the change back to 5.0. That's why I wrote "we intend."

[rprouse]

I vote yes on @CharliePoole's proposed wording.

[jnm2]

👍 Sounds good to me.

[ChrisMaddock]

Sounds good - although if the extension isn't going to be installed by default, then I wonder if the flag is redundant! That's something to figure out when we come to 4.0 though, the revised version here sounds good for now.

[CharliePoole]

@ChrisMaddock Extensions are capable of being enabled or disabled. The teamcity extension makes use of this and the command-line option enables it.

[OsirisTerje]

Yes from me.

[NikolayPianikov]

We at JetBrains understand the intent of removing the TeamCity event listener from the NUnit official distribution package, although such a change is undesirable for us.

We understand and respect the decision of the NUnit project team to keep NUnit vendor agnostic.

As for us, our primary desire is to provide the best possible experience for .NET developers using our tool. That is why we’ve initially made the decision to contribute to NUnit source base on a daily basis.

To ensure the user experience doesn’t worsen much for the users who use NUnit and TeamCity together, we would like to carefully discuss possible usage scenarios. We would like to be able to assist the existing users to migrate to the new NUnit version without the out-of-the-box TeamCity extension with the least pain as possible.

[CharliePoole]

@nunit/core-team FYI, @NikolayPianikov consulted with his company and was unable to accept my offer to add him to the copyright for the teamcity extension. I closed that issue on teamcity-event-listener.

[CharliePoole]

I created a page at https://github.com/nunit/docs/wiki/Notes-Toward-NUnit-4.0 with the wording we discussed for the teamcity extension. If it looks OK, we need to figure out where to link it in the menu. I'm thinking it can be top-level in the Developer section.

[CharliePoole]

@nunit/core-team I'd like to close this. Any comments?

[OsirisTerje]

We're not blocking anyone by this, are we ? From the options on top, I think option 3b looks like the most nice one right now.

[ChrisMaddock]

Looks good to me. Maybe it would be good for the new wiki page to reference @NikolayPianikov's post above, as JetBrains' response to the issue.

https://github.com/nunit/governance/issues/11#issuecomment-308443900

[CharliePoole]

@OsirisTerje Only thing blocked is my own checklist of things I'm supposed to complete by June 30!

I think we are past the option choices, having said we will stop distributing the combined package with the 4.0 release. I'm just trying to get the details of the notice out of the way now.

@ChrisMaddock That comment doesn't really say much except that we'll try to make it as easy as possible on users. I'll just add a line from it so we are saying the same.

[rprouse]

I think we've had plenty of discussion and think we can close 😄

[CharliePoole]

Where would you like that page linked?