META: Pull request volume too high

[michaelbprice]

@philsquared has a lot of work on his plate. @philsquared, have you considered pulling in additional maintainers to keep up with the volume of PRs?

[philsquared]

I would love to!

There are a handful of very helpful, responsive and smart individuals in the community who frequently step in respond to questions and issues - for which I'm very grateful. But keeping up with PRs is a different proposition. What I'm really lacking right now is better testing infrastructure (ironically) - in particular running on multiple versions of GCC, Clang and VC++ - preferably on multiple platforms. Personally I develop in Xcode (Clang) on my Mac, but tend to give it the most use during the day on a Windows project (currently VS2013). There are Travis builds for Clang and GCC - but only specific fixed versions (rarely the ones that seem to have all the problems, it seems).

Despite all that sometimes quite embarrassing regressions slip through if I don't review everything in excruciating detail (and even then). A more comprehensive set of environments would be a huge help - and would let me - or some other trusted maintainers - accept PRs much more quickly. My biggest bottleneck is that I do most of my Catch work on my commute (when I'm not preparing talks, writing blog posts, learning new languages or working on iOS projects) - with intermittent connectivity. It's not very conducive to setting such an environment up. If anyone was in a position to help there it would be transformative!

[michaelbprice]

As someone who works in test automation, you are describing a familiar problem. Getting the right systems set up in the right way to build/test the right build configurations across all "supported" platforms is certainly a challenge, especially when you might have to deal with "non-free" software like Windows, where you need licenses to even run the operating system.

Is there any "requirements" written down anywhere about what sorts of things would be most helpful? Is it Visual Studio versions? Static analysis? Build permutations? Virtualization?

Figuring this kind of stuff out is basically my day job, and I'd be happy to help you figure out a path forward.

[DavidEGrayson]

There are some free hosting options from Microsoft Azure; maybe those would work for testing Catch with a few versions of Visual Studio.

If could run some service on my Windows machine that quietly assists with testing Catch in the background, I probably would. It would be like SETI@Home but for software testing. Just a random thought.

[refi64]

cough AppVeyor cough

They have a few versions of VS to run your tests with. And it's free.

[philsquared]

Thanks for the replies. Just to be clear the issue is one of time and connectivity - not services and platform availability (although I am open to suggestions there). To give you an idea: it just took me five minutes to open this page and whether I can post this response straight away or will have to wait another 10-20 minutes I don't even know yet. Yesterday, on my way home, I could't get a single page downloaded! So, for me, working with any cloud hosted environments is just not possible. If anyone else would be happy to do that, though - at least to do most of the work of getting something set up - that's what would really help.

As for Windows licenses: Azure might work out to be a good option. But I also have a Mac Mini sitting idle at home that can run Windows in a VM (and I have licenses for XP and Windows 7, at least). I can also run different flavours of Linux in a VM (which is something I started doing on my laptop but any time I get stuck it just takes too long to be able to work stuff out without a reliable connection). I can run TeamCity on my Mac Mini - which, apparently, can interact with local VMs - as well as Azure and AWS.

If anyone wants to volunteer I'd be happy to open up VNC ports to my Mac Mini if it would help. If you have a preference for other cloud solutions I'm just as happy.

Next time I have a stable connection I'll take a look at AppVeyor (I currently can't reach it).

[philsquared]

@michaelbprice - you asked about "requirements".

I'd like to be able to use as many representative OSes as possible (Windows 7 & 10, OS X - latest is fine - as I say I can provide that, and as many popular Linuxes as people are happy to set up and maintain). On Windows I'd like to test with as many as possible of: VS2008 (still used by many), 2010, 2012, 2013, 2015. Ideally gcc/ clang, too. Any other compilers (e.g. Intel) a bonus.

On Mac & Linux a representative range of GCCs and Clangs in common use is the obvious requirement. Exactly which versions I couldn't say - the main thing would be getting it set up to run multiple versions (this is where I've got stuck before - even manually switching GCC versions on my Mac seems to be a lot more work than I would have expected). I would imagine that once it can work with 2-3 versions, adding more as required (e.g. a bug is filed against a specific version) would be relatively straightforward.

In terms of what needs to be run I have a reasonable start at a test suite. It can certainly do with being extended (in particular exercising more of the command line options - and combinations thereof). But I can do that here. I might still need to do some more work to make it runnable in a platform independent way. It does require Python, but I doubt that is an issue.

The way it works (currently) is to run the CatchSelfTest project with a range of command line options (at the moment it's mostly different reporters with some combinations of only successful tests, as well as (deliberately) failing tests). It captures the output in a text file, strips out some volatile information: e.g. line numbers and memory addresses, then does a textual diff against baselines.

As I say this can (and should) be extended in at least a couple of ways: (1) more command line combinations and (2) using both the multiple and single-file versions (currently it only runs the multi file version). I am, of course, open to other ways that this could be extended. One opposing force here is that the regression-based approach can tend to be quite noisy with false positives because any change in the output - including just adding new tests - requires manually "approving" the changes. This is usually manageable enough, although sometimes can be quite subtle and I've let bugs in - even though they did show up - because they've slipped past in the noise. Any suggestions for doing better there would be welcome.

[RossBencina]

+1 for AppVeyor for Windows testing, I've seen it used to great effect here: https://github.com/nlohmann/json

[michaelbprice]

@philsquared If there existed systems that would do all of this coverage for you, but you were unable to see the results without asking someone with access... would that be a non-starter for you?

@Lexmark has an interest in keeping this project going, but we aren't really set up to expose our build infrastructure out to the public-at-large. Of course, there's also the matter of whether or not you want your processes to depend so heavily on a corporation over which you have little-to-no control over.

[philsquared]

It's been a while since this discussion started but this is a good time to reinvigorate it, I think. As discussed here: http://www.levelofindirection.com/journal/2016/12/22/on-joining-jetbrains-1.html my circumstances have changed and (a) I should have more time for Catch (despite a slow start there) and (b) when I am working on it it's in a decent environment with stable internet now!

Additionally, I'm pleased to say that @horenmar has become a key maintainer and has been generously investing his time in helping to work through the backlog (thanks again, @horenmar!).

There's a way to go but it's going in the right direction now.

One of my focuses now is to get the test environments I mentioned before up and running. I've already done some work to make my release scripts more portable - as well as making CMake more useful.

I'm planning on getting a TeamCity server up and running, which can interact with build agents on different platforms. This was my plan even before I joined JetBrains - but makes even more sense now :-)

That doesn't preclude extending the use of Travis further, or using AppVeyor - but it feels like I should have more control over my own TC instance (especially after the issues with the Travis builds last year). That could just be my lack of experience with Travis or AV - feel free to try to persuade me otherwise.

[horenmar]

I don't think there is a reason not to have Travis/AppVeyor as well as TC. To me the benefits of Travis and AppVeyor is that you get a free* CI service and CI is an incredibly useful thing (assuming you have good tests). You also don't have to care about standing up your own buildhosts, getting proper licences etc.

* Free as in money, they can eat up a lot of time while trying to work around their pecularities and figure out their configurations.

[refi64]

As an aside: as someone who has set up AppVeyor many times, I can safely say that it works extremely well. The support from both them and Travis is really awesome, too.

[horenmar]

The CI situation has improved (larger coverage on Travis, including C++11 builds) and is still improving (Cygwin on AppVeyor is coming soon, TeamCity is also slowly moving along), and the backlog is decreasing, so I am going to close this.