Auto-detect number of parts

[bmulholland]

We have one part per test suite per container, currently adding up to 12 parts. If any of those parts doesn't get to the end of the build for any reason, the test reporter crashes because the number of parts was set wrong. This gives a false impression of why the build failed (because of codeclimate instead of the original issue). If the reporter could just count how many files apply, then there wouldn't be a mismatch in expectations and there would be no confusion about why a build failed.

[bmulholland]

This actually just broke our build. I'm not sure of the details, but short version is that CircleCI retries nodes if there was a problem with them. In this case CircleCI rebuilt one of the nodes, apparently after it had submitted the results, and that meant that there were 15 (3 test suites x 5 containers) instead of 12 (3 x 4) that was expected.

From what I see in the source, the only place the number of parts is used is to throw the error. Why is that?

https://github.com/codeclimate/test-reporter/blob/8a0c606eed0d0b4bcd9b018d24beded504515c59/cmd/sum-coverage.go#L26

[larkinscott]

Hey @bmulholland 👋. I'm sorry this broke your build! Let me dig into this a little deeper and see what I can find out.

[larkinscott]

Hey @bmulholland. Sorry for the delay in getting back to you! Would you be able to provide a little more information about how you're writing the partial test results and what filenames they end up in? Are you using the index/name of the container in some way?

[bmulholland]

Yes, that's right. We've got three test suites, and each test suite is split across four containers. So I've named them codeclimate.<node number>.<test suite>.json. Ideally I'd just call the test reporter with something like codeclimate-test-reporter codeclimate.*.json without any parts argument. Aside from the extra reports issue I ran into, that's also because we may change the number of containers we have. It's a pain to have to change the number of parts to go along with that (actually, since it's not obviously connected in any way, it probably means a failing run every time we change it).

Honestly, I'm tempted to just write a small script to count the number of matching files and then pass that in to the test reporter... but this seems like something that the test reporter should handle on its own! This is the only example of a tool I can think of that requires you to pass in both the files and the exact number of files - seems redundant?

[jrnail23]

I was wondering about the same thing, so I just tried omitting the --parts option. It seems to work, and I can't really tell the difference between the summed file with or without it.