Open WardF opened 5 years ago
It's worth noting that the issues don't appear to show up in the Jenkins tests run by @edhartnett, nor do they show up on the private Jenkins install I've set up.
I use --disable-dap-remote-tests in many of my CI builds on Jenkins. If I have more than a few jobs trying to run DAP tests at the same time, I frequently get a timeout on tst_remote.sh.
So perhaps try that on a bunch of the travis test cases, and let only one of them actually to DAP remote tests. If that fixes it, you can expand it to 2 or 3.
(Or is there a way to lengthen the timeout of the tests in tst_remote.sh? Or is it just one test in tst_remote,sh, which is getting a particularly big chunk of data?)
You can increase the timeout by creating a file called .dodsrc using e.g. cat >.dodsrc <<EOF HTTP.TIMEOUT=dddd EOF where dddd is the timeout in seconds.
If the issue turns out to be a timeout, that would make sense to address it as described, although we are only running a small number of concurrent tests. I'll be really dismayed if 4 people running 'make check' at once is enough to cause problems with our test servers.
Thanks @DennisHeimbigner. It occured to me "Hey, actually, I thought I disabled DAP testing in travis" so now I'm reviewing how the test scripts parse the options passed to Docker at test time.
We are seeing a lot of temporary (apparently) breakage in TravisCI. This is causing problems managing pull requests; manually restarting failing tests is time consuming, and there's no reason to need to do this in the first place, especially since the failures are seldom reproducible locally.
The issues tend to be centered on remote tests. I posit the issue is either an instability in the Travis infrastructure, instability in the remote test server, or instability in the Docker configuration used for the Travis tests. I'm working on trying to narrow this down, so that we can get the 4.7.1 release out the door and move on with the C++ and Fortran releases.