Appveyor failing with "Access denied"

[mhvk]

Many appveyor checks seem to fail recently with failures like:

astropy/io/misc/tests/test_hdf5.py ..error: [WinError 5] Access is denied: 'C:\\Users\\appveyor\\AppData\\Local\\Temp\\astropy-test-dbbgg2vy\\lib.win-amd64-3.4\\astropy\\convolution\\boundary_extend.pyd'
Command exited with code 1

[mwcraig]

Looks like these fails occur only on python 3.4, the first one was a few days ago in #3860. Restarting that one to see what happens...

[mwcraig]

Does anyone have a local python 3 Windows they can try this on? The error seems unlikely to actually be related to hdf5, though...if no one can reproduce it locally, I'll ask the appveyor people...

[mhvk]

@mwcraig - thanks for looking into it. The error does not always occur: eg., no problem with #3224, but in (very related) #3681, it did happen again, and in the same place: https://ci.appveyor.com/project/Astropy/astropy/build/1.0.1544/job/w27aosclr6kfmpbw#L2688

[astrofrog]

I compared the logs for the last build that succeeded with the next one and it seems the only difference is that now the HDF5 library is installed separately from the h5py:

    package                    |            build
    ---------------------------|-----------------
    zlib-1.2.8                 |                0         116 KB
    hdf5-1.8.15.1              |                1         1.4 MB

so it has to be related to this.

[mhvk]

Nice tracking down!

[mwcraig]

@astrofrog -- I'm not entirely sure it really is HDF5-related. The first failure with this message is in appveyor build 1529, where it occurs in a different test. The next build, appveyor 1530 on the same PR, passes.

Part of the reason I wonder if this is really HDF5 related is that I've seen several intermittent Access errors like this over the last couple of weeks in building affiliated packages on a local windows machine.

In any event, #3876 may help narrow this down.

[mwcraig]

@astrofrog is correct about the fail being HDF5-related.

A few days ago the conda package for h5py for Windows was updated to list HDF5 as a dependency; prior to that it was not listed at all in the Windows h5py dependencies, even though it is listed in the mac packages for h5py (I only checked a few earlier packages of h5py 2.5.0 and 2.4.0).

It isn't clear to me how the HDF5 tests ran at all in Windows in the past, given that HDF5 wasn't listed as a requirement.....

In any event, next step is to start skipping tests on Windows.

[astrofrog]

@mwcraig - I think h5py was installed before, but it started failing when the HDF5 library itself started being installed. But indeed I'm not sure how h5py worked before then. Maybe it used to somehow bundle the HDF5 libraries or something like that? The tests were definitely running.

@asmeurer - did anything change from the conda side re: HDF5 on Windows 3-4 days ago?

[mwcraig]

@astrofrog @asmeurer -- it looks like there was a version bump to HDF5, from 1.8.14 to 1.8.15.1

[astrofrog]

@mwcraig - in that case an easy workaround is to add hdf5=1.8.14 to the conda requirement list.

[mwcraig]

@astrofrog -- first attempt at hdf5<1.8.15 failed (see https://ci.appveyor.com/project/Astropy/astropy/build/1.0.1556) Most recent Windows version of HDF5 is 1.8.13. Can test more this evening, in the meantime maybe disable all HDF5 tests on Windows/Python 3? (Can't do myself right now).

[embray]

I used to see this sometimes when testing on Windows manually, and it was really annoying. See #1765.

I don't think it has anything to do with h5py or hdf5--note that in the test log quoted in #1765 it isn't even installed.

This seems to be a problem the interpreter is having with freeing certain modules in the convolution package, perhaps because they are somehow still in use. This in turn prevents much of the temporary test directory from being deleted.

[mwcraig]

I agree that it fundamentally isn't an HDF5 issue, but it looks like pinning the HDF5 version to 1.8.13 makes the builds succeed. If that fix works, I'll open both a PR for that workaround and an issue to try to resolve the underlying issue.

[asmeurer]

Is there anything that we need to do on our side?

[mwcraig]

@asmeurer -- I don't think so, at this point. I haven't been able to reproduce the problem locally, and we have workaround on appveyor now.

[mwcraig]

Closed by #3882