astropy / halotools

Python package for studying large scale structure, cosmology, and galaxy evolution using N-body simulations and halo models
http://halotools.rtfd.org
98 stars 62 forks source link

CacheManager fails with h5py 2.8.0 #912

Open rainwoodman opened 6 years ago

rainwoodman commented 6 years ago
(base) [yfeng1@waterfall nbodykit]$ rebuild_halo_table_cache_log.py

Number of files detected in standard cache location = 1

Each of these files must be opened, and the metadata will be checked, 
and consistency between the metadata and halo table will be checked.
Estimated runtime = 3 seconds.

Number of files passing verification tests = 0

Number of files that fail verification tests = 1

There were some filenames that were rejected and will NOT be added to your new cache log.
The reason for the rejection appears below each filename.
-----------------------------------
Rejected file #1

/home/yfeng1/.astropy/cache/halotools/halo_catalogs/bolshoi/rockstar/hlist_0.67035.list.halotools_v0p4.hdf5
The halo catalog and/or its associated metadata fail the following tests:

1. The hdf5 file must be readable with Astropy 
using the following syntax:

>>> halo_data = Table.read(fname, path='data')

2. The halo table must at a minimum have the following columns:
``halo_id``, ``halo_x``, ``halo_y``, ``halo_z``,
plus at least one additional column storing a mass-like variable.

These rejected filenames are now stored in the following location:
/home/yfeng1/.astropy/cache/halotools/rejected_halo_table_filenames.txt

The file is downloaded by looks like the underlying astropy stuff fails to read the hdf5 file for some reason? Should this bug be forwarded to astropy?

rainwoodman commented 6 years ago

It is because astropy uses a deprecated h5py module. The fix is in but no new releases yet.

rainwoodman commented 6 years ago

I wonder why the CI of halotools didn't catch this -- it seems to be major failure.

aphearin commented 6 years ago

Thanks for pointing this out @rainwoodman - this is definitely a bug and a failure of unit-testing. Release v0.7 will be out very soon, I'll make sure a patch makes it into the release.