Instantiating empty library, adding filters and writing to HDF5 produces strange results

[pscicluna]

Hi!

We're (me and @sundarjhu) seeing some strange behaviour when trying to create new libraries. We are instantiating an empty UnitHDF_Library, then adding a number of UnitFilters to it using add_filter(), and trying to write it out. When attempting to read back in with get_library_content() or load_all_filters() we get the following exceptions:

Traceback (most recent call last):
  File "/scratch/home/psciclun/anaconda3/envs/sedutils/lib/python3.8/site-packages/pyphot/sandbox.py", line 1456, in get_library_content
    filters = s.hdf.root.content.cols.TABLENAME[:]
  File "/scratch/home/psciclun/anaconda3/envs/sedutils/lib/python3.8/site-packages/tables/group.py", line 836, in __getattr__
    return self._f_get_child(name)
  File "/scratch/home/psciclun/anaconda3/envs/sedutils/lib/python3.8/site-packages/tables/group.py", line 708, in _f_get_child
    self._g_check_has_child(childname)
  File "/scratch/home/psciclun/anaconda3/envs/sedutils/lib/python3.8/site-packages/tables/group.py", line 393, in _g_check_has_child
    raise NoSuchNodeError(
tables.exceptions.NoSuchNodeError: group ``/`` does not have a child named ``content``

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/scratch/home/psciclun/anaconda3/envs/sedutils/lib/python3.8/site-packages/pyphot/sandbox.py", line 1458, in get_library_content
    filters = list(s.hdf.root.filters._v_children.keys())
  File "/scratch/home/psciclun/anaconda3/envs/sedutils/lib/python3.8/site-packages/tables/group.py", line 836, in __getattr__
    return self._f_get_child(name)
  File "/scratch/home/psciclun/anaconda3/envs/sedutils/lib/python3.8/site-packages/tables/group.py", line 708, in _f_get_child
    self._g_check_has_child(childname)
  File "/scratch/home/psciclun/anaconda3/envs/sedutils/lib/python3.8/site-packages/tables/group.py", line 393, in _g_check_has_child
    raise NoSuchNodeError(
tables.exceptions.NoSuchNodeError: group ``/`` does not have a child named ``filters``

and opening the resulting library file with h5py to examine the contents shows that the file only appears to contain the root group, and indeed no filters group has been created (not does it seem to contain any datasets).

This is in a new conda environment with pyphot v1.1. Are we missing some step?

[mfouesneau]

That is a good question. Do you have a minimal example I could test?

[sundarjhu]

Hi Morgan, I've added you to a repository containing a file sedutils.py. We were using the function get_library to try and add filters into a freshly-created UnitHDF_Library, but the resulting .hd5 file only preserves the last filter added to it. While we're able to read the hd5 file in with h5py.File, as @pscicluna pointed out we're having issues with pyphot itself. Thanks!

[jvines]

Try adding the keyword mode='a' to whatever method you're using to write the filter to the library

[sundarjhu]

Thanks! That worked.

[pscicluna]

Unfortunately, I'm still seeing the same behaviour as the original comment on my end. Do you have any other suggestions? I'm pretty sure I have all dependencies fulfilled.

[jvines]

Maybe if you share an example to reproduce the issue I could try and help further

[pscicluna]

You have been invited to a repo with the code, but I also copy it below:

from astropy.table import Table
from io import BytesIO
import requests
from astropy.io import votable
import numpy as np
# from astropy import units
import asdf
from pyphot import sandbox as pyphot
import warnings
from tables import NaturalNameWarning
warnings.filterwarnings('ignore', category = NaturalNameWarning)

def make_FilterLibrary(filterFile = 'filters.csv', filterList = None, \
                      outfile = 'filters.asdf'):
    """
    Given a list of filter names, query them on the SVO FPS database
    and construct a library of said filters that is saved into an
    ASDF file.
    filterFile -- a CSV file containing at least one column
        named 'filtername' compatible with the filters available 
        through the SVO
    filterList -- a list of filter names compatible with the
        filters available through the SVO, fed in lieu of
        filterFile.
    outfile -- name of output ASDF file with extension to which
        filter information is saved.
    """
    if filterList is None:
        filters = Table.read(filterFile, format = 'csv')
        filterList = filters['filtername'].tolist()
    af = {'filters': {}}
    url = 'http://svo2.cab.inta-csic.es/theory/fps/fps.php'
    detectorTypes = ['energy', 'photon']
    for f in filterList:
        r = requests.get(url, params = {'ID': f})
        v = votable.parse_single_table(BytesIO(r.content))
        detectorType = detectorTypes[int(v.get_field_by_id('DetectorType').value)]
        t = v.to_table()
        t['Wavelength'] = t['Wavelength'].to('um')
        af['filters'][f] = {'name': f, 'Wavelength': t['Wavelength'].tolist(), \
                            'Transmission': np.array(t['Transmission']), \
                            'detectorType': detectorType, \
                            'Wavelength_unit': t['Wavelength'].unit}

    asdf.AsdfFile(af).write_to(outfile)

def get_library(libraryFile = 'filters.asdf', outfile = 'filters.hd5'):
    """
    Retrieve filter information from an ASDF library
    libraryFile -- input ASDF file (with extension) containing
        filter information
    filterLibrary -- list of pyphot UnitFilter objects, one
        for each filter in libraryFile
    """
    af = asdf.open(libraryFile)
    filterLibrary = pyphot.UnitHDF_Library(source = outfile, mode = 'w')
    um = pyphot.Unit['micron']
    for f in af['filters'].keys():
        print("Filter: {}.".format(af['filters'][f]['name'].replace('/', '_')))
        pf = pyphot.UnitFilter(np.array(af['filters'][f]['Wavelength']) * um, \
                               np.array(af['filters'][f]['Transmission']), \
                               dtype = af['filters'][f]['detectorType'], \
                               name = af['filters'][f]['name'].replace('/', '_'), \
                               unit = um)
        filterLibrary.add_filter(pf, mode = 'a')
    filterLibrary.__exit__()
    return filterLibrary

then I just import these functions and call them one after another.

[mfouesneau]

Thanks. FYI, v1.2.1 now has a SVO interface ;)

[mfouesneau]

the following example runs for me:

from pyphot.svo import get_pyphot_astropy_filter    # pyphot 1.2.1 (git)
from pyphot.astropy import UnitHDF_Library
lst = "2MASS/2MASS.J 2MASS/2MASS.H 2MASS/2MASS.Ks HST/ACS_WFC.F475W HST/ACS_WFC.F814W".split()
objects = [get_pyphot_astropy_filter(k) for k in lst]

with UnitHDF_Library('tmp.hdf5', 'a') as lib:
    for obj in objects:
        lib.add_filter(obj, append=True)

the append=True is important. Some default keyword missing somewhere.

[mfouesneau]

be1f06e should now have append=True as default.

from pyphot.svo import get_pyphot_astropy_filter    # pyphot 1.2.1 (git)
from pyphot.astropy import UnitHDF_Library
lst = "2MASS/2MASS.J 2MASS/2MASS.H 2MASS/2MASS.Ks HST/ACS_WFC.F475W HST/ACS_WFC.F814W".split()
objects = [get_pyphot_astropy_filter(k) for k in lst]

with UnitHDF_Library('tmp.hdf5', 'a') as lib:
    for obj in objects:
        lib.add_filter(obj, append=True)
lib.get_library_content()

Should print

['2MASS_2MASS.H',
 '2MASS_2MASS.J',
 '2MASS_2MASS.Ks',
 'HST_ACS_WFC.F475W',
 'HST_ACS_WFC.F814W']

[mfouesneau]

Release as v1.3 (also on pypi) @pscicluna can you check it works for you too?

[pscicluna]

Sorry for the delay, everything seems to work with v1.3! Thanks.

[mfouesneau]

Great. Closing issue