HDF5 pytables encoding

[svdhoog]

1 Make sure that writing HDF5 files with pytables is using the same format when encoding and decoding (current code uses utf-8 or unicode by default?)

See: https://github.com/pandas-dev/pandas/issues/11126

2 Compatibility between HDF5 and xarray See here: http://xarray.pydata.org/en/stable/io.html

"Chunk based compression zlib, complevel, fletcher32, continguous and chunksizes can be used for enabling netCDF4/HDF5’s chunk based compression"

Settings in the script merge_hdf_agentwise

            pd.set_option('io.hdf.default_format','table')
...
            #store_out = pd.HDFStore(outFileName, 'w')  # to store without compression
            store_out = pd.HDFStore(outFileName, 'w', chunksize = 500, complevel = 1, complib ='bzip2', fletcher32 = True) # store with compression

Error:

• This error was probably caused by different settings in the script that writes out h5 versus the script that reads-in h5 files:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc8 in position 0: unexpected end of data

• See: db_hdf5_2.py, line 81:

store = pd.HDFStore(fname_tostore, 'w', chunksize = 500, complevel = 1, complib ='bzip2', fletcher32 = True)

Full details:

• using a large h5 file of 21 GB
• only one agent, Eurostat
• trying to read 2 sets of 513 sets
• Could the problem be caused by the fact that the h5 file was created using the python2 version of the scripts, while now we are using the python3 version to do the read-in of the h5 file?

Traceback (most recent call last):
  File "/home/svdhoog/files-server/GIT/GitHub/FLAViz@0xfabi/ETACE/src/visualisation_scripts/main.py", line 141, in <module>
    agent_storelist[key] = pd.io.pytables.HDFStore(f_p)
  File "/usr/local/lib/python3.4/dist-packages/pandas/io/pytables.py", line 464, in __init__
    self.open(mode=mode, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/pandas/io/pytables.py", line 628, in open
    raise e
  File "/usr/local/lib/python3.4/dist-packages/pandas/io/pytables.py", line 603, in open
    self._handle = tables.open_file(self._path, self._mode, **kwargs)
  File "/usr/lib/python3/dist-packages/tables/file.py", line 318, in open_file
    return File(filename, mode, title, root_uep, filters, **kwargs)
  File "/usr/lib/python3/dist-packages/tables/file.py", line 826, in __init__
    root._g_post_init_hook()
  File "/usr/lib/python3/dist-packages/tables/group.py", line 265, in _g_post_init_hook
    if 'VERSION' in self._v_attrs._v_attrnamessys:
  File "/usr/lib/python3/dist-packages/tables/utils.py", line 244, in newfget
    mydict[name] = value = fget(self)
  File "/usr/lib/python3/dist-packages/tables/node.py", line 174, in _v_attrs
    return self._AttributeSet(self)
  File "/usr/lib/python3/dist-packages/tables/attributeset.py", line 243, in __init__
    self.__getattr__(attr)
  File "/usr/lib/python3/dist-packages/tables/attributeset.py", line 295, in __getattr__
    value = self._g_getattr(self._v_node, name)
  File "hdf5extension.pyx", line 755, in tables.hdf5extension.AttributeSet._g_getattr (tables/hdf5extension.c:7234)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc8 in position 0: unexpected end of data
Closing remaining open files:/home/svdhoog/backup/Big_Archive_of_Everything/Estimation_and_Calibration/Data/calibration-mode-3/h5_agentwise/Eurostat.h5...done

[svdhoog]

To create new full h5 file with merge_hdf_agentwise and check if plotting from this 21 GB file now works.

[svdhoog]

• The code now explicitly uses 'utf-8' encoding for the agent names. Since these age being used as keys in the HDFStores, this should resolve the issue above.