Error thrown when wrirting large output files: Number of elements in chunk must be < 4GB

I am trying to run binned images on 5000 wavelength points and the code throws an error when writing out the image file.

The error is thrown because the maximum chunk size of HDF5 is 4GB. Is there a way to set a lower chunk size?

The full error traceback is as follows:

 [main] exiting raytracing iteration
 [image_write] writing out images
HDF5-DIAG: Error detected in HDF5 (1.10.1) thread 0:
  #000: H5Pdcpl.c line 2019 in H5Pset_chunk(): number of elements in chunk must be < 4GB
    major: Invalid arguments to routine
    minor: Out of range
HDF5-DIAG: Error detected in HDF5 (1.10.1) thread 0:
  #000: H5D.c line 145 in H5Dcreate2(): unable to create dataset
    major: Dataset
    minor: Unable to initialize object
  #001: H5Dint.c line 490 in H5D__create_named(): unable to create and link to dataset
    major: Dataset
    minor: Unable to initialize object
  #002: H5L.c line 1695 in H5L_link_object(): unable to create new link to object
    major: Links
    minor: Unable to initialize object
  #003: H5L.c line 1939 in H5L_create_real(): can't insert link
    major: Symbol table
    minor: Unable to insert object
  #004: H5Gtraverse.c line 867 in H5G_traverse(): internal path traversal failed
    major: Symbol table
    minor: Object not found
  #005: H5Gtraverse.c line 639 in H5G_traverse_real(): traversal operator failed
    major: Symbol table
    minor: Callback failed
  #006: H5L.c line 1742 in H5L_link_cb(): unable to create object
    major: Object header
    minor: Unable to initialize object
  #007: H5O.c line 3178 in H5O_obj_create(): unable to open object
    major: Object header
    minor: Can't open object
  #008: H5Doh.c line 291 in H5O__dset_create(): unable to create dataset
    major: Dataset
    minor: Unable to initialize object
  #009: H5Dint.c line 1202 in H5D__create(): filters can only be used with chunked layout
    major: Dataset
    minor: Bad value
HDF5-DIAG: Error detected in HDF5 (1.10.1) thread 0:
  #000: H5Dio.c line 226 in H5Dwrite(): not a dataset
    major: Invalid arguments to routine
    minor: Inappropriate type
HDF5-DIAG: Error detected in HDF5 (1.10.1) thread 0:
  #000: H5D.c line 332 in H5Dclose(): not a dataset
    major: Invalid arguments to routine
    minor: Inappropriate type
 -------------------------------------------------------------------------------
  HDF5 returned an error in hdf5_write_k_h5t_ieee_f64le [2]
  See above for traceback

I have figured out a workaround. I want to run it by you to make sure that this makes sense.

As an example, consider I want peeled images for a galaxy with 4500 wavelength points between 0.3 to 15 microns. I first divide it up into chunks of 500 wavelength points. With chunk 1 going from 0.3 to 0.45, chunk 2 going from 0.45 to 0.7, and so on and so forth. Then I call add_peeled_images() for each chunk by setting the wavelength range accordingly.

lam_lim = [0.3, 0.47, 0.7....,15]
for k in range(len(lam_lim)-1):
        image = m_imaging.add_peeled_images(sed=False, image=True)
        image.set_wavelength_range(500, lam_lim[k], lam_lim[k+1])

m_imaging.write(model.inputfile+'.image', overwrite=True)
m_imaging.run(model.outputfile+'.image', mpi=True, n_processes=par.n_MPI_processes, overwrite=True)

By doing so Hyperion saves the info for each chunk in a different group in the rtout file. Which I can then just access by passing the group number to the get_image() function. Does this approach seem reasonable? I want to make sure it's not breaking anything in the background that I should be worried about.

Thanks

hyperion-rt / hyperion

Error thrown when wrirting large output files: Number of elements in chunk must be < 4GB #228