saving metadata is not consistent

[fjorka]

Hi, I'm using PyImarisWriter following the example that you provided but I have troubles with saving metadata consistently. The longer the time series, the more probable that the attributes like 'HistogramMax' will not be saved for later time points (below you can see that what was saved for time point 9 is different than for time point 8). Any suggestions how to make it perform better?

The script below was used to save test numpy arrays:

Windows 10 PyImarisWriter 0.0.7

import os
import h5py
import numpy as np
from datetime import datetime
from PyImarisWriter import PyImarisWriter as PW
import warnings
warnings.filterwarnings(action='always')

class MyCallbackClass(PW.CallbackClass):
    def __init__(self):
        self.mUserDataProgress = 0

    def RecordProgress(self, progress, total_bytes_written):
        progress100 = int(progress * 100)
        if progress100 - self.mUserDataProgress >= 5:
            self.mUserDataProgress = progress100
            print('User Progress {}, Bytes written: {}'.format(self.mUserDataProgress, total_bytes_written))

def create_color_table(rgb):

    color_table = np.zeros([256,4])

    color_table[:,0] = np.linspace(0,rgb[0]/255,256)
    color_table[:,1] = np.linspace(0,rgb[1]/255,256)
    color_table[:,2] = np.linspace(0,rgb[2]/255,256)
    color_table[:,3] = 1 

    # change to imaris friendly format
    color_table = [PW.Color(*x) for x in color_table]

    return color_table

color_table = create_color_table([0,255,255])

results = np.zeros([10,2,1,2000,2000]).astype('uint16')
results[0,0,0,:,:] = 1
results[0,1,0,:,:] = 60

channels_names_selected = ['test_ch0','test_ch1']

x = results.shape[4]
y = results.shape[3]
z = results.shape[2]
c = results.shape[1]
t = results.shape[0]

im_resolution = (0.1, 0.1, 0.1)
im_dtype = np.dtype(np.uint16)

im_extent = np.array([z,y,x])*np.array(im_resolution)

image_size = PW.ImageSize(x=x, y=y, z=z, c=c, t=t)

dimension_sequence = PW.DimensionSequence('x','y','z','c','t')
sample_size = PW.ImageSize(x=1, y=1, z=1, c=1, t=1)
block_size = image_size
output_filename = 'test.ims'

options = PW.Options()
options.mNumberOfThreads = 8
options.mCompressionAlgorithmType = PW.eCompressionAlgorithmGzipLevel2
options.mEnableLogProgress = True

application_name = 'PyImarisWriter'
application_version = '0.0.7'

callback_class = MyCallbackClass()
converter = PW.ImageConverter(im_dtype.name, image_size, sample_size, dimension_sequence, block_size,
                              output_filename, options, application_name, application_version, callback_class)

num_blocks = image_size / block_size

block_index = PW.ImageSize()
for c in range(num_blocks.c):
    block_index.c = c
    for t in range(num_blocks.t):
        block_index.t = t
        for z in range(num_blocks.z):
            block_index.z = z
            for y in range(num_blocks.y):
                block_index.y = y
                for x in range(num_blocks.x):
                    block_index.x = x
                    if converter.NeedCopyBlock(block_index):
                        converter.CopyBlock(results, block_index)

adjust_color_range = True
image_extents = PW.ImageExtents(0, 0, 0, im_extent[2], im_extent[1], im_extent[0])

parameters = PW.Parameters()
color_infos = [PW.ColorInfo() for _ in range(image_size.c)]
for i,ch in enumerate(channels_names_selected):
    parameters.set_channel_name(i, ch)

    color_infos[i].set_color_table(color_table)

time_infos = [datetime.today()]

# sending metadata
converter.Finish(image_extents, parameters, time_infos, color_infos, adjust_color_range)

converter.Destroy()

print('Results have been saved.')

While using the writer I get warnings: sys:1: ResourceWarning: unclosed socket <zmq.Socket(zmq.PUSH) at 0x2db684769e8> ResourceWarning: Enable tracemalloc to get the object allocation traceback but they are displayed even if the file is saved correctly.

[imaris]

Thank you for the detailed report, this is something that will need to be fixed inside the library. Pausing the execution of the script for just 1 second before calling converter.Finish seems to avoid the issue:

import time
[...]
time_infos = [datetime.today()]

time.sleep(1)

# sending metadata
converter.Finish(image_extents, parameters, time_infos, color_infos, adjust_color_range)

On pcs other than mine or for different image or block sizes it may be necessary to wait longer, but we had no reports of this issue for other image sizes.

According to this answer in stackoverflow, the ResourceWarning is due to the setup of the development environment.

[fjorka]

It solved my problem - thank you!