The generated data is black

[supersteven]

Hi, all

I test the ImarisWriterTest.cxx, the simulated random data is all black, why?

[imaris]

Hi, do you mean that the content of vFileBlock is all zero, or the content of the image file generated is all black? Neither should be... Does ImarisWriterTestSimple.cxx work as expected or does it have the same problem?

[supersteven]

the content of the image file generated is all black. Then I test the ImarisWriterTestSimple.cxx, it works as expected. If I want to convert my data to ims, It seems that I should write like the following?

for(all blocks) { copy(vBlockData, myDataBlock, blockSize); //this funciotn is very slow vImageConverter::CopyBlock(vBlockData, vBlockPosition); }

严程

From: igor Date: 2022-09-05 17:43 To: imaris/ImarisWriterTest CC: Cheng Yan; Author Subject: Re: [imaris/ImarisWriterTest] The generated data is black (Issue #5) Hi, do you mean that the content of vFileBlock is all zero, or the content of the image file generated is all black? Neither should be... Does ImarisWriterTestSimple.cxx work as expected or does it have the same problem? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[imaris]

Copying of the data should not make a difference, the converter takes what is given as a pointer. Is the code of ImarisWriter up-to-date? If you have Imaris 9.8 or Imaris 9.9 installed, you can copy bpImarisWriter98.dll or bpImarisWriter99.dll, copy/rename it at the place of bpImarisWriter96.dll to check if there is a problem with how the library has been compiled.

[supersteven]

Mybe you misunderstand me. In the follow code:

for(all blocks) { copy(vBlockData, myDataBlock, blockSize); //this funciotn is very slow vImageConverter::CopyBlock(vBlockData, vBlockPosition); }

the copy() funciton contains the procedure of copying my own data block into the vBlockData, like following:

for (bpSize vVoxelIndex = 0; vVoxelIndex < vBlockSize; ++vVoxelIndex) { int zIndex = vBlockIndexZ vBlockSize5D[bpConverterTypes::Z] + vVoxelIndex / (vBlockSize5D[bpConverterTypes::X] vBlockSize5D[bpConverterTypes::Y]); int yIndex = vBlockIndexY vBlockSize5D[bpConverterTypes::Y] + (vVoxelIndex / vBlockSize5D[bpConverterTypes::X]) % vBlockSize5D[bpConverterTypes::Y]; int xIndex = vBlockIndexX vBlockSize5D[bpConverterTypes::X] + vVoxelIndex % vBlockSize5D[bpConverterTypes::X];

    if (zIndex < vImageSize[bpConverterTypes::Z] && yIndex < vImageSize[bpConverterTypes::Y] && xIndex < vImageSize[bpConverterTypes::X])
    {
        int vRealImageVoxelIndex = zIndex * vImageSize[bpConverterTypes::X] * vImageSize[bpConverterTypes::Y] + yIndex * vImageSize[bpConverterTypes::X] + xIndex;
        vFileBlock[vVoxelIndex] = buffer8[vRealImageVoxelIndex];
    } 

} Is there any other way to speed the copy() procedure?

严程

From: igor Date: 2022-09-05 18:22 To: imaris/ImarisWriterTest CC: Cheng Yan; Author Subject: Re: [imaris/ImarisWriterTest] The generated data is black (Issue #5) Copying of the data should not make a difference, the converter takes what is given as a pointer. Is the code of ImarisWriter up-to-date? If you have Imaris 9.8 or Imaris 9.9 installed, you can copy bpImarisWriter98.dll or bpImarisWriter99.dll, copy/rename it at the place of bpImarisWriter96.dll to check if there is a problem with how the library has been compiled. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[imaris]

If buffer8 contains the data of the entire image, you can specify blockSize = imageSize and call CopyBlock only once, passing buffer8 as data pointer without any copy. You can specify blockSize to be equal to the size of the content of buffer8 and pass all of it at once, even if it is not the entire image. Passing the data in blocks is convenient when acquiring tiles of large images, but one can choose how big the blocks are (this may be limited by how much can be held in RAM). The library internally builds a file structured in blocks (for faster reading) in any case, but the block size specified in the API and the block size in the ims file are independent.

[supersteven]

Thank you, it has been work well by specifying the blockSize = imageSize of buffer8. We just want to use the library during the acquring tiles of large images. It seems that we must know the total image size of total 3D volume first, and set the vImageSize parameter in the vImageConverter correctly before each blocl copy of buffer8? In our application, we don't know the total 3D volume size at first, so how can we do?

严程

From: igor Date: 2022-09-05 19:22 To: imaris/ImarisWriterTest CC: Cheng Yan; Author Subject: Re: [imaris/ImarisWriterTest] The generated data is black (Issue #5) If buffer8 contains the data of the entire image, you can specify blockSize = imageSize and call CopyBlock only once, passing buffer8 as data pointer without any copy. You can specify blockSize to be equal to the size of the content of buffer8 and pass all of it at once, even if it is not the entire image. Passing the data in blocks is convenient when acquiring tiles of large images, but one can choose how big the blocks are (this may be limited by how much can be held in RAM). The library internally builds a file structured in blocks (for faster reading) in any case, but the block size specified in the API and the block size in the ims file are independent. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[imaris]

We don't have a good solution when the total image size is not known in advance, unfortunately. One way is to write the blocks in raw data files, and then use the converter to create a single big ims file reading (and deleting) the raw data files. A similar solution would be to write every block in a different file using the converter, and then create a single big ims file using ImarisStitcher (not free, unfortunately: https://imaris.oxinst.com/products/imaris-stitcher).

I am sorry we cannot provide a better solution, the library needs to know the entire image size in advance to compute the multiresolution images during copying. A future improvement for the library I can think of is to allocate an image that can be expanded and delay the computation of the multiresolution images until all data has been acquired and the image size is known.

[supersteven]

Thank you very much for your nicely help. Good luck for you and Imaris

严程

From: igor Date: 2022-09-05 22:39 To: imaris/ImarisWriterTest CC: Cheng Yan; Author Subject: Re: [imaris/ImarisWriterTest] The generated data is black (Issue #5) We don't have a good solution when the total image size is not known in advance, unfortunately. One way is to write the blocks in raw data files, and then use the converter to create a single big ims file reading (and deleting) the raw data files. A similar solution would be to write every block in a different file using the converter, and then create a single big ims file using ImarisStitcher (not free, unfortunately: https://imaris.oxinst.com/products/imaris-stitcher). I am sorry we cannot provide a better solution, the library needs to know the entire image size in advance to compute the multiresolution images during copying. A future improvement for the library I can think of is to allocate an image that can be expanded and delay the computation of the multiresolution images until all data has been acquired and the image size is known. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[imaris]

Glad to help, I hope the library will serve you well.

[supersteven]

I'm sorry, another problem. It seems that current library can not support append mode after an abnormal crash during last convert?

严程

From: igor Date: 2022-09-06 14:04 To: imaris/ImarisWriterTest CC: Cheng Yan; Author Subject: Re: [imaris/ImarisWriterTest] The generated data is black (Issue #5) Closed #5 as completed. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[imaris]

You are right, it doesn't. There is no API and no code internally to support this. It would be a valuable feature, but it will be hard to find out which part has been acquired correctly and which part of the multiresolution images needs to be recomputed. Another problem is that hdf5 produces corrupt files in case of a crash, unless H5_flush is called regularly. H5_flush has a very negative impact on perfromance, unfortunately.

[supersteven]

I see. thank you again Best wishes~

严程

From: igor Date: 2022-09-06 14:31 To: imaris/ImarisWriterTest CC: Cheng Yan; Author Subject: Re: [imaris/ImarisWriterTest] The generated data is black (Issue #5) You are right, it doesn't. There is no API and no code internally to support this. It would be a valuable feature, but it will be hard to find out which part has been acquired correctly and which part of the multiresolution images needs to be recomputed. Another problem is that hdf5 produces corrupt files in case of a crash, unless H5_flush is called regularly. H5_flush has a very negative impact on perfromance, unfortunately. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[supersteven]

Another problem. In current library, if the total image size is larger than the RAM, it seems that we should set the blockSize to a certain size that the whole image size can be splitted to in average? Otherwise we must set another blockSize in the last section of image data.

严程

From: 严程 Date: 2022-09-05 22:42 To: imaris/ImarisWriterTest; imaris/ImarisWriterTest CC: Author Subject: Re: Re: [imaris/ImarisWriterTest] The generated data is black (Issue #5) Thank you very much for your nicely help. Good luck for you and Imaris

严程

From: igor Date: 2022-09-05 22:39 To: imaris/ImarisWriterTest CC: Cheng Yan; Author Subject: Re: [imaris/ImarisWriterTest] The generated data is black (Issue #5) We don't have a good solution when the total image size is not known in advance, unfortunately. One way is to write the blocks in raw data files, and then use the converter to create a single big ims file reading (and deleting) the raw data files. A similar solution would be to write every block in a different file using the converter, and then create a single big ims file using ImarisStitcher (not free, unfortunately: https://imaris.oxinst.com/products/imaris-stitcher). I am sorry we cannot provide a better solution, the library needs to know the entire image size in advance to compute the multiresolution images during copying. A future improvement for the library I can think of is to allocate an image that can be expanded and delay the computation of the multiresolution images until all data has been acquired and the image size is known. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[imaris]

The block size does not need to be an integer divisor of the total image size, but if it is not, you will have to add padding data (a black border) to the blocks at the border to fill them. For example, if the imge is 10x5 and block size is 4x3, the data passed for the 6 blocks covering the entire image will look like this: 1111 2222 3300 1111 2222 3300 1111 2222 3300

4444 5555 6600 4444 5555 6600 0000 0000 0000

All blocks need to provide 4x3 values, so the last block will require data 660066000000 instead of 6666, (The padding data does not need to be zero, it will be ignored in any case.)