Memory usage >60GB when running merge step

[JulianPitney]

Our scan consists of 9 tiles with dimensions 2448x2048x1500. Is it normal to hit >60GB memory usage on the merge phase? Am I doing something wrong?

Note: Estimated memory usage is about 1.2GB before starting the merge process.

I tried setting the "XYZC Selection" to a smaller subvolume (5000x5000x1500) which worked for dropping the memory usage down to about 20GB, but the program now crashes after about 10 seconds with no error reported.

I managed to get it to work by setting "XYZC Selection" to a very low value (2000x2000x500). Any higher than that and it always crashes.

[iannellog]

No, memory occupancy should be less than 12 GB per channel (this is because there is an optimization that trades off the number of times files are opened and closed with memory occupancy).

Could you please send me the following information:

• how much memory you have
• the xml generated by TeraStitcher
• the total size of your complete dataset
• how many channels you have
• if you downloaded the binaries or you compiled the code with CMake (this question is because in your case there is perhaps a simple change to the code that may reduce memory occupancy).

[JulianPitney]

RAM: 64GB Total size of dataset: 126GB Channels: 1

I installed the most recent binaries. If you think building from source could help I'm happy to try. We'd really love to get this tool working for our analysis pipeline. Here's a GDrive link to the "xml_merging.xml" generated file (https://drive.google.com/open?id=1NgOj0SCgrevq2-yLTamCko5DlVg0l_U8)

[iannellog]

The problem is not the optimization. For some reasons your import step has problems and this is likely why merge aborts. From the xml I see that voxel dims in X-Y are 1.4 um and mechanical displacement are 11111.1 um in both dimensions. This means that TeraStitcher try to align tiles starting from a nominal displacement of 11111.1 * 1.43 = 7770 pixels, which is much more than tile size (2448 and 2048 in X or Y). Consequently, for TeraStitcher there is no overlapping between tiles, which is why alignment have (almost) always 0 reliability (I cannot explain why in a few cases reliability is relatively high). Actually TeraStitcher see a volume as large as about 18000 x 18000 x 1500 voxels which corresponds to a memory occupancy of much more than 60 GB (but remember that the problem is that tiles do not overlap and internal algorithms have been designed with no this case in mind). How have you imported the dataset? Have you used the two-level hierarchy of folders? Have you correctly named both folders and filenames (see https://github.com/abria/TeraStitcher/wiki/Supported-volume-formats#two-level-hierarchy-of-folders)? If your tiles are 2448x2048 in X-Y is not surprising that merge star fails with this xml (tile displacements turn out to be about 7770 pixels).

[JulianPitney]

Here is a link to 3 pictures showing the file structure: (https://drive.google.com/open?id=1cpY-Y8085lwIJxhqADPuS0TLVAp5_7D0) - I believe I followed the instructions correctly though please correct me if I messed it up somehow.

Here is a GDrive link to the generated file. (https://drive.google.com/open?id=1AbAqDEATTiIsvCXUCNn6GP-Thh8VuaKe)

[JulianPitney]

UPDATE: I somehow previously failed to notice the flashing yellow exclamation mark in the documentation that says:

"Coordinates FFFFFF, SSSSSS and ZZZZZZ are expressed in tenths of microns. However, you can use your own scale provided that you scale accordingly the voxel size parameters."

I had been treating the folder and file names as unit-less indices specifying the tile position in the 2D matrix. Going to try again with the names specifying the coordinates in microns relative to the origin. Hopefully that's the only thing I'm doing wrong and this clears everything up. I'll report back today or tomorrow with results. I appreciate the help!

[iannellog]

Pay attention that if voxel size is given in um, folder names are assumed to be TENTHs of um Hence if a tile is named 000000 and the adjacent one is shifted by 512 um, its name should be 005120 and NOT 000512

Il gio 14 nov 2019, 21:47 Julian Pitney notifications@github.com ha scritto:

UPDATE: I somehow previously failed to notice the flashing yellow exclamation mark that says "Coordinates FFFFFF, SSSSSS and ZZZZZZ are expressed in tenths of microns. However, you can use your own scale provided that you scale accordingly the voxel size parameters."

I had been treating the folder and file names as unit-less indexes specifying the tile position in the 2D matrix. Going to try again with the names instead specifying the coordinates in microns relative to the origin. Hopefully that's the only thing I'm doing wrong and this clears everything up. I'll report back today or tomorrow with results. I appreciate the help!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/abria/TeraStitcher/issues/61?email_source=notifications&email_token=ACDW7VXUCHTQ6HRYVZW6BQ3QTW2MJA5CNFSM4JNAY6FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEDHSJQ#issuecomment-554072358, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACDW7VWTJEHYAWV4IR2H4KDQTW2MJANCNFSM4JNAY6FA .

[JulianPitney]

After fixing the names of the folders everything worked beautifully. Thanks a lot for your help!