search

abria / TeraStitcher

A tool for fast automatic 3D-stitching of teravoxel-sized microscopy images
http://abria.github.io/TeraStitcher/
Other
82 stars 32 forks source link

Long time taken during import step #64

Closed snehashis-roy closed 4 years ago

snehashis-roy commented 4 years ago

Hello. While running terastitcher on some images, the import step takes about a second, but on some other image sets, it takes ~1hr, e.g. the following.

This is the stitching command I am using, TeraStitcher-portable-1.11.6-Linux/terastitcher --import --volin=c0Decon_rescale_stitched/symlinks/ --volin_plugin="TiledXY|3Dseries" --volout=c0Decon_rescale_stitched/result --projout=c0Decon_rescale_stitched/result/import.xml --ref1=Y --ref2=X --ref3=Z --vxl1=1.2081 --vxl2=1.2081 --vxl3=1.0000 --imin_plugin=tiff3D

I have a 2x9 mosaic data, with the following folder structure,

image

Each sub-directory contains a 343x2241x3453 3D bigtiff image with resolutions 1.2x1.2x1.0 um. On this particular image set, the import step takes ~1hr. Is this normal or am I doing something wrong? Thank you for your help.

iannellog commented 4 years ago

I understand that you have 2x9=18 tiles each consisting of a single multi-page TIFF of 343x2241x3453 = 2654193339 voxels. I assume that color depth is 16 bits, so the total size of each tile is more than 5 GBytes, which justifies the use of BigTiff format. One question is: how many pages are there in the multi-page TIFFS? It is likely they are 343, but please confirm it to me. I have never tried such a configuration and it is possible that there is some inefficiency in dealing with such big 3D files. Actually, TeraStitcher needs to know the number of pages (i.e. how many planes there are in Z). It is common that multi-page TIFFs do not store this information (there exists a PAGENUMBER tag, but it is not completely standard). In this case, our code go through all pages until it finds the end of the file. This could be expensive for big files. This is just an hypothesis. As soon as I have time I will test it. Best.

-- Giulio

Il giorno lun 3 feb 2020 alle ore 21:14 piby2 notifications@github.com ha scritto:

Hello. While running terastitcher on some images, the import step takes about a second, but on some other image sets, it takes ~1hr, e.g. the following.

This is the stitching command I am using, TeraStitcher-portable-1.11.6-Linux/terastitcher --import --volin=c0Decon_rescale_stitched/symlinks/ --volin_plugin="TiledXY|3Dseries" --volout=c0Decon_rescale_stitched/result --projout=c0Decon_rescale_stitched/result/import.xml --ref1=Y --ref2=X --ref3=Z --vxl1=1.2081 --vxl2=1.2081 --vxl3=1.0000 --imin_plugin=tiff3D

I have a 2x9 mosaic data, with the following folder structure,

[image: image] https://user-images.githubusercontent.com/27572901/73687036-30be3800-4697-11ea-9b39-bc9c99c2e508.png

Each sub-directory contains a 343x2241x3453 3D bigtiff image with resolutions 1.2x1.2x1.0 um. On this particular image set, the import step takes ~1hr. Is this normal or am I doing something wrong? Thank you for your help.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/abria/TeraStitcher/issues/64?email_source=notifications&email_token=ACDW7VU4XAYXKKV43OZRXATRBB3JNA5CNFSM4KPLLJNKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IKWLWNQ, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACDW7VREIC6KXX5H3QQO663RBB3JNANCNFSM4KPLLJNA .

--

Giulio Iannello Preside della Facolta' Dipartimentale di Ingegneria Universita' Campus Bio-Medico di Roma v. Alvaro del Portillo, 21 00128 Roma, Italy

Tel: +39-06-22541-9602 E-mail: g.iannello@unicampus.it Fax: +39-06-22541-9609 URL: https://scholar.google.it/citations?user=L-UJxIgAAAAJ

snehashis-roy commented 4 years ago

Sorry for the confusion. The size 343x2241x3453 actually refers to WxHxD, so there are 3453 slices in the z direction. Each tile is 5GB with 16 bit data. There are total 2x9=18 tiles. Is the slowdown caused by so many slices?

Is there a good way to create bigtiff files so that the tags can be created properly? I am creating bigtiff files using matlab (https://www.mathworks.com/matlabcentral/fileexchange/35684-multipage-tiff-stack), which has the bigtiff option. Please let me know if there is a better way to write the bigtiff images, so that the PAGENUMBER tag is correctly used.

iannellog commented 4 years ago

If the pages are 3453 (actually I feared that was the case, but of course I could not be sure :-) ) it is likely that the reason is what I assumed in my previous comment.

Try the following. In function'saveastiff' change lines_

for d = 1:depth tfile.setTag(tagstruct); tfile.write(data(:, :, :, d)); if d ~= depth tfile.writeDirectory(); end end

as follows:

for d = 1:depth tagstruct.PageNumber = [d, depth]; tfile.setTag(tagstruct); tfile.write(data(:, :, :, d)); if d ~= depth tfile.writeDirectory(); end end

Let me know if this works and reduces the import time. Best.

-- Giulio

Il giorno mar 4 feb 2020 alle ore 17:36 piby2 notifications@github.com ha scritto:

Sorry for the confusion. The size 343x2241x3453 actually is WxHxD, so there are 3453 slices in the z direction. The file size is indeed 5GB with 16 bit data. Is the slowdown caused by so many slices?

Is there a good way to create bigtiff files so that the tags can be created properly? I am creating this bigtiff files using matlab ( https://www.mathworks.com/matlabcentral/fileexchange/35684-multipage-tiff-stack), which has the bigtiff option. Please let me know if there is a better way to write the bigtiff images, so that the PAGENUMBER tag is correctly used.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/abria/TeraStitcher/issues/64?email_source=notifications&email_token=ACDW7VSBRTIUJLEFUOLYPC3RBGKSJA5CNFSM4KPLLJNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKYJMZQ#issuecomment-581998182, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACDW7VVCUMYQAOK3OAKZZKDRBGKSJANCNFSM4KPLLJNA .

--

Giulio Iannello Preside della Facolta' Dipartimentale di Ingegneria Universita' Campus Bio-Medico di Roma v. Alvaro del Portillo, 21 00128 Roma, Italy

Tel: +39-06-22541-9602 E-mail: g.iannello@unicampus.it Fax: +39-06-22541-9609 URL: https://scholar.google.it/citations?user=L-UJxIgAAAAJ

snehashis-roy commented 4 years ago

Thank you so much. I am going to try this out and report back.

iannellog commented 4 years ago

Only a little correction with respect to what I wrote yesterday. Instead of:

tagstruct.PageNumber = [d, depth];

write:

tagstruct.PageNumber = [d-1, depth];

since it is better to number pages starting form 0.

-- Giulio

Il giorno mar 4 feb 2020 alle ore 19:25 piby2 notifications@github.com ha scritto:

Thank you so much. I am going to try this out and report back.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/abria/TeraStitcher/issues/64?email_source=notifications&email_token=ACDW7VXWWVNHFVN3VHUICK3RBGXJLA5CNFSM4KPLLJNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKYVU2I#issuecomment-582048361, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACDW7VXC4J24INLL33UPFH3RBGXJLANCNFSM4KPLLJNA .

--

Giulio Iannello Preside della Facolta' Dipartimentale di Ingegneria Universita' Campus Bio-Medico di Roma v. Alvaro del Portillo, 21 00128 Roma, Italy

Tel: +39-06-22541-9602 E-mail: g.iannello@unicampus.it Fax: +39-06-22541-9609 URL: https://scholar.google.it/citations?user=L-UJxIgAAAAJ

snehashis-roy commented 4 years ago

Thank you. Now the import time has reduced to 1s from ~3000s.