search

abria / TeraStitcher

A tool for fast automatic 3D-stitching of teravoxel-sized microscopy images
http://abria.github.io/TeraStitcher/
Other
78 stars 32 forks source link

Sparse data tiling #6

Open miketaormina opened 7 years ago

miketaormina commented 7 years ago

First, I'd like to compliment you on your useful software - I've had some great initial results with TeraSticher on large volumes of microscopy data. As you most likely know, this type of data is very memory intensive, rapidly filling hard drives. In order to be more efficient, I often tile samples in an irregular way, e.g. different depths at different locations etc. When trying to align a data set today I get the error message in StackedVolume::init(...): Number of second-level directories is not the same for all first-level directories! This stems from exactly the scenario stated -- I have two X positions, one with two and one with three Y positions.

Is it possible to align such a data set natively, or would I need to figure out a way to pad the raw data to make the volume rectangular prior to loading into TeraStitcher?

abria commented 7 years ago

Hi Mike,

TeraStitcher supports sparse datasets, but you must tell this in the Import step using the --sparse_data option.

Please also consider the following limitations:

  1. data can be sparse along arbitrary axes (empty or incomplete tiles), but the two-level hierarchy of folders must be complete and folder names must be their nominal positions. In your case, this means you should add the missing folders, name them correctly, and leave them empty.
  2. tiles must be regularly spaced. Sparse datasets in which tiles are acquired at arbitrary positions w/o following a 2D regular grid pattern (for instance, like these) are not supported.

Point (1) only holds if you ask TeraStitcher to scan your two-level hierarchy of folders and generate the .xml descriptor automatically (default option). Another option is to directly import your own generated/modified .xml descriptor (see this page). In this case you need to insert into the .xml the missing tiles and for each of them an empty Z_RANGES attribute.

If you have further questions, we will be happy to help.

miketaormina commented 7 years ago

Thanks for the quick reply, Abria,

I still have a few issues, but I think that they are perhaps best solved during data acquisition in the future, keeping these points in mind. For the sake of feedback, however, I thought I would let you know what happens when I try to fulfill the requirements mentioned above.

First I added the empty directories so that both x directories contain identical y directories. Using the GUI with the Sparse data box checked raises an error Unable to find imagefiles... Next I tried using the command line and got a different error, saying Stacks have unequal X, Y dimensions. This makes it seem that the Sparse data checkbox on the GUI is failing to append the option --sparse_data.

As for the second error, I suppose that my having changed the ROI from tile to tile is the cause (XXXXXX_YYYYYY images are 2048x2048 while XXXXXX_yyyyyy images are 1420x2048). Is it true that all tiles need to have the same dimensions in X and Y? If so, is it also true of Z? I assume that the volume should be sampled such that Z coordinates coincide, but can one XY tile be thicker in Z than others? I think this all amounts to asking in what ways tiles can be incomplete in a sparse data set.

Thanks again for the help, I am convinced that it will be worth finding the data acquisition pipeline that allows use of TeraStitcher, as it is well executed program for these large image datasets.

abria commented 7 years ago

Hi Mike,

I have just re-tested our sparse data sets with the latest GUI version (TeraStitcher 1.9.62), and it works. Please check that your file names are named correctly (see here). Also try to import your volume from scratch, i.e. you should delete all the files generated by TeraStitcher (xml_import.xml and mdata.bin). If you still have this issue, please attach a screenshot of your folder/file hierarchy and I will check it.

As to your questions:

BTW, a data set can be sparse along X, Y, and Z at the same time.

I am uploading some sparse data sets to my DropBox so that you can check yourself how this works. I will let you know once they are ready for download.

Alessandro

abria commented 7 years ago

Hi Mike,

you can download a 2(X) x 2(Y) data set with one empty tile (sparse in XY) at this link and another 2(X) x 2(Y) data set with uneven Z dimensions (sparse in Z) at this link.

Alessandro

JeffreyStirman commented 4 years ago

Hello. I am also having an issue with the import step when using sparse tiling. In my case I have all XY tiles and folders, but within the folders there might be a different number of z-planes, but they are all named appropriately. When running the import with the --sparse_data flag,

C:\terastitcher\terastitcher -1 --volin="D:\2019_09_17_tiffs\20190917_18_46_25_RaccoonSlab-LectinNeuN-4x\Ex_488_Em_0" --ref1=H --ref2=V --ref3=D --vxl1=1.86 --vxl2=1.86 --vxl3=2 --projout=xml_import.xml --sparse_data

I get the error : ERROR: in StackedVolume::normalize_stacks_attributes(...): Stacks have unequal X,Y dimensions. This feature is not supported yet.

I have checked the folders and they are fine. Is there anything else I can do or check?

Thanks!

JeffreyStirman commented 4 years ago

I use the command line version as I use Parastitcher after.

Also, the sparse_data works if the folder is COMPLETELY empty, but if there any images in the folder and the number is not equal across all folders it gives the error.

iannellog commented 4 years ago

The error you get means that there is some non-empty folder with images that have different dimensions (i.e. differ in the number of rows and/or in the number of columns). I checked the code and I do not see another reason for this error. If you are sure that images have all the same size, I will send you an executable that bypass the check and prints more information. I assume also that you are using the two-level hierarchy of folders and use TeraStitcher to generate the first xml import file, is that right?

-- Giulio

Il giorno ven 20 set 2019 alle ore 21:22 JeffreyStirman < notifications@github.com> ha scritto:

Hello. I am also having an issue with the import step when using sparse tiling. In my case I have all XY tiles and folders, but within the folders there might be a different number of z-planes, but they are all named appropriately. When running the import with the --sparse_data flag,

C:\terastitcher\terastitcher -1 --volin="D:\2019_09_17_tiffs\20190917_18_46_25_RaccoonSlab-LectinNeuN-4x\Ex_488_Em_0" --ref1=H --ref2=V --ref3=D --vxl1=1.86 --vxl2=1.86 --vxl3=2 --projout=xml_import.xml --sparse_data

I get the error : ERROR: in StackedVolume::normalize_stacks_attributes(...): Stacks have unequal X,Y dimensions. This feature is not supported yet.

I have checked the folders and they are fine. Is there anything else I can do or check?

Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/abria/TeraStitcher/issues/6?email_source=notifications&email_token=ACDW7VUPB2ZN5CB2HQPNI2TQKUPFHA5CNFSM4CPZ7QJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7HU2IA#issuecomment-533679392, or mute the thread https://github.com/notifications/unsubscribe-auth/ACDW7VVCAHTXLJXZVASXPNDQKUPFHANCNFSM4CPZ7QJQ .

--

Giulio Iannello Preside della Facolta' Dipartimentale di Ingegneria Universita' Campus Bio-Medico di Roma v. Alvaro del Portillo, 21 00128 Roma, Italy

Tel: +39-06-22541-9602 E-mail: g.iannello@unicampus.it Fax: +39-06-22541-9609 URL: https://scholar.google.it/citations?user=L-UJxIgAAAAJ

JeffreyStirman commented 4 years ago

You were right. Though I had the correct naming of XY tile folders, in some of the folders I had one image that was a different size. This was a bug in my data saving. Once that was corrected everything went fine. Thanks so much and so sorry to waste your time on that!