Not converting tif to npy

[notevenclosee]

Hello, I was trying to run the covert raw data to npy files. I run the initialization and initialize workspace cell. They run smoothly. I still have to figure out what parameter to put for initialize alignment so skipped this cell. I then try to run the tile conversion code to convert raw data to npy files. In the code this cell has these lines io.convert_files(ws.file_list('raw', extension='tif'), extension='npy', processes=12, verbose=True); When I run it, it displays runcell('Convet raw data to npy files', '/home/mbu/programs/ClearMap2/ClearMap/Scripts/Wobbly stitcher.py') Converting 18435 files to npy! Converting 18435 files to npy: elapsed time: 0:01:20.587 There is no error. But It doesn't show progress of converting each file to npy like the tutorial suggested. I checked the directory where the raw data lives but didn't find the new files either. Could someone please provide some tips on how to solve this problem? I attached the console history below in case it's useful. Thank you!

runcell(1, '/home/mbu/programs/ClearMap2/ClearMap/Scripts/Wobbly stitcher.py') Elastix sucessfully initialized from path: /home/mbu/programs/ClearMap2/ClearMap/External/elastix/build

runcell('Initialize workspace', '/home/mbu/programs/ClearMap2/ClearMap/Scripts/Wobbly stitcher.py') Workspace[TubeMap]{/blue/m.farrer/mbu/DISCO pilot/210421_TH647_auto488_disco_tiled_14-52-31} raw: 14-52-31_TH647_auto488_disco_tiled_Blaze[<X,2> x <Y,2>]_C01_xyz-Table Z<Z,4>.ome.tif {18435 files, ('X', 'Y', 'Z'): (0, 0, 0) -> (4, 2, 1228)} autofluorescence: 14-52-31_TH647_auto488_disco_tiled_Blaze[<X,2> x <Y,2>]_C00_xyz-Table Z<Z,4>.ome.tif {18435 files, ('X', 'Y', 'Z'): (0, 0, 0) -> (4, 2, 1228)} stitched: no file layout: no file background: no file resampled: no file resampled_to_auto: no file auto_to_reference: no file arteries: no file binary: no file binary_status: no file skeleton: no file graph: no file density: no file

runcell('Convet raw data to npy files', '/home/mbu/programs/ClearMap2/ClearMap/Scripts/Wobbly stitcher.py') Converting 18435 files to npy! Converting 18435 files to npy: elapsed time: 0:01:20.587

[ChristophKirst]

Hi,

i think there might be some confusion. There were similar issues regarding this now. you have tiles and than a single tif file for each plane. I would recommend to convert each tile into a single npy file so the number of files is the number of tiles. If you only convert each plane into npy files there will be not much speedup of the code etc. The current code assumes the tiff files are 3d tiles.

You might need to write some code for this like this:

    e = io.te.Expression('14-52-31_TH647_auto488_disco_tiled_Blaze[<X,2> x <Y,2>]_C00_xyz-Table Z<Z,4>.ome.tif')
    s0 = io.source(e.string(values={'Z':0}))

    for f in s0.file_list:
      values = s0.expression.values(f);
      s = io.source(e.string(values=values));
      npy = e.string(values=values)[:-3] + 'npy'
      io.convert(s,npy)

[atduffaut]

Hi Christoph,

Thank you for helping us out with this problem. I am working on the same project as user notevenclose who opened the issue. When I try to run the code you suggested above to convert each tile into a npy file, the following error is raised:

NotImplementedError: create for FileList not implemented yet!

Would you have any further suggestions to avoid this?

Thanks

[andrewsilberfeld]

Hi @notevenclosee , did you ever solve this? I'm encountering the same problem using the tutorial data available at https://osf.io/5qzn7/. My data doesn't appear to be tiled in XY plane - it looks like the whole XY plane is visible... each tif seems to be 1 z-plane of the whole brain.

If I run: io.convert_files(ws.file_list('raw', extension='tif'), extension='npy', processes=12, verbose=True);

I get: converting 1796 files to npy! Converting file 9/1796 Tif-Source(1802, 2671, 1796)[uint16]{/n/.../andrew/idisco/salin...2-19_0_8X-cfos_UltraII_C00_xyz-Table Z0009.ome.tif} -> /n/.../andrew/idisco/saline/1273/150819_0_8X-cfos_15-42-19/15-42-19_0_8X-cfos_UltraII_C00_xyz-Table Z0009.ome.npy Converting file 11/1796 Tif-Source(1802, 2671, 1796)[uint16]{/n/.../andrew/idisco/salin...2-19_0_8X-cfos_UltraII_C00_xyz-Table Z0011.ome.tif} -> /n/.../andrew/idisco/saline/1273/150819_0_8X-cfos_15-42-19/15-42-19_0_8X-cfos_UltraII_C00_xyz-Table Z0011.ome.npy Converting file 7/1796 Tif-Source(1802, 2671, 1796)[uint16]{/n/.../andrew/idisco/salin...2-19_0_8X-cfos_UltraII_C00_xyz-Table Z0007.ome.tif} -> /n/.../andrew/idisco/saline/1273/150819_0_8X-cfos_15-42-19/15-42-19_0_8X-cfos_UltraII_C00_xyz-Table Z0007.ome.npy Converting file 0/1796 Tif-Source(1802, 2671, 1796)[uint16]{/n/.../andrew/idisco/salin...2-19_0_8X-cfos_UltraII_C00_xyz-Table Z0000.ome.tif} -> /n/.../andrew/idisco/saline/1273/150819_0_8X-cfos_15-42-19/15-42-19_0_8X-cfos_UltraII_C00_xyz-Table Z0000.ome.npy Converting file 6/1796 Tif-Source(1802, 2671, 1796)[uint16]{/n/.../andrew/idisco/salin...2-19_0_8X-cfos_UltraII_C00_xyz-Table Z0006.ome.tif} -> /n/.../andrew/idisco/saline/1273/150819_0_8X-cfos_15-42-19/15-42-19_0_8X-cfos_UltraII_C00_xyz-Table Z0006.ome.npy Converting file 10/1796 Tif-Source(1802, 2671, 1796)[uint16]{/n/.../andrew/idisco/salin...2-19_0_8X-cfos_UltraII_C00_xyz-Table Z0010.ome.tif} -> /n/.../andrew/idisco/saline/1273/150819_0_8X-cfos_15-42-19/15-42-19_0_8X-cfos_UltraII_C00_xyz-Table Z0010.ome.npy Converting file 5/1796 Tif-Source(1802, 2671, 1796)[uint16]{/n/.../andrew/idisco/salin...2-19_0_8X-cfos_UltraII_C00_xyz-Table Z0005.ome.tif} -> /n/.../andrew/idisco/saline/1273/150819_0_8X-cfos_15-42-19/15-42-19_0_8X-cfos_UltraII_C00_xyz-Table Z0005.ome.npy Converting file 3/1796 Tif-Source(1802, 2671, 1796)[uint16]{/n/.../andrew/idisco/salin...2-19_0_8X-cfos_UltraII_C00_xyz-Table Z0003.ome.tif} -> /n/.../andrew/idisco/saline/1273/150819_0_8X-cfos_15-42-19/15-42-19_0_8X-cfos_UltraII_C00_xyz-Table Z0003.ome.npy Converting file 8/1796 Tif-Source(1802, 2671, 1796)[uint16]{/n/.../andrew/idisco/salin...2-19_0_8X-cfos_UltraII_C00_xyz-Table Z0008.ome.tif} -> /n/.../andrew/idisco/saline/1273/150819_0_8X-cfos_15-42-19/15-42-19_0_8X-cfos_UltraII_C00_xyz-Table Z0008.ome.npy Converting file 1/1796 Tif-Source(1802, 2671, 1796)[uint16]{/n/.../andrew/idisco/salin...2-19_0_8X-cfos_UltraII_C00_xyz-Table Z0001.ome.tif} -> /n/.../andrew/idisco/saline/1273/150819_0_8X-cfos_15-42-19/15-42-19_0_8X-cfos_UltraII_C00_xyz-Table Z0001.ome.npy Converting file 2/1796 Tif-Source(1802, 2671, 1796)[uint16]{/n/.../andrew/idisco/salin...2-19_0_8X-cfos_UltraII_C00_xyz-Table Z0002.ome.tif} ->/n/.../andrew/idisco/saline/1273/150819_0_8X-cfos_15-42-19/15-42-19_0_8X-cfos_UltraII_C00_xyz-Table Z0002.ome.npy Converting file 4/1796 Tif-Source(1802, 2671, 1796)[uint16]{/n/.../andrew/idisco/salin...2-19_0_8X-cfos_UltraII_C00_xyz-Table Z0004.ome.tif} -> /n/.../andrew/idisco/saline/1273/150819_0_8X-cfos_15-42-19/15-42-19_0_8X-cfos_UltraII_C00_xyz-Table Z0004.ome.npy Converting 1796 files to npy: elapsed time: 0:00:50.698 ['/n/.../andrew/idisco/saline/1273/150819_0_8X-cfos_15-42-19/15-42-19_0_8X-cfos_UltraII_C00_xyz-Table Z0000.ome.npy', '/n/.../andrew/idisco/saline/1273/150819_0_8X-cfos_15-42-19/15-42-19_0_8X-cfos_UltraII_C00_xyz-Table Z0001.ome.npy', '/n/.../andrew/idisco/saline/1273/150819_0_8X-cfos_15-42-19/15-42-19_0_8X-cfos_UltraII_C00_xyz-Table Z0002.ome.npy', ... all the way to .... '/n/.../andrew/idisco/saline/1273/150819_0_8X-cfos_15-42-19/15-42-19_0_8X-cfos_UltraII_C00_xyz-Table Z1793.ome.npy', '/n/.../andrew/idisco/saline/1273/150819_0_8X-cfos_15-42-19/15-42-19_0_8X-cfos_UltraII_C00_xyz-Table Z1794.ome.npy', '/n/.../andrew/idisco/saline/1273/150819_0_8X-cfos_15-42-19/15-42-19_0_8X-cfos_UltraII_C00_xyz-Table Z1795.ome.npy']

I'm doing this on my university's supercomputing cluster so the visualization part isn't working. I'm using 16GB of ram for this (I know I need more RAM later). Do I need more now?

Has anyone else solved this?

[ChristophKirst]

Hi,

This seems to cause a lot of confusion.

The problem is the naming of the files and how those names are interpreted.

ClearMap does not know how the microscope organized the date into files.

If you have one tif file with 3d data for each tile the io.convert_files should work (assuming your file names are set correctly)

If you have a lot of tif files with one tif file per plane I recommend to specify the files for a single file in a source and then convert that to a single npy file and loop over all the files.

If you pass the full expression (tiles and z planes) the io.convert_files routine will convert each individual file!

Thus, for a single tile do this:

source = io.source(`filename_expresion_for tile.tif`
io.convert(source, `filename_for_tile.npy`)

In your above example

filename_expresion_for tile.tif would be something like

filename_<Z,4>.tif

By replacing filename with the actual filename for one single tile.

Then loop this over all tiles replacing the file names according to each tile.

Does this help?

[andrewsilberfeld]

Thank you! I think that did work:

imgdir = "/n/.../idisco/saline/1273/150819_0_8X-cfos_15-42-19/" source = io.source(imgdir+'15-42-19_0_8X-cfos_UltraII_C00_xyz-Table Z0001.ome.tif') io.convert(source,imgdir+"Z-merged.npy") expression_raw = 'saline/1273/150819_0_8X-cfos_15-42-19/Z-merged.npy' ws.update(raw=expression_raw, autofluorescence=expression_auto);

this seems to have made all the tif files of the mouse #1273 saline practice data into one large npy file. I can't validate yet through the viewer because I'm struggling with the X11/VcXSrv on my univ's cluster but print(ws.source('raw')) of the new NPY file shows an object of (1802,2671,1796) which I think is right!

[atduffaut]

Thank you for all the suggestions Christoph.

I was still having trouble with using io.convert, so for anyone struggling with that, a solution I found for converting all the ome.tif files for each plane into one tile with a merged Z-stack is using ImageJ and opening all the .tif files for one tile as an image sequence and saving the sequence as a single .tif file ("[00 x 00]_C01_merged.tif" for example). After doing this, the regular convert_files feature from ClearMap works as it should and the visualization feature confirms that the tile contains the correct image data.

[andrewsilberfeld]

I'm revisiting this because I'm now using a dataset that requires stitching and my original solution was insufficient when stitching was needed... Below is my solution which should be a complete fix. I'm leaving this here for the community.

The "fileList not implemented error" appeared for me too. I believe the reason is because is because the npy string from Christoph's code has a tag (the Z<Z,4> part) that doesn't get a value in the values dictionary. I added Z = 0 to the "values" dictionary and it solved the problem. However, I think there are 2 additional problems, which is that the io.convert function tries to convert all the files within the "s" source file_list (my Z-axis; 343 of them for me) and it made a file that is a few terabytes large and crashes my computer. Turns out it was exactly 343 times the size of the correct file... To get around this, I think the "source" file here should be only the z0000 file. To do this, set s = io.source(f). Lastly, I believe a tag for the fluorescence channel is also necessary, if you are like me and have the autofluo and raw files all in the same folder. It seems like the output is a file that has all the images of a given tile in correct Z-order, but does both channels simultaneously. The first dimension of the npy shape addresses this and viewing the file in p3d.plot with the first dimension set to 0 (autofluo) or 1 (raw) will show both files..

This code worked for me.

e = io.te.Expression(directory+expression_raw) s0 = io.source(e.string(values={'Z':0}))

for f in s0.file_list: values = s0.expression.values(f); s = io.source(f) #make source as the first file values['Z'] = 0; #add Z=0 to values dict. npy = e.string(values=values)[:-3] + 'npy' io.convert(s,npy, verbose = True)

p3d.plot(ws.source('raw',extension = 'npy', values = {'Y':1,'X':0,'Z':})[0,:,:,:]) or p3d.plot(ws.source('raw',extension = 'npy', values = {'Y':1,'X':0,'Z':})[1,:,:,:])

for reference, this was my directory+expression_raw string:

directory+expression_raw='/n/.../idisco/20210917_testbrains/211106_847_left_mosaic_21-41-10_1.6x_z15/847_left_mosaic[<Y,2> x <X,2>]_C01_z<Z,4>.ome.tif'