Main script

[bwtan]

Hi Christoph,

Could you share a copy of the main script running through the whole pipeline?

[ChristophKirst]

The script is called Tubemap.py in the Scripts fodder.

[bwtan]

The script is called Tubemap.py in the Scripts fodder.

Thanks! Can I assume this Clearmap2 still runs in linux system (ubuntu)? What parts that may need to be tweaked if using windows?

[saskra]

I get this error message when I try to run the script:

  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/IO/TIF.py", line 72, in array
    array = self._tif.asarray(validate = False, maxworkers=processes);
TypeError: asarray() got an unexpected keyword argument 'validate'

[bwtan]

I get this error message when I try to run the script:

  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/IO/TIF.py", line 72, in array
    array = self._tif.asarray(validate = False, maxworkers=processes);
TypeError: asarray() got an unexpected keyword argument 'validate'

I also got this error message. Did you manage to fix it?

[ChristophKirst]

Yes we fixed it, please wait for the new version and Documentation to get online very soon.

Am Mi., 10. Juni 2020 um 14:03 Uhr schrieb bwtan95 <notifications@github.com

:

Reopened #2 https://github.com/ChristophKirst/ClearMap2/issues/2.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ChristophKirst/ClearMap2/issues/2#event-3431457844, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMCTWVWXOV2J7BHT3RWENDRV7YJZANCNFSM4NESWNWA .

[ChristophKirst]

The new version should fix this issue. If not please reopen this issue.

[saskra]

Now I get this error:

Traceback (most recent call last):
  File "/home/saskra/anaconda3/envs/ClearMap/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3331, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-788a7e8ab86d>", line 1, in <module>
    runfile('/home/saskra/PycharmProjects/ClearMap2/ClearMap/Scripts/TubeMap.py', wdir='/home/saskra/PycharmProjects/ClearMap2/ClearMap/Scripts')
  File "/snap/pycharm-professional/201/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/snap/pycharm-professional/201/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/Scripts/TubeMap.py", line 84, in <module>
    layout = stw.WobblyLayout(expression=ws.filename('raw'), tile_axes=['X','Y'], overlaps=(45, 155));
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/Alignment/Stitching/StitchingWobbly.py", line 725, in __init__
    strg.TiledLayout.__init__(self, sources = sources, expression = expression, tile_axes = tile_axes, tile_shape = tile_shape, tile_positions = tile_positions, positions = positions, overlaps = overlaps, alignments = alignments, position = position, shape = shape, dtype = dtype, order = order);
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/Alignment/Stitching/StitchingRigid.py", line 1909, in __init__
    sources, alignments, tile_positions = _initialize_tiles_from_expression(expression, tile_axes=tile_axes, tile_shape=tile_shape, tile_positions=tile_positions, overlaps=overlaps, positions=positions, alignments=alignments);
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/Alignment/Stitching/StitchingRigid.py", line 2270, in _initialize_tiles_from_expression
    tile_shape = np.max(tile_positions, axis = 0) + 1; # assume that numbering starts with 0!
  File "<__array_function__ internals>", line 6, in amax
  File "/home/saskra/anaconda3/envs/ClearMap/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 2668, in amax
    keepdims=keepdims, initial=initial, where=where)
  File "/home/saskra/anaconda3/envs/ClearMap/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 90, in _wrapreduction
    return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
ValueError: zero-size array to reduction operation maximum which has no identity

Does it rely on files which were not published together with the code?

[ChristophKirst]

TubeMap is a data analysis script for TB data sets. We do not include those in our software. We have sample data here: https://osf.io/sa3x8/

[saskra]

Many thanks for the link! Are there perhaps besides the complete brains also small toy examples for quick testing of the pipeline?

[cudmore]

I agree with @saskra, can we get a small toy example with maybe 4-stacks?

Each .tif on osf.io is 3.69 GB with (3501, 404, 1302) pixels. I can't run the entire pipeline on my non super-computer machine. If I could it would take way too long to play with different parameters in the code.

In particular, I am trying to play with Scripts/TubeMap.py. I am most interested in per-stack algorithms to generate a mask of vasculature. Not as interested in the stitching.

Can we get a toy example that can run on a consumer machine? If I understand the osf.io website, the dataset has been downloaded just 2 times. By providing a smaller dataset, maybe 10+ or 20+ people/labs would give your code a try?

[ChristophKirst]

Hi,

thanks for the feed back.

We can provide a smaller data set at some point together with some tests etc.

In the mean time, you can always just download a single tile (from the center of the brain ideally) from our data set and work with that. If that is still too big, you can also easily cut out a smaller subset.

There is a debug/test mode for ClearMap does that for you (see the tutorial)

If you prefer to work with a small data set directly, you can simply use this code

import ClearMap.IO.IO as io
source = io.as_source('filename_of_tile.tif');
subset = source[500:1000, 500:1000, 500:1000];
io.write('filename_of_subset.npy', subset);

to generate a small block (here 500**3) of data and work with that. It should be small enough for a standard computer to work with.

To run the tutorial then simply modify the file names accordingly and skip the stitching part in the tutorial.

Hope this helps.

Out of curiosity: This code is designed to work for large 3d images but I would be curious what smaller data sets you are planning to use ClearMap with ? It is possible to add another expert pipeline for other data and we are currently working on analyzing super resolution / expansion microscopy data sets.

Christoph

[saskra]

Out of curiosity: This code is designed to work for large 3d images but I would be curious what smaller data sets you are planning to use ClearMap with ?

We have many "small" 3D ROIs and are therefore interested in automatizable and fast algorithms for segmentation and skeletonization.

[saskra]

In the mean time, you can always just download a single tile (from the center of the brain ideally) from our data set and work with that.

Thanks for the tip! So I took the autofluorescence from the first brain 39L completely and from the raw data only four consecutive files from the middle (Y02, X05-06, Filter0000-0001).

  #directories and files
  directory = '/home/saskra/PycharmProjects/ClearMap2/ClearMap/Tests/Data/TubeMap_Example'
  expression_raw      = 'Raw/14-16-41_tricocktail_UltraII[<Y,2> x <X,2>]_C00_UltraII Filter0001.ome.tif'
  expression_arteries = 'Raw/14-16-41_tricocktail_UltraII[<Y,2> x <X,2>]_C00_UltraII Filter0000.ome.tif'
  expression_auto     = 'Autofluorescence/14-02-13_auto_UltraII_C00_xyz-Table Z<Z,4>.ome.tif'

Unfortunately the script only runs until this error message:

Traceback (most recent call last):
  File "/home/saskra/anaconda3/envs/ClearMap/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3331, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-788a7e8ab86d>", line 1, in <module>
    runfile('/home/saskra/PycharmProjects/ClearMap2/ClearMap/Scripts/TubeMap.py', wdir='/home/saskra/PycharmProjects/ClearMap2/ClearMap/Scripts')
  File "/snap/pycharm-professional/201/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/snap/pycharm-professional/201/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/Scripts/TubeMap.py", line 184, in <module>
    "result_directory" :  ws.filename('elastix_resampled_to_auto')
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/IO/Workspace.py", line 286, in filename
    debug=debug, **kwargs);
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/IO/Workspace.py", line 152, in filename
    raise ValueError('Cannot find name for type %r!' % ftype);
ValueError: Cannot find name for type 'elastix_resampled_to_auto'!

[saskra]

This was solved with one of the last commits.

[saskra]

Next error, but I might open a new issue for that one:

Graph reduction: initialized.
Graph reduction: Found 20393 branching and 592581 non-branching nodes: elapsed time: 0:00:00.591
Graph reduction: Scanned 20393/20393 branching nodes, found 1018 branches: elapsed time: 0:00:01.420
Graph reduction: Scanned 250000/592581 non-branching nodes found 11667 branches: elapsed time: 0:00:09.205
Graph reduction: Scanned 500000/592581 non-branching nodes found 22705 branches: elapsed time: 0:00:16.715
Graph reduction: Scanned 592581/592581 non-branching nodes found 26871 branches: elapsed time: 0:00:19.219
Graph reduction: Graph reduced from 612974 to 20393 nodes and 619452 to 26871 edges: elapsed time: 0:00:20.528
Transforming vertex property: coordinates -> coordinates_atlas
Traceback (most recent call last):
  File "/snap/pycharm-professional/209/plugins/python/helpers/pydev/pydevd.py", line 1438, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/snap/pycharm-professional/209/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/Scripts/TubeMap.py", line 482, in <module>
    verbose=True);
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/Analysis/Graphs/GraphGt.py", line 1366, in transform_properties
    values = transformation(values);
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/Scripts/TubeMap.py", line 465, in transformation
    sink_shape=io.shape(ws.filename('resampled')));
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/IO/IO.py", line 258, in shape
    source = as_source(source);
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/IO/IO.py", line 206, in as_source
    source =  mod.Source(source, *args, **kwargs);
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/IO/TIF.py", line 36, in __init__
    self._tif = tif.TiffFile(location, multifile = multi_file);
  File "/home/saskra/anaconda3/envs/ClearMap/lib/python3.7/site-packages/tifffile/tifffile.py", line 2183, in __init__
    fh = FileHandle(arg, mode='rb', name=name, offset=offset, size=size)
  File "/home/saskra/anaconda3/envs/ClearMap/lib/python3.7/site-packages/tifffile/tifffile.py", line 6408, in __init__
    self.open()
  File "/home/saskra/anaconda3/envs/ClearMap/lib/python3.7/site-packages/tifffile/tifffile.py", line 6421, in open
    self._fh = open(self._file, self._mode)
FileNotFoundError: [Errno 2] Datei oder Verzeichnis nicht gefunden: '/home/saskra/PycharmProjects/ClearMap2/ClearMap/Tests/Data/TubeMap_Example/debug_resampled.tif'