search

knossos-project / knossos_utils

Python library for interacting with KNOSSOS data sets and annotation files
GNU General Public License v2.0
9 stars 9 forks source link

Export Image Sequence of Knossos Dataset #10

Closed unidesigner closed 7 years ago

unidesigner commented 8 years ago

Hi, I try to use the knossos_python_tools to extract the image sequence of part of the whole connectomics dataset here: http://connectomes.org/index.php?p=whole-brain-connectomics-datasets I used this code:

from knossos_python_tools.knossos_utils.knossosdataset import *
k=KnossosDataset()
k.initialize_from_knossos_path("/path/to/downloaded/knossos/file/")
k.export_to_image_stack(out_path="/tmp/out/", out_format="png", mag=1)

But I am only getting black images. What is the correct way to do this?

mdraw commented 8 years ago

I currently can't reproduce this issue, but the completely black images probably come from an error that occurs somewhere in this try-block: https://github.com/knossos-project/knossos_python_tools/blob/dd2a42dbf979aca451391095c0afcde10c2cd05d/knossos_utils/knossosdataset.py#L651

Could you remove the following except block (starting in line 666) and look if an error with a meaningful traceback occurs?

unidesigner commented 8 years ago

Ah, I get IOError: [Errno 2] No such file or directory: './mag1/x0000/y0000/z0000/BRAX-striatum-2k_mag1_x0000_y0000_z0000.raw' Shouldn't the missing folders be created automatically? Is it correct that already the first images should show some data?

mdraw commented 8 years ago

What exactly was "/path/to/downloaded/knossos/file/" in your code snippet? Maybe @sdorkenw knows what went wrong here.

jmrk84 commented 8 years ago

In case this happened on windows: There is a hardcoded '/' as path separator (at least in my ancient version), that caused a similar problem for me. The missing folders contain the dataset, that is what you need to do anything. The function takes a knossos dataset and can export tif stacks from it. Could you try to provide an absolute path maybe instead of '.'?

Optiligence commented 8 years ago

/ should be no problem tho

jmrk84 commented 8 years ago

When used like this, it is: mag_folder = glob.glob(path+"mag")[0].split('/')

sdorkenw commented 8 years ago

This is not working, because the dataset is only for remote use only (apparently). As @unidesigner said, there are no downloaded cubes, hence no data to export / as @mdraw pointed out it defaults to black images. @jmrk84 "/" problems only occured afaik for loading knossos.conf, which seems fine here.

Knossos does not write the data to disk for remote datasets nor does knossos_python_tools support remote datasets to this point (we should capture this case though to give a meaningful error message).

The solution is to download the datatset (>20GB) first. I do not know how far the ftp link given in the knossos.conf can be used for that (@jmrk84 ?), otherwise one has to ask Shawn Mikula to provide a downloadable version of the dataset.

unidesigner commented 8 years ago

Thanks for the help. I was under the wrong assumption that the cubes are downloaded automatically if not present. I'll see if I can download the cubes first and then call the export. I report back here if I run into more issues.

sdorkenw commented 7 years ago

@unidesigner: Some time has passed, but I just implemented support for remote datasets. I tested it with the mouse dataset and it works fine, just in case you still have use for that.

unidesigner commented 7 years ago

Hi @sdorkenw Thanks for updating. Can you post a short code snippet in how you would download the dataset at http://connectomes.org/knossos.conf ? I get an "Magnification not supported" exception. Thanks!

sdorkenw commented 7 years ago

Hi @unidesigner . Thanks for reporting! Unfortunately, I was not able to reproduce your error. It is very likely that the initialization failed.

I took the opportunity and write up a basic usage script that should cover your case: https://github.com/knossos-project/knossos_utils/blob/master/examples/basic_usage.py

Please let me know if that solves your problem.

unidesigner commented 7 years ago

Hi @sdorkenw - thanks for the nice example, this works now with one caveat. I see black squares in the images, at random locations. Seems to be a problem with fetching a few blocks. When I retry, the locations of the the black squares changes. Maybe one could do some check if a block was retrieved correctly.

The show_progress = True parameter seems not to work, it does not show any continuous progress while fetching the data. (I'm using Python 2.7.12)

sdorkenw commented 7 years ago

Hi @unidesigner. Thank you for the feedback! There might be also a bug in the export, but let's first check if its the connection:

The latest version of from_raw_cubes_to_matrix does retries (default=10) and reports errors. In case of copy_dataset that is activated with return_errors=True and for from_raw_cubes_to_matrix that's http_verbose=True. Could you please try that and see if that reports anything?

And you are right about show_progress. Since I implemented multi-threading for from_raw_cubes_to_matrix this is currently only showing the overall speed...

unidesigner commented 7 years ago

The http_verbose=True option worked, I don't see the black squares anymore. It is not superfast as I assume that it has to download complete cubes for slices in the data, but in any case it works now. Thanks!

sdorkenw commented 7 years ago

Great! You are right, it always has to download 128^3 cubes. That's were a complete download first and then operating on the offline dataset can be useful. That particular dataset is about ~12GB on disk.