[Uncompression ] Error while constructing cell map: Invalid cell size for lines.

tashrifbillah commented 3 years ago

I compressed a vtk to tko and then tried uncompression--tko to vtp but ran in to the following error:

2020-10-27 18:25:22.244 ( 41.474s) [ B2E47740] vtkPolyData.cxx:961 ERR| vtkPolyData (0x5628f187a470): Error while constructing cell map: Invalid cell size for lines.

I did not provide a --config file for the former.

Does the error look familiar?

tashrifbillah commented 3 years ago

Hi @zhangfanmark , do you need the data for this?

zhangfanmark commented 3 years ago

Hi @tashrifbillah yes, please share the data. Thanks!

Hi @haehn could you look into the issue. I will also take a look to see if there is something unusual with the vtk file. Thanks!

tashrifbillah commented 3 years ago

Way to reproduce:

wget privately_shared_link
gunzip 2T-01009-state.vtk.gz

# works fine
trakofy -i 2T-01009-state.vtk -o 2T-01009-state.tko

# runs into error
untrakofy -i 2T-01009-state.tko -o 2T-01009-state.vtp

pieper commented 3 years ago

I wasn't able to reproduce this issue with the data provided. I tested mac, by running pip_install("trako") into a local release build of Slicer (so built against Slicer's python and vtk).

bash-3.2$ trakofy -i 2T-01009-state.vtk -o 2T-01009-state.tko

>> T R A K O.

Loading scalar tensors

From 1100853532 bytes to 442791096 bytes.
Ratio 2.486169080509243

 (x_x) tko-ed.

vtkDebugLeaks has found no leaks.
bash-3.2$ untrakofy -i 2T-01009-state.tko -o 2T-01009-state.vtp

>> T R A K O.

Restored scalar tensors

Restored 777594696 bytes.

 (x_x) tko-ed.

vtkDebugLeaks has found no leaks.
bash-3.2$

@tashrifbillah what python/vtk/os environment were you using when you got the error?

pieper commented 3 years ago

This is the before and after data file (they look the same to me).

tashrifbillah commented 3 years ago

what python/vtk/os environment were you using when you got the error?

Hi @pieper , can you give my Singularity container a try on your MAC?

singularity pull library://tbillah/collection/trako.sif

singularity run --bind /path/to/host/data:/data trako.sif trakofy -i /data/2T-01009-state.vtk -o /data/2T-01009-state.tko

But to answer your question, I installed Python 3.8.3 and trako==0.3.4.dev9:

    # install py3
    wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
    /bin/bash Miniconda3-latest-Linux-x86_64.sh -b -p miniconda3/
    source miniconda3/bin/activate

    # install TRAKO
    CC=clang CXX=clang++ pip install trako==0.3.4.dev9

tashrifbillah commented 3 years ago

Hi @pieper , I just realized you posted the log about trakofy but I am having issue with untrakofy. Did you try the latter and didn't get any error either? To reiterate, I didn't get any error with trakofy.

pieper commented 3 years ago

I did the untrakofy too (look again at my post above).

Interesting that there could be a bug like this that depends on the version of python. I don't believe it's possible to install singularity on mac, but I might be able to test on linux.

tashrifbillah commented 3 years ago

Hi Steve, sorry about missing that. Actually, you can try to install just Python 3.8.3 and trako==0.3.4.dev9 on your MAC and try to reproduce the error. However, what are the versions of the above two you have in your MAC?

pieper commented 3 years ago

Hi Tashrif - I was able to install singularity on linux and exactly reproduce the issue you reported with untrakofy. It says it is failing inside the vtk code.

It seems like this might be a singularity issue, or maybe a python or vtk issue. Are you able to try different version in your singularity image?

tashrifbillah commented 3 years ago

Hi Steve, I am happy to try a different version of Python/trako. However, the trako I installed came straight from PyPi i.e. Daniel. Nevertheless, what version of Python/trako would you recommend?

pieper commented 3 years ago

Maybe we can try the same versions that work in my build? Python 3.6.7 and VTK 8.2.0 are the current once we use for Slicer. Which version of VTK is in your singularity image?

tashrifbillah commented 3 years ago

Hi Steve, my VTK is 8.1.2 that came with the trako PyPi package.

pieper commented 3 years ago

Thanks @tashrifbillah, it seems the vtk version is part of the problem. I installed trako on my ubuntu 18.04 system using pip3 install trako and it failed with the same core dump as you got in singularity.

When I installed trako it picked up vtk 9.0.1, which is the latest release so trako's requirement of >=8.1.2 resolved to that so I'm guessing that is the underlying problem.

It looks like the VTK team is only providing vtk9 for python3 so we'll need to see how to port. I'm going to see if I can get a developer build working with vtk9 to see if I can reproduce the issue.

In the mean time, it could make sense to try using Slicer's python environment in your singularity container. Can you give that a try?

pieper commented 3 years ago

It seems that the vtk file in question has several polylines with only a single point. In vtk8 this was a warning condition but in vtk9 it is now considered an error condition and leads to the observed error message and exit when using the vtp writer.

Here is an example of the warning when loading this data in slicer with vtk8:

Warning: In /opt/sr/VTK/Common/DataModel/vtkPolyData.cxx, line 993
vtkPolyData (0x7fd6646341f0): Building VTK_LINE 1140468 with only one point, but VTK_LINE needs at least two points. Check the input.

So I guess this data came from UKF and maybe the model decided there as no good first step for the seed so it puts out a degenerate polyline with just the seed point. @ljod let's discuss whether if this is the case and if so whether we need to keep these or not.

The same warnings occur with loading in vtk8 slicer build with either the original vtk file or with the trakofied/untrakofied vtp file, so I it seems trako is not introducing the issue.

untrakofy crashes at this location (note frame 5):

1   std::vector<vtkPolyData_detail::TaggedCellId>::operator[] stl_vector.h                         1043 0x7f3e1ff04516 
2   vtkPolyData_detail::CellMap::GetTag                       vtkPolyDataInternals.h               239  0x7f3e1ff04325 
3   vtkPolyData::GetCellType                                  vtkPolyData.h                        730  0x7f3e2012759a 
4   vtkDataSet::GetCellTypes                                  vtkDataSet.cxx                       326  0x7f3e1ff39865 
5   vtkXMLUnstructuredDataWriter::ProcessRequest              vtkXMLUnstructuredDataWriter.cxx     166  0x7f3e1b31418e 
6   vtkExecutive::CallAlgorithm                               vtkExecutive.cxx                     746  0x7f3e1fe25449 
7   vtkDemandDrivenPipeline::ExecuteData                      vtkDemandDrivenPipeline.cxx          462  0x7f3e1fe1cf92 
8   vtkCompositeDataPipeline::ExecuteData                     vtkCompositeDataPipeline.cxx         161  0x7f3e1fe113a7 
9   vtkDemandDrivenPipeline::ProcessRequest                   vtkDemandDrivenPipeline.cxx          261  0x7f3e1fe1c642 
10  vtkStreamingDemandDrivenPipeline::ProcessRequest          vtkStreamingDemandDrivenPipeline.cxx 343  0x7f3e1fe8977e 
11  vtkDemandDrivenPipeline::UpdateData                       vtkDemandDrivenPipeline.cxx          419  0x7f3e1fe1cdb1 
12  vtkStreamingDemandDrivenPipeline::Update                  vtkStreamingDemandDrivenPipeline.cxx 417  0x7f3e1fe89b32 
13  vtkStreamingDemandDrivenPipeline::Update                  vtkStreamingDemandDrivenPipeline.cxx 380  0x7f3e1fe899b7 
14  vtkAlgorithm::Update                                      vtkAlgorithm.cxx                     1421 0x7f3e1fe082b8 
15  vtkAlgorithm::Update                                      vtkAlgorithm.cxx                     1415 0x7f3e1fe0827f 
16  PyvtkAlgorithm_Update_s2                                  vtkAlgorithmPython.cxx               2338 0x7f3e3b09da0f 
17  PyvtkAlgorithm_Update                                     vtkAlgorithmPython.cxx               2425 0x7f3e3b09dd30 
18  _PyCFunction_FastCallDict                                 methodobject.c                       234  0x7f3e4c514713 
19  _PyCFunction_FastCallKeywords                             methodobject.c                       294  0x7f3e4c514a05 
20  call_function                                             ceval.c                              4837 0x7f3e4c58e69c 
... <More>

A good solution for this could be to run vtkCleanPolyData at one (or more? or all?) of the following spots: before saving from UKF, after reading in trakofy, or before writing in trakofy.

ljod commented 3 years ago

I remember a long time ago asking for this single point output to be fixed in UKF and I believe it was, so I wonder about the origin and age of this input file.

However I think the problem is not limited to single points and is more severe based on the images. There are many straight lines near the cortex that are connecting one fiber end to another fiber beginning point. This is an off by one error in the vtk file. This looks like a bad conversion between tractography file formats or a major file write error in the program that created this tractography.

Yes I agree trako should guard against this and output a warning. But this file is really bad and should be fixed and not used for any analysis.

pieper commented 3 years ago

@haehn - can you have a look and if you like it update the pypi package?

@tashrifbillah - you may want to add the lines from the PR to your local version for now.

tashrifbillah commented 3 years ago

Hi @pieper , unfortunately adding lines in my local version did not resolve the error. If I pip install the PR, do you think that would make a difference?

pieper commented 3 years ago

@tashrifbillah the change is implemented only for trakofy so be sure you create a new tko file from the vtk file with the bad lines and then untrakofy should work on that data. Can you confirm?

tashrifbillah commented 3 years ago

I did, but to no avail.

jcfr commented 3 years ago

Would it be possible to (1) share here a snippet (python or c++) allowing to reproduce the problem or (2) create an issue in the VTK tracker ?

tashrifbillah commented 3 years ago

singularity pull library://tbillah/collection/trako.sif
singularity run --bind /path/to/host/data:/data trako.sif trakofy -i /data/2T-01009-state.vtk -o /data/2T-01009-state.tko
singularity run --bind /path/to/host/data:/data trako.sif untrakofy -i /data/2T-01009-state.tko -o /data/2T-01009-state.vtp

Hi, is this what you asked for? But this is before Steve's PR.

jcfr commented 3 years ago

is this what you asked for? Not exactly

I was looking to have a snippet using only VTK 9.

Indeed, based on the previous comment:

In vtk8 this was a warning condition but in vtk9 it is now considered an error condition and leads to the observed error message and exit when using the vtp writer

I anticipate that having such snippet would allow us to answer these questions:

Should VTK be updated to report better message ?
Should we revert to warning ?
Should we add flag to support the old behavior ?

cc: @pieper

pieper commented 3 years ago

Looking a bit more, I do agree @jcfr that this is a regression in vtk that should be fixed.

The code in vtk9 aborts from building cells when it hits a degenerate primitive, meaning that the vtkCellMap is null, leading to a crash in vtkXMLUnstructuredDataWriter.

The corresponding code in vtk8.2 prints the warning but keeps the cell as-is and keeps looping, so the vtkCellTypes is valid.

It should be trivial to make an example for this (try to save a vtkPolyData with one line with one point to vtp), and even to fix the code in vtk9 (generate a warning instead of an error), but for trako the first priority will be to clean the old data so I'll try sort out why my PR code isn't working for @tashrifbillah.

pieper commented 3 years ago

I did, but to no avail.

Hi @tashrifbillah can you give the PR another try? I just ran your test again on the data you provided with vtk9 and as long as I generate a new tko file using trakofy that has the patch I do not get a crash on vtk9 (and confirmed it does crash if I don't include the PR code).

Since I confirmed that it works for me I went ahead and merged this in trako, so the next pypi package version will have the fix.

tashrifbillah commented 3 years ago

Hi @pieper , my apologies for the last report. You were right--no more error. In retrospect, I might not have affected the change inside my container properly. Thank you.

tashrifbillah commented 3 years ago

I updated my container with the patch. In future, I can try to build against the next PyPi package. In the meantime, you can access it as:

singularity pull trako.sif library://tbillah/collection/trako.sif:0.1

pieper commented 3 years ago

For reference, I filed an issue on the vtk tracker: https://gitlab.kitware.com/vtk/vtk/-/issues/18059

bostongfx / TRAKO

[Uncompression ] Error while constructing cell map: Invalid cell size for lines. #8