Closed tashrifbillah closed 3 years ago
Hi @zhangfanmark , do you need the data for this?
Hi @tashrifbillah yes, please share the data. Thanks!
Hi @haehn could you look into the issue. I will also take a look to see if there is something unusual with the vtk file. Thanks!
Way to reproduce:
wget privately_shared_link
gunzip 2T-01009-state.vtk.gz
# works fine
trakofy -i 2T-01009-state.vtk -o 2T-01009-state.tko
# runs into error
untrakofy -i 2T-01009-state.tko -o 2T-01009-state.vtp
I wasn't able to reproduce this issue with the data provided. I tested mac, by running pip_install("trako")
into a local release build of Slicer (so built against Slicer's python and vtk).
bash-3.2$ trakofy -i 2T-01009-state.vtk -o 2T-01009-state.tko
>> T R A K O.
Loading scalar tensors
From 1100853532 bytes to 442791096 bytes.
Ratio 2.486169080509243
(x_x) tko-ed.
vtkDebugLeaks has found no leaks.
bash-3.2$ untrakofy -i 2T-01009-state.tko -o 2T-01009-state.vtp
>> T R A K O.
Restored scalar tensors
Restored 777594696 bytes.
(x_x) tko-ed.
vtkDebugLeaks has found no leaks.
bash-3.2$
@tashrifbillah what python/vtk/os environment were you using when you got the error?
This is the before and after data file (they look the same to me).
what python/vtk/os environment were you using when you got the error?
Hi @pieper , can you give my Singularity container a try on your MAC?
singularity pull library://tbillah/collection/trako.sif
singularity run --bind /path/to/host/data:/data trako.sif trakofy -i /data/2T-01009-state.vtk -o /data/2T-01009-state.tko
But to answer your question, I installed Python 3.8.3
and trako==0.3.4.dev9
:
# install py3
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
/bin/bash Miniconda3-latest-Linux-x86_64.sh -b -p miniconda3/
source miniconda3/bin/activate
# install TRAKO
CC=clang CXX=clang++ pip install trako==0.3.4.dev9
Hi @pieper , I just realized you posted the log about trakofy
but I am having issue with untrakofy
. Did you try the latter and didn't get any error either? To reiterate, I didn't get any error with trakofy
.
I did the untrakofy
too (look again at my post above).
Interesting that there could be a bug like this that depends on the version of python. I don't believe it's possible to install singularity on mac, but I might be able to test on linux.
Hi Steve, sorry about missing that. Actually, you can try to install just Python 3.8.3
and trako==0.3.4.dev9
on your MAC and try to reproduce the error. However, what are the versions of the above two you have in your MAC?
Hi Tashrif - I was able to install singularity on linux and exactly reproduce the issue you reported with untrakofy
. It says it is failing inside the vtk code.
It seems like this might be a singularity issue, or maybe a python or vtk issue. Are you able to try different version in your singularity image?
Hi Steve, I am happy to try a different version of Python/trako. However, the trako I installed came straight from PyPi i.e. Daniel. Nevertheless, what version of Python/trako would you recommend?
Maybe we can try the same versions that work in my build? Python 3.6.7 and VTK 8.2.0 are the current once we use for Slicer. Which version of VTK is in your singularity image?
Hi Steve, my VTK is 8.1.2
that came with the trako PyPi package.
Thanks @tashrifbillah, it seems the vtk version is part of the problem. I installed trako on my ubuntu 18.04 system using pip3 install trako
and it failed with the same core dump as you got in singularity.
When I installed trako it picked up vtk 9.0.1, which is the latest release so trako's requirement of >=8.1.2 resolved to that so I'm guessing that is the underlying problem.
It looks like the VTK team is only providing vtk9 for python3 so we'll need to see how to port. I'm going to see if I can get a developer build working with vtk9 to see if I can reproduce the issue.
In the mean time, it could make sense to try using Slicer's python environment in your singularity container. Can you give that a try?
It seems that the vtk file in question has several polylines with only a single point. In vtk8 this was a warning condition but in vtk9 it is now considered an error condition and leads to the observed error message and exit when using the vtp writer.
Here is an example of the warning when loading this data in slicer with vtk8:
Warning: In /opt/sr/VTK/Common/DataModel/vtkPolyData.cxx, line 993
vtkPolyData (0x7fd6646341f0): Building VTK_LINE 1140468 with only one point, but VTK_LINE needs at least two points. Check the input.
So I guess this data came from UKF and maybe the model decided there as no good first step for the seed so it puts out a degenerate polyline with just the seed point. @ljod let's discuss whether if this is the case and if so whether we need to keep these or not.
The same warnings occur with loading in vtk8 slicer build with either the original vtk file or with the trakofied/untrakofied vtp file, so I it seems trako is not introducing the issue.
untrakofy crashes at this location (note frame 5):
1 std::vector<vtkPolyData_detail::TaggedCellId>::operator[] stl_vector.h 1043 0x7f3e1ff04516
2 vtkPolyData_detail::CellMap::GetTag vtkPolyDataInternals.h 239 0x7f3e1ff04325
3 vtkPolyData::GetCellType vtkPolyData.h 730 0x7f3e2012759a
4 vtkDataSet::GetCellTypes vtkDataSet.cxx 326 0x7f3e1ff39865
5 vtkXMLUnstructuredDataWriter::ProcessRequest vtkXMLUnstructuredDataWriter.cxx 166 0x7f3e1b31418e
6 vtkExecutive::CallAlgorithm vtkExecutive.cxx 746 0x7f3e1fe25449
7 vtkDemandDrivenPipeline::ExecuteData vtkDemandDrivenPipeline.cxx 462 0x7f3e1fe1cf92
8 vtkCompositeDataPipeline::ExecuteData vtkCompositeDataPipeline.cxx 161 0x7f3e1fe113a7
9 vtkDemandDrivenPipeline::ProcessRequest vtkDemandDrivenPipeline.cxx 261 0x7f3e1fe1c642
10 vtkStreamingDemandDrivenPipeline::ProcessRequest vtkStreamingDemandDrivenPipeline.cxx 343 0x7f3e1fe8977e
11 vtkDemandDrivenPipeline::UpdateData vtkDemandDrivenPipeline.cxx 419 0x7f3e1fe1cdb1
12 vtkStreamingDemandDrivenPipeline::Update vtkStreamingDemandDrivenPipeline.cxx 417 0x7f3e1fe89b32
13 vtkStreamingDemandDrivenPipeline::Update vtkStreamingDemandDrivenPipeline.cxx 380 0x7f3e1fe899b7
14 vtkAlgorithm::Update vtkAlgorithm.cxx 1421 0x7f3e1fe082b8
15 vtkAlgorithm::Update vtkAlgorithm.cxx 1415 0x7f3e1fe0827f
16 PyvtkAlgorithm_Update_s2 vtkAlgorithmPython.cxx 2338 0x7f3e3b09da0f
17 PyvtkAlgorithm_Update vtkAlgorithmPython.cxx 2425 0x7f3e3b09dd30
18 _PyCFunction_FastCallDict methodobject.c 234 0x7f3e4c514713
19 _PyCFunction_FastCallKeywords methodobject.c 294 0x7f3e4c514a05
20 call_function ceval.c 4837 0x7f3e4c58e69c
... <More>
A good solution for this could be to run vtkCleanPolyData
at one (or more? or all?) of the following spots: before saving from UKF, after reading in trakofy, or before writing in trakofy.
I remember a long time ago asking for this single point output to be fixed in UKF and I believe it was, so I wonder about the origin and age of this input file.
However I think the problem is not limited to single points and is more severe based on the images. There are many straight lines near the cortex that are connecting one fiber end to another fiber beginning point. This is an off by one error in the vtk file. This looks like a bad conversion between tractography file formats or a major file write error in the program that created this tractography.
Yes I agree trako should guard against this and output a warning. But this file is really bad and should be fixed and not used for any analysis.
@haehn - can you have a look and if you like it update the pypi package?
@tashrifbillah - you may want to add the lines from the PR to your local version for now.
Hi @pieper , unfortunately adding lines in my local version did not resolve the error. If I pip install
the PR, do you think that would make a difference?
@tashrifbillah the change is implemented only for trakofy
so be sure you create a new tko file from the vtk file with the bad lines and then untrakofy
should work on that data. Can you confirm?
I did, but to no avail.
Would it be possible to (1) share here a snippet (python or c++) allowing to reproduce the problem or (2) create an issue in the VTK tracker ?
singularity pull library://tbillah/collection/trako.sif
singularity run --bind /path/to/host/data:/data trako.sif trakofy -i /data/2T-01009-state.vtk -o /data/2T-01009-state.tko
singularity run --bind /path/to/host/data:/data trako.sif untrakofy -i /data/2T-01009-state.tko -o /data/2T-01009-state.vtp
Hi, is this what you asked for? But this is before Steve's PR.
is this what you asked for? Not exactly
I was looking to have a snippet using only VTK 9.
Indeed, based on the previous comment:
In vtk8 this was a warning condition but in vtk9 it is now considered an error condition and leads to the observed error message and exit when using the vtp writer
I anticipate that having such snippet would allow us to answer these questions:
cc: @pieper
Looking a bit more, I do agree @jcfr that this is a regression in vtk that should be fixed.
The code in vtk9 aborts from building cells when it hits a degenerate primitive, meaning that the vtkCellMap
is null, leading to a crash in vtkXMLUnstructuredDataWriter
.
The corresponding code in vtk8.2 prints the warning but keeps the cell as-is and keeps looping, so the vtkCellTypes
is valid.
It should be trivial to make an example for this (try to save a vtkPolyData
with one line with one point to vtp), and even to fix the code in vtk9 (generate a warning instead of an error), but for trako the first priority will be to clean the old data so I'll try sort out why my PR code isn't working for @tashrifbillah.
I did, but to no avail.
Hi @tashrifbillah can you give the PR another try? I just ran your test again on the data you provided with vtk9 and as long as I generate a new tko file using trakofy that has the patch I do not get a crash on vtk9 (and confirmed it does crash if I don't include the PR code).
Since I confirmed that it works for me I went ahead and merged this in trako, so the next pypi package version will have the fix.
Hi @pieper , my apologies for the last report. You were right--no more error. In retrospect, I might not have affected the change inside my container properly. Thank you.
I updated my container with the patch. In future, I can try to build against the next PyPi package. In the meantime, you can access it as:
singularity pull trako.sif library://tbillah/collection/trako.sif:0.1
For reference, I filed an issue on the vtk tracker: https://gitlab.kitware.com/vtk/vtk/-/issues/18059
I compressed a
vtk
totko
and then tried uncompression--tko
tovtp
but ran in to the following error:I did not provide a
--config
file for the former.Does the error look familiar?