search

constantinpape / cluster_tools

Distributed segmentation for bio-image-analysis
MIT License
35 stars 14 forks source link

Pipeline has problems with non-contiguous numpy arrays #6

Closed snajder-r closed 3 years ago

snajder-r commented 5 years ago

I use numpy version 1.16.1 from conda-forge.

Whenever vigra.analysis.relabelConsecutive is called within the Multicut workflow, I get an error that the argument types did not match with any C++ signature. i checked that the datatype is correct. What fixes the problem for me is adding a line that makes the numpy arrays contiguous.

Below is a small code example that recreates the problem by writing an n5 volume of a graph, loading the graph, and then calling relabelConsecutive on the nodes. 

import nifty.distributed as ndist
import z5py
import numpy as np
import vigra.analysis

filename = "test.n5"

with z5py.File(filename, "w") as f:
    g = f.create_group('/graph')

    g.attrs['numberOfEdges'] = 1
    g.attrs['numberOfNodes'] = 4

    g.create_dataset("labels", (4,), dtype="uint64", data=np.array([0,1,2,3],dtype=np.uint64))
    g.create_dataset("edges", (1,2), dtype="uint64", data=np.array([[0,1]],dtype=np.uint64))

graph = ndist.Graph(filename+"/graph/")
nodes = graph.nodes()

# nodes = np.ascontiguousarray(nodes)
nodes_relabeled,_,_ = vigra.analysis.relabelConsecutive(nodes.astype(np.uint64))
print(nodes_relabeled)

If the line "nodes = np.ascontiguousarray(nodes)" is activated, it works perfectly. Without it, I get the following error:

Boost.Python.ArgumentError: Python argument types in
    vigra.analysis.relabelConsecutive(numpy.ndarray)
did not match C++ signature:
    relabelConsecutive(vigra::NumpyArray<1u, vigra::Singleband<unsigned char>, vigra::StridedArrayTag> labels, unsigned char start_label=1, bool keep_zeros=True, vigra::NumpyArray<1u, vigra::Singleband<unsigned char>, vigra::StridedArrayTag> out=None)
    relabelConsecutive(vigra::NumpyArray<1u, vigra::Singleband<unsigned int>, vigra::StridedArrayTag> labels, unsigned int start_label=1, bool keep_zeros=True, vigra::NumpyArray<1u, vigra::Singleband<unsigned int>, vigra::StridedArrayTag> out=None)
    relabelConsecutive(vigra::NumpyArray<1u, vigra::Singleband<unsigned long>, vigra::StridedArrayTag> labels, unsigned long start_label=1, bool keep_zeros=True, vigra::NumpyArray<1u, vigra::Singleband<unsigned long>, vigra::StridedArrayTag> out=None)
    relabelConsecutive(vigra::NumpyArray<1u, vigra::Singleband<unsigned long>, vigra::StridedArrayTag> labels, unsigned int start_label=1, bool keep_zeros=True, vigra::NumpyArray<1u, vigra::Singleband<unsigned int>, vigra::StridedArrayTag> out=None)
    relabelConsecutive(vigra::NumpyArray<2u, vigra::Singleband<unsigned char>, vigra::StridedArrayTag> labels, unsigned char start_label=1, bool keep_zeros=True, vigra::NumpyArray<2u, vigra::Singleband<unsigned char>, vigra::StridedArrayTag> out=None)
    relabelConsecutive(vigra::NumpyArray<2u, vigra::Singleband<unsigned int>, vigra::StridedArrayTag> labels, unsigned int start_label=1, bool keep_zeros=True, vigra::NumpyArray<2u, vigra::Singleband<unsigned int>, vigra::StridedArrayTag> out=None)
    relabelConsecutive(vigra::NumpyArray<2u, vigra::Singleband<unsigned long>, vigra::StridedArrayTag> labels, unsigned long start_label=1, bool keep_zeros=True, vigra::NumpyArray<2u, vigra::Singleband<unsigned long>, vigra::StridedArrayTag> out=None)
    relabelConsecutive(vigra::NumpyArray<2u, vigra::Singleband<unsigned long>, vigra::StridedArrayTag> labels, unsigned int start_label=1, bool keep_zeros=True, vigra::NumpyArray<2u, vigra::Singleband<unsigned int>, vigra::StridedArrayTag> out=None)
    relabelConsecutive(vigra::NumpyArray<3u, vigra::Singleband<unsigned char>, vigra::StridedArrayTag> labels, unsigned char start_label=1, bool keep_zeros=True, vigra::NumpyArray<3u, vigra::Singleband<unsigned char>, vigra::StridedArrayTag> out=None)
    relabelConsecutive(vigra::NumpyArray<3u, vigra::Singleband<unsigned int>, vigra::StridedArrayTag> labels, unsigned int start_label=1, bool keep_zeros=True, vigra::NumpyArray<3u, vigra::Singleband<unsigned int>, vigra::StridedArrayTag> out=None)
    relabelConsecutive(vigra::NumpyArray<3u, vigra::Singleband<unsigned long>, vigra::StridedArrayTag> labels, unsigned long start_label=1, bool keep_zeros=True, vigra::NumpyArray<3u, vigra::Singleband<unsigned long>, vigra::StridedArrayTag> out=None)
    relabelConsecutive(vigra::NumpyArray<3u, vigra::Singleband<unsigned long>, vigra::StridedArrayTag> labels, unsigned int start_label=1, bool keep_zeros=True, vigra::NumpyArray<3u, vigra::Singleband<unsigned int>, vigra::StridedArrayTag> out=None)
constantinpape commented 5 years ago

This is indeed weird. Could you please do the following:

Also, you mentioned that you get a seg-fault with different numpy versions earlier. Can you also reproduce this with such an easy example?

constantinpape commented 5 years ago

I just checked and for me this runs without any issues (DON'T need the ascontiguousarray). I am using vigra 1.11.1 and numpy 1.15.4 from conda-forge and z5py / nifty build from recent master.

So I assume that this is some subtle numpy incompatibility issue. Should be fixed once nifty is on conda-forge and the conda env has proper numpy version pinning. See https://github.com/conda-forge/staged-recipes/pull/7763

snajder-r commented 5 years ago

Sorry for the late reply.

print the array flags before and after the call to ascontiguousarray and post it here

Before:

C_CONTIGUOUS : True
F_CONTIGUOUS : True
OWNDATA : True
WRITEABLE : True
ALIGNED : True
WRITEBACKIFCOPY : False
UPDATEIFCOPY : False

After:

C_CONTIGUOUS : True
F_CONTIGUOUS : True
OWNDATA : False
WRITEABLE : True
ALIGNED : True
WRITEBACKIFCOPY : False
UPDATEIFCOPY : False

So, in the version it works, OWNDATA is False and in the version where it doesn't work, OWNDATA is True. That seems to be the only difference in the flags. It even says it is contiguous before I run ascontiguousarray.

post the conda env you are using (conda list) here

This is the conda environment where I see the 'did not match C++ signature' error:

# Name                    Version                   Build  Channel
asn1crypto                0.24.0                py37_1003    conda-forge
backcall                  0.1.0                      py_0    conda-forge
blas                      1.1                    openblas    conda-forge
blosc                     1.15.1            hf484d3e_1002    conda-forge
boost                     1.68.0          py37h8619c78_1001    conda-forge
boost-cpp                 1.68.0            h11c811c_1000    conda-forge
botocore                  1.12.82                    py_0  
bzip2                     1.0.6                h14c3975_5  
ca-certificates           2018.11.29           ha4d7672_0    conda-forge
certifi                   2018.11.29            py37_1000    conda-forge
cffi                      1.12.1           py37h9745a5d_0    conda-forge
cloudpickle               0.7.0                      py_0  
cryptography              2.5              py37h1ba5d50_0  
cycler                    0.10.0                     py_1    conda-forge
cytoolz                   0.9.0.1         py37h14c3975_1001    conda-forge
dask-core                 1.1.1                      py_0  
dbus                      1.13.6               h746ee38_0  
decorator                 4.3.2                      py_0    conda-forge
docutils                  0.14                  py37_1001    conda-forge
expat                     2.2.6                he6710b0_0  
fftw                      3.3.8                h7b6447c_3  
fontconfig                2.13.0               h9420a91_0  
freetype                  2.9.1                h8a8886c_1  
glib                      2.56.2               hd408876_0  
gst-plugins-base          1.14.0               hbbd80ab_1  
gstreamer                 1.14.0               hb453b48_1  
h5py                      2.8.0           py37hfb8609e_1004    conda-forge
hdf5                      1.10.3            hba1933b_1001    conda-forge
icu                       58.2                 h9c2bf20_1  
idna                      2.8                   py37_1000    conda-forge
imageio                   2.5.0                    py37_0    conda-forge
intel-openmp              2019.1                      144  
ipykernel                 5.1.0           py37h24bf2e0_1002    conda-forge
ipython                   7.3.0            py37h24bf2e0_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
jedi                      0.13.2                py37_1000    conda-forge
jmespath                  0.9.3                      py_1    conda-forge
jpeg                      9c                h14c3975_1001    conda-forge
jupyter_client            5.2.4                      py_1    conda-forge
jupyter_core              4.4.0                      py_0    conda-forge
kiwisolver                1.0.1           py37h6bb024c_1002    conda-forge
libedit                   3.1.20181209         hc058e9b_0  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 8.2.0                hdf63c60_1  
libgfortran-ng            7.3.0                hdf63c60_0  
libopenblas               0.3.3                h5a2b251_3  
libpng                    1.6.36               hbc83047_0  
libsodium                 1.0.16               h1bed415_0  
libstdcxx-ng              8.2.0                hdf63c60_1  
libtiff                   4.0.10               h2733197_2  
libuuid                   1.0.3                h1bed415_2  
libxcb                    1.13                 h1bed415_1  
libxml2                   2.9.9                he19cac6_0  
lockfile                  0.12.2                     py_1    conda-forge
luigi                     2.8.3                 py37_1000    conda-forge
matplotlib-base           3.0.2           py37h167e16e_1002    conda-forge
mkl                       2019.1                      144  
ncurses                   6.1                  he6710b0_1  
networkx                  2.2                        py_1    conda-forge
nifty                     v0.4.1                   py37_0    cpape
numpy                     1.16.1          py37_blas_openblash1522bff_0  [blas_openblas]  conda-forge
numpy-base                1.15.4           py37h2f8d375_0  
olefile                   0.46                       py_0    conda-forge
openblas                  0.3.3             h9ac9557_1001    conda-forge
openlibm                  0.6.0             h14c3975_1000    conda-forge
openssl                   1.1.1a            h14c3975_1000    conda-forge
parso                     0.3.4                      py_0    conda-forge
pcre                      8.42                 h439df22_0  
pexpect                   4.6.0                 py37_1000    conda-forge
pickleshare               0.7.5                 py37_1000    conda-forge
pillow                    5.4.1           py37h00a061d_1000    conda-forge
pip                       19.0.2                   py37_0    conda-forge
prompt_toolkit            2.0.8                      py_0  
ptyprocess                0.6.0                 py37_1000    conda-forge
pycparser                 2.19                       py_0    conda-forge
pygments                  2.3.1                      py_0    conda-forge
pyopenssl                 19.0.0                   py37_0    conda-forge
pyparsing                 2.3.1                      py_0    conda-forge
pyqt                      5.9.2            py37h05f1152_2  
pysocks                   1.6.8                 py37_1002    conda-forge
python                    3.7.2                h0371630_0  
python-daemon             2.2.1                    py37_0  
python-dateutil           2.8.0                      py_0    conda-forge
pytz                      2018.9                     py_0    conda-forge
pywavelets                1.0.1           py37h3010b51_1000    conda-forge
pyzmq                     17.1.2           py37he6710b0_2  
qt                        5.9.7                h5867ecd_1  
readline                  7.0                  h7b6447c_5  
scikit-image              0.14.2           py37hf484d3e_1    conda-forge
scikit-learn              0.20.2          py37_blas_openblashebff5e3_1400  [blas_openblas]  conda-forge
scipy                     1.2.1           py37_blas_openblash1522bff_0  [blas_openblas]  conda-forge
setuptools                40.8.0                   py37_0    conda-forge
sip                       4.19.8          py37hf484d3e_1000    conda-forge
six                       1.12.0                py37_1000    conda-forge
sqlite                    3.26.0               h7b6447c_0  
tk                        8.6.9             h84994c4_1000    conda-forge
toolz                     0.9.0                      py_1    conda-forge
tornado                   4.5.3           py37h14c3975_1001    conda-forge
traitlets                 4.3.2                 py37_1000    conda-forge
urllib3                   1.24.1                py37_1000    conda-forge
vigra                     1.11.1          py37hb287d74_1007    conda-forge
wcwidth                   0.1.7                      py_1    conda-forge
wheel                     0.33.0                   py37_0    conda-forge
xz                        5.2.4                h14c3975_4  
z5py                      1.4.1            py37h9082789_1    conda-forge
zeromq                    4.3.1                he6710b0_3  
zlib                      1.2.11               h7b6447c_3  
zstd                      1.3.7                h0b5b093_0

Also, you mentioned that you get a seg-fault with different numpy versions earlier. Can you also reproduce this with such an easy example?

Yes, actually the same exact code from above reproduces that error. The conda environment where I see the segfault is:

# Name                    Version                   Build  Channel
asn1crypto                0.24.0                py37_1003    conda-forge
blas                      1.0                         mkl  
blosc                     1.15.1            hf484d3e_1002    conda-forge
boost                     1.68.0          py37h8619c78_1001    conda-forge
boost-cpp                 1.68.0            h11c811c_1000    conda-forge
botocore                  1.12.96                    py_0    conda-forge
bzip2                     1.0.6             h14c3975_1002    conda-forge
ca-certificates           2018.11.29           ha4d7672_0    conda-forge
certifi                   2018.11.29            py37_1000    conda-forge
cffi                      1.12.1           py37h9745a5d_0    conda-forge
cloudpickle               0.7.0                      py_0    conda-forge
cryptography              2.5              py37hb7f436b_1    conda-forge
cycler                    0.10.0                     py_1    conda-forge
cytoolz                   0.9.0.1         py37h14c3975_1001    conda-forge
dask-core                 1.1.1                      py_0    conda-forge
decorator                 4.3.2                      py_0    conda-forge
docutils                  0.14                  py37_1001    conda-forge
fftw                      3.3.8             h14c3975_1002    conda-forge
freetype                  2.9.1             h94bbf69_1005    conda-forge
h5py                      2.8.0           py37hfb8609e_1004    conda-forge
hdf5                      1.10.3            hba1933b_1001    conda-forge
icu                       58.2              hf484d3e_1000    conda-forge
idna                      2.8                   py37_1000    conda-forge
imageio                   2.5.0                    py37_0    conda-forge
intel-openmp              2019.1                      144  
jmespath                  0.9.3                      py_1    conda-forge
jpeg                      9c                h14c3975_1001    conda-forge
kiwisolver                1.0.1           py37h6bb024c_1002    conda-forge
libffi                    3.2.1             hf484d3e_1005    conda-forge
libgcc-ng                 7.3.0                hdf63c60_0    conda-forge
libgfortran-ng            7.2.0                hdf63c60_3    conda-forge
libpng                    1.6.36            h84994c4_1000    conda-forge
libstdcxx-ng              7.3.0                hdf63c60_0    conda-forge
libtiff                   4.0.10            h648cc4a_1001    conda-forge
lockfile                  0.12.2                     py_1    conda-forge
luigi                     2.8.3                 py37_1000    conda-forge
matplotlib-base           3.0.2           py37h167e16e_1002    conda-forge
mkl                       2019.1                      144  
mkl_fft                   1.0.10           py37h14c3975_1    conda-forge
mkl_random                1.0.2            py37h637b7d7_2    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
networkx                  2.2                        py_1    conda-forge
nifty                     v0.4.1                   py37_0    cpape
numpy                     1.15.4           py37h7e9f1db_0  
numpy-base                1.15.4           py37hde5b4d6_0  
olefile                   0.46                       py_0    conda-forge
openblas                  0.3.3             h9ac9557_1001    conda-forge
openssl                   1.0.2p            h14c3975_1002    conda-forge
pillow                    5.4.1           py37h00a061d_1000    conda-forge
pip                       19.0.2                   py37_0    conda-forge
pycparser                 2.19                       py_0    conda-forge
pyopenssl                 19.0.0                   py37_0    conda-forge
pyparsing                 2.3.1                      py_0    conda-forge
pysocks                   1.6.8                 py37_1002    conda-forge
python                    3.7.0             hd21baee_1006    conda-forge
python-daemon             2.2.1                    py37_0  
python-dateutil           2.8.0                      py_0    conda-forge
pywavelets                1.0.1           py37h3010b51_1000    conda-forge
readline                  7.0               hf8c457e_1001    conda-forge
scikit-image              0.14.2           py37hf484d3e_1    conda-forge
scikit-learn              0.20.2           py37hd81dba3_0  
scipy                     1.2.0            py37h7c811a0_0  
setuptools                40.8.0                   py37_0    conda-forge
six                       1.12.0                py37_1000    conda-forge
sqlite                    3.26.0            h67949de_1000    conda-forge
tk                        8.6.9             h84994c4_1000    conda-forge
toolz                     0.9.0                      py_1    conda-forge
tornado                   4.5.3           py37h14c3975_1001    conda-forge
urllib3                   1.24.1                py37_1000    conda-forge
vigra                     1.11.1          py37hb287d74_1007    conda-forge
wheel                     0.33.0                   py37_0    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
z5py                      1.4.1            py37h9082789_0    conda-forge
zlib                      1.2.11            h14c3975_1004    conda-forge