Closed mezwick closed 2 years ago
Can you check the values of g.ecount()
and g.vcount()
just before the g.es['weight']
line and let us know the output?
I looked up the source of Python 3.9.12 and it seems like the offending call is a call to PyList_New()
with a negative argument. It's completely a stab in the dark yet but I'd suspect this call inside src/_igraph/edgeseqobject.c
:
list = PyList_New(no_of_edges);
where no_of_edges
is a long
. If this is 32-bit and signed, then you are likely to get problems if no_of_edges
is larger than or equal to 2^31.
Yes, but 18 million is far from 2^31, so how did we get here?
18 million is the number of vertices, not the number of edges.
Ah, right.
@mezwick igraph 0.10 will have overflow checks, and will also use 64-bit integers, so it should be immune to such problems (hopefully!). But with 0.9, the effective limits are at most 2^31 - 1
vertices and 2^30
edges. These values are very likely to cause troubles, and slightly smaller values might too.
If you are willing to compile it, you could try the development version of 0.10 right now. Instructions are here: https://github.com/igraph/python-igraph/#compiling-the-development-version Use the develop
branch.
I'm not sure what the state of overflow checks is in the Python glue code, @ntamas ? The C core is in a pretty good shape already, though some stochastic graph generators can still overflow. Anyway, with 64-bit integers overflow will not be practically possible.
The Python glue code still uses a mixture of Py_ssize_t
, long int
and igraph_integer_t
and it has to be sorted out eventually, but I'm already doing too many things at the same time so I don't know when I can sort it out properly.
Thanks for the quick responses. :) So illuminating already! Sounds like i have hit graph size limits.
g.ecount() = -1939291626
how do you interpret a negative ecount?
g.vcount() = 18543700
adjacency
= <18543700x18543700 sparse matrix of type '<class 'numpy.float32'>'
with 2,355,675,670 stored elements in Compressed Sparse Row format>
A 0.9 subset of my data (16,689,330 vertices) is able to run.
So, it would seem the number of edges in the full dataset are too many for the 32 bit integer limit? I will look into running igraph 0.10 to see if the dev version already works in my case.
Thanks for taking the time to respond.
Best, Mezwick
@mezwick I've briefly checked the code in develop
and it seems like we've already made some efforts to make sure that python-igraph
works correctly even when you hit the 32-bit limit on the edge counts, but we haven't had the chance to test it thoroughly. Please give it a go with the develop
branch and report any issues that you are running into as we are really keen on fixing issues of this kind for the next version.
If you need help with compiling the development branch, let me know.
how do you interpret a negative ecount?
It's the result of integer overflow. You have more than 2^31 - 1 = 2,147,483,647 edges.
Thanks.
I have run...
pip install git+https://github.com/igraph/python-igraph@develop
Is that all i need to do?
(scanpy_py3pt9_igraph0pt10) root@dcd19bf51c7b:/opt/notebooks# pip install git+https://github.com/igraph/python-igraph@develop
Collecting git+https://github.com/igraph/python-igraph@develop
Cloning https://github.com/igraph/python-igraph (to revision develop) to /tmp/pip-req-build-utwxolmm
Running command git clone -q https://github.com/igraph/python-igraph /tmp/pip-req-build-utwxolmm
Running command git checkout -b develop --track origin/develop
Switched to a new branch 'develop'
Branch 'develop' set up to track remote branch 'develop' from 'origin'.
Resolved https://github.com/igraph/python-igraph to commit dedb177b58463a720fad5bfd5862fa769aea4ac6
Running command git submodule update --init --recursive -q
Requirement already satisfied: texttable>=1.6.2 in /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages (from igraph==0.9.10) (1.6.4)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
It does not appear to have updated the python-igraph
package number
Sorry, i suspect i am missing something obvious. Installing the dev branch is new territory for me
You might need to run pip install --upgrade --force-reinstall git+https://github.com/igraph/python-igraph@develop Collecting git+https://github.com/igraph/python-igraph@develop
in case pip
is trying to be smart and considers the new version being identical to the existing one.
i ran the following which took me a little further...
pip install cmake
pip install --upgrade --force-reinstall git+https://github.com/igraph/python-igraph@develop
but i appear to have hit a wall as need to install flex
and bison
. These appear to require sudo install permissions, which i don't have. Will look into getting these packages (but think i will not get them easily/quickly)
(scanpy_py3pt9_igraph0pt10) root@dcd19bf51c7b:/opt/git/python-igraph# pip install cmake
Collecting cmake
Downloading cmake-3.22.4-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (22.5 MB)
|████████████████████████████████| 22.5 MB 16.0 MB/s
Installing collected packages: cmake
Successfully installed cmake-3.22.4
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
(scanpy_py3pt9_igraph0pt10) root@dcd19bf51c7b:/opt/git/python-igraph# pip install --upgrade --force-reinstall git+https://github.com/igraph/python-igraph@develop
Collecting git+https://github.com/igraph/python-igraph@develop
Cloning https://github.com/igraph/python-igraph (to revision develop) to /tmp/pip-req-build-h6yacg3_
Running command git clone -q https://github.com/igraph/python-igraph /tmp/pip-req-build-h6yacg3_
Running command git checkout -b develop --track origin/develop
Switched to a new branch 'develop'
Branch 'develop' set up to track remote branch 'develop' from 'origin'.
Resolved https://github.com/igraph/python-igraph to commit 7c81be69411bfb4cb0f2da7c84489ebf60cb2d0e
Running command git submodule update --init --recursive -q
Collecting texttable>=1.6.2
Using cached texttable-1.6.4-py2.py3-none-any.whl (10 kB)
Building wheels for collected packages: igraph
Building wheel for igraph (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /opt/conda/envs/scanpy_py3pt9_igraph0pt10/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-h6yacg3_/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-h6yacg3_/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-47s6j66q
cwd: /tmp/pip-req-build-h6yacg3_/
Complete output (145 lines):
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/igraph
copying src/igraph/matching.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/sparse_matrix.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/summary.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/version.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/__init__.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/adjacency.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/automorphisms.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/basic.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/bipartite.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/clustering.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/community.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/configuration.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/cut.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/datatypes.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/formula.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/layout.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/seq.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/statistics.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/structural.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/utils.py -> build/lib.linux-x86_64-3.9/igraph
creating build/lib.linux-x86_64-3.9/igraph/app
copying src/igraph/app/__init__.py -> build/lib.linux-x86_64-3.9/igraph/app
copying src/igraph/app/shell.py -> build/lib.linux-x86_64-3.9/igraph/app
creating build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/metamagic.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/baseclasses.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/colors.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/shapes.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/text.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing
creating build/lib.linux-x86_64-3.9/igraph/remote
copying src/igraph/remote/__init__.py -> build/lib.linux-x86_64-3.9/igraph/remote
copying src/igraph/remote/gephi.py -> build/lib.linux-x86_64-3.9/igraph/remote
creating build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/__init__.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/adjacency.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/bipartite.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/files.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/images.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/libraries.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/objects.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/random.py -> build/lib.linux-x86_64-3.9/igraph/io
creating build/lib.linux-x86_64-3.9/igraph/operators
copying src/igraph/operators/__init__.py -> build/lib.linux-x86_64-3.9/igraph/operators
copying src/igraph/operators/functions.py -> build/lib.linux-x86_64-3.9/igraph/operators
copying src/igraph/operators/methods.py -> build/lib.linux-x86_64-3.9/igraph/operators
creating build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/base.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/coord.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/dendrogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/edge.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/histogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/matrix.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/palette.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/plot.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/polygon.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/text.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/vertex.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
creating build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/dendrogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/edge.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/histogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/matrix.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/palette.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/polygon.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/vertex.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
creating build/lib.linux-x86_64-3.9/igraph/drawing/plotly
copying src/igraph/drawing/plotly/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
copying src/igraph/drawing/plotly/edge.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
copying src/igraph/drawing/plotly/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
copying src/igraph/drawing/plotly/polygon.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
copying src/igraph/drawing/plotly/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
copying src/igraph/drawing/plotly/vertex.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
running build_ext
running build_c_core
We are going to build the C core of igraph.
Source folder: vendor/source/igraph
Build folder: vendor/build/igraph
Install folder: vendor/install/igraph
Configuring build...
-- Setting build type to 'Release' as none was specified.
-- Version number from Git: 0.10.0-dev+c9ed1c6b
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test COMPILER_SUPPORTS_NO_VARARGS_FLAG
-- Performing Test COMPILER_SUPPORTS_NO_VARARGS_FLAG - Success
-- Performing Test COMPILER_SUPPORTS_NO_UNKNOWN_WARNING_OPTION_FLAG
-- Performing Test COMPILER_SUPPORTS_NO_UNKNOWN_WARNING_OPTION_FLAG - Success
-- Performing Test COMPILER_HAS_DEPRECATED_ENUMVAL_ATTR
-- Performing Test COMPILER_HAS_DEPRECATED_ENUMVAL_ATTR - Success
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Could NOT find FLEX (missing: FLEX_EXECUTABLE)
-- Could NOT find BISON (missing: BISON_EXECUTABLE)
-- Looking for stpcpy
-- Looking for stpcpy - found
-- Looking for strcasecmp
-- Looking for strcasecmp - found
-- Looking for strdup
-- Looking for strdup - found
-- Looking for strndup
-- Looking for strndup - found
-- Looking for _stricmp
-- Looking for _stricmp - not found
CMake Error at src/CMakeLists.txt:24 (bison_target):
Unknown CMake command "bison_target".
-- Configuring incomplete, errors occurred!
See also "/tmp/pip-req-build-h6yacg3_/vendor/build/igraph/CMakeFiles/CMakeOutput.log".
See also "/tmp/pip-req-build-h6yacg3_/vendor/build/igraph/CMakeFiles/CMakeError.log".
Build failed for the C core of igraph.
----------------------------------------
ERROR: Failed building wheel for igraph
Running setup.py clean for igraph
Failed to build igraph
Installing collected packages: texttable, igraph
Attempting uninstall: texttable
Found existing installation: texttable 1.6.4
Uninstalling texttable-1.6.4:
Successfully uninstalled texttable-1.6.4
Attempting uninstall: igraph
Found existing installation: igraph 0.9.10
Uninstalling igraph-0.9.10:
Successfully uninstalled igraph-0.9.10
Running setup.py install for igraph ... error
ERROR: Command errored out with exit status 1:
command: /opt/conda/envs/scanpy_py3pt9_igraph0pt10/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-h6yacg3_/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-h6yacg3_/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-zix7h65k/install-record.txt --single-version-externally-managed --compile --install-headers /opt/conda/envs/scanpy_py3pt9_igraph0pt10/include/python3.9/igraph
cwd: /tmp/pip-req-build-h6yacg3_/
Complete output (109 lines):
running install
/opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/igraph
copying src/igraph/matching.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/sparse_matrix.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/summary.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/version.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/__init__.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/adjacency.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/automorphisms.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/basic.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/bipartite.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/clustering.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/community.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/configuration.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/cut.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/datatypes.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/formula.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/layout.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/seq.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/statistics.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/structural.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/utils.py -> build/lib.linux-x86_64-3.9/igraph
creating build/lib.linux-x86_64-3.9/igraph/app
copying src/igraph/app/__init__.py -> build/lib.linux-x86_64-3.9/igraph/app
copying src/igraph/app/shell.py -> build/lib.linux-x86_64-3.9/igraph/app
creating build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/metamagic.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/baseclasses.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/colors.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/shapes.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/text.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing
creating build/lib.linux-x86_64-3.9/igraph/remote
copying src/igraph/remote/__init__.py -> build/lib.linux-x86_64-3.9/igraph/remote
copying src/igraph/remote/gephi.py -> build/lib.linux-x86_64-3.9/igraph/remote
creating build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/__init__.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/adjacency.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/bipartite.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/files.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/images.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/libraries.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/objects.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/random.py -> build/lib.linux-x86_64-3.9/igraph/io
creating build/lib.linux-x86_64-3.9/igraph/operators
copying src/igraph/operators/__init__.py -> build/lib.linux-x86_64-3.9/igraph/operators
copying src/igraph/operators/functions.py -> build/lib.linux-x86_64-3.9/igraph/operators
copying src/igraph/operators/methods.py -> build/lib.linux-x86_64-3.9/igraph/operators
creating build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/base.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/coord.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/dendrogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/edge.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/histogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/matrix.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/palette.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/plot.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/polygon.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/text.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/vertex.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
creating build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/dendrogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/edge.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/histogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/matrix.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/palette.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/polygon.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/vertex.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
creating build/lib.linux-x86_64-3.9/igraph/drawing/plotly
copying src/igraph/drawing/plotly/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
copying src/igraph/drawing/plotly/edge.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
copying src/igraph/drawing/plotly/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
copying src/igraph/drawing/plotly/polygon.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
copying src/igraph/drawing/plotly/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
copying src/igraph/drawing/plotly/vertex.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
running build_ext
running build_c_core
We are going to build the C core of igraph.
Source folder: vendor/source/igraph
Build folder: vendor/build/igraph
Install folder: vendor/install/igraph
Configuring build...
-- Version number from Git: 0.10.0-dev+c9ed1c6b
-- Could NOT find FLEX (missing: FLEX_EXECUTABLE)
-- Could NOT find BISON (missing: BISON_EXECUTABLE)
CMake Error at src/CMakeLists.txt:24 (bison_target):
Unknown CMake command "bison_target".
-- Configuring incomplete, errors occurred!
See also "/tmp/pip-req-build-h6yacg3_/vendor/build/igraph/CMakeFiles/CMakeOutput.log".
See also "/tmp/pip-req-build-h6yacg3_/vendor/build/igraph/CMakeFiles/CMakeError.log".
Build failed for the C core of igraph.
----------------------------------------
Rolling back uninstall of igraph
Moving to /opt/conda/envs/scanpy_py3pt9_igraph0pt10/bin/igraph
from /tmp/pip-uninstall-t33v86fj/igraph
Moving to /opt/conda/envs/scanpy_py3pt9_igraph0pt10/include/python3.9/igraph/
from /opt/conda/envs/scanpy_py3pt9_igraph0pt10/include/python3.9/~graph
Moving to /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages/igraph-0.9.10.dist-info/
from /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages/~graph-0.9.10.dist-info
Moving to /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages/igraph/
from /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages/~graph
ERROR: Command errored out with exit status 1: /opt/conda/envs/scanpy_py3pt9_igraph0pt10/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-h6yacg3_/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-h6yacg3_/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-zix7h65k/install-record.txt --single-version-externally-managed --compile --install-headers /opt/conda/envs/scanpy_py3pt9_igraph0pt10/include/python3.9/igraph Check the logs for full command output.
If you don't have root access, you should still be able to install flex and bison with conda.
i did try pip install of flex and bison in my conda-env, but appear to get the same error...?
(scanpy_py3pt9_igraph0pt10) root@dcd19bf51c7b:/opt/git/python-igraph# pip install bison
Collecting bison
Downloading bison-0.1.3-py2.py3-none-any.whl (24 kB)
Collecting pyyaml>=5.4
Downloading PyYAML-6.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (661 kB)
|████████████████████████████████| 661 kB 33.0 MB/s
Installing collected packages: pyyaml, bison
Successfully installed bison-0.1.3 pyyaml-6.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
(scanpy_py3pt9_igraph0pt10) root@dcd19bf51c7b:/opt/git/python-igraph# pip install flex
Collecting flex
Downloading flex-6.14.1.tar.gz (40 kB)
|████████████████████████████████| 40 kB 4.3 MB/s
Requirement already satisfied: six>=1.7.3 in /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages (from flex) (1.16.0)
Requirement already satisfied: PyYAML>=3.11 in /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages (from flex) (6.0)
Collecting validate-email>=1.2
Downloading validate_email-1.3.tar.gz (4.7 kB)
Collecting rfc3987>=1.3.4
Downloading rfc3987-1.3.8-py2.py3-none-any.whl (13 kB)
Requirement already satisfied: requests>=2.4.3 in /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages (from flex) (2.27.1)
Collecting strict-rfc3339>=0.7
Downloading strict-rfc3339-0.7.tar.gz (17 kB)
Collecting click>=3.3
Downloading click-8.1.2-py3-none-any.whl (96 kB)
|████████████████████████████████| 96 kB 6.4 MB/s
Collecting jsonpointer>=1.7
Downloading jsonpointer-2.3-py2.py3-none-any.whl (7.8 kB)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages (from requests>=2.4.3->flex) (1.26.9)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages (from requests>=2.4.3->flex) (2021.10.8)
Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages (from requests>=2.4.3->flex) (2.0.12)
Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages (from requests>=2.4.3->flex) (3.3)
Building wheels for collected packages: flex, strict-rfc3339, validate-email
Building wheel for flex (setup.py) ... done
Created wheel for flex: filename=flex-6.14.1-py3-none-any.whl size=77276 sha256=ce28b92100fcdd97b5cfef49b531df243f1779830a0fa640ff61acf565df5514
Stored in directory: /home/dnanexus/.cache/pip/wheels/e9/9a/ef/ea72c07b1be4b3dfebba94d22f905ff8ad53be48419e6114d1
Building wheel for strict-rfc3339 (setup.py) ... done
Created wheel for strict-rfc3339: filename=strict_rfc3339-0.7-py3-none-any.whl size=18149 sha256=33106798d06e29316fdd30084a2f5766e3ae27448d79e03f7a12c91ae435ad38
Stored in directory: /home/dnanexus/.cache/pip/wheels/25/38/74/7ec7f77ec64b2907430120931ba588b40e6e26f02d4df5be35
Building wheel for validate-email (setup.py) ... done
Created wheel for validate-email: filename=validate_email-1.3-py3-none-any.whl size=5482 sha256=fb788ddd7e3faf3a923b38a61bea2c01518db8d62513c0859aff2426542f7891
Stored in directory: /home/dnanexus/.cache/pip/wheels/84/19/e3/329ee20a29b195fa193ee3af2475fa9392b08dde8ba893c714
Successfully built flex strict-rfc3339 validate-email
Installing collected packages: validate-email, strict-rfc3339, rfc3987, jsonpointer, click, flex
Successfully installed click-8.1.2 flex-6.14.1 jsonpointer-2.3 rfc3987-1.3.8 strict-rfc3339-0.7 validate-email-1.3
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
(scanpy_py3pt9_igraph0pt10) root@dcd19bf51c7b:/opt/git/python-igraph# pip install --upgrade --force-reinstall git+https://github.com/igraph/python-igraph@develop
Collecting git+https://github.com/igraph/python-igraph@develop
Cloning https://github.com/igraph/python-igraph (to revision develop) to /tmp/pip-req-build-91k96vz1
Running command git clone -q https://github.com/igraph/python-igraph /tmp/pip-req-build-91k96vz1
Running command git checkout -b develop --track origin/develop
Switched to a new branch 'develop'
Branch 'develop' set up to track remote branch 'develop' from 'origin'.
Resolved https://github.com/igraph/python-igraph to commit 7c81be69411bfb4cb0f2da7c84489ebf60cb2d0e
Running command git submodule update --init --recursive -q
Collecting texttable>=1.6.2
Using cached texttable-1.6.4-py2.py3-none-any.whl (10 kB)
Building wheels for collected packages: igraph
Building wheel for igraph (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /opt/conda/envs/scanpy_py3pt9_igraph0pt10/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-91k96vz1/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-91k96vz1/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-cckilrhh
cwd: /tmp/pip-req-build-91k96vz1/
Complete output (145 lines):
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/igraph
copying src/igraph/matching.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/sparse_matrix.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/summary.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/version.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/__init__.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/adjacency.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/automorphisms.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/basic.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/bipartite.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/clustering.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/community.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/configuration.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/cut.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/datatypes.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/formula.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/layout.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/seq.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/statistics.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/structural.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/utils.py -> build/lib.linux-x86_64-3.9/igraph
creating build/lib.linux-x86_64-3.9/igraph/app
copying src/igraph/app/__init__.py -> build/lib.linux-x86_64-3.9/igraph/app
copying src/igraph/app/shell.py -> build/lib.linux-x86_64-3.9/igraph/app
creating build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/metamagic.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/baseclasses.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/colors.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/shapes.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/text.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing
creating build/lib.linux-x86_64-3.9/igraph/remote
copying src/igraph/remote/__init__.py -> build/lib.linux-x86_64-3.9/igraph/remote
copying src/igraph/remote/gephi.py -> build/lib.linux-x86_64-3.9/igraph/remote
creating build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/__init__.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/adjacency.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/bipartite.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/files.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/images.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/libraries.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/objects.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/random.py -> build/lib.linux-x86_64-3.9/igraph/io
creating build/lib.linux-x86_64-3.9/igraph/operators
copying src/igraph/operators/__init__.py -> build/lib.linux-x86_64-3.9/igraph/operators
copying src/igraph/operators/functions.py -> build/lib.linux-x86_64-3.9/igraph/operators
copying src/igraph/operators/methods.py -> build/lib.linux-x86_64-3.9/igraph/operators
creating build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/base.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/coord.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/dendrogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/edge.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/histogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/matrix.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/palette.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/plot.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/polygon.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/text.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/vertex.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
creating build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/dendrogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/edge.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/histogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/matrix.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/palette.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/polygon.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/vertex.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
creating build/lib.linux-x86_64-3.9/igraph/drawing/plotly
copying src/igraph/drawing/plotly/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
copying src/igraph/drawing/plotly/edge.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
copying src/igraph/drawing/plotly/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
copying src/igraph/drawing/plotly/polygon.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
copying src/igraph/drawing/plotly/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
copying src/igraph/drawing/plotly/vertex.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
running build_ext
running build_c_core
We are going to build the C core of igraph.
Source folder: vendor/source/igraph
Build folder: vendor/build/igraph
Install folder: vendor/install/igraph
Configuring build...
-- Setting build type to 'Release' as none was specified.
-- Version number from Git: 0.10.0-dev+c9ed1c6b
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test COMPILER_SUPPORTS_NO_VARARGS_FLAG
-- Performing Test COMPILER_SUPPORTS_NO_VARARGS_FLAG - Success
-- Performing Test COMPILER_SUPPORTS_NO_UNKNOWN_WARNING_OPTION_FLAG
-- Performing Test COMPILER_SUPPORTS_NO_UNKNOWN_WARNING_OPTION_FLAG - Success
-- Performing Test COMPILER_HAS_DEPRECATED_ENUMVAL_ATTR
-- Performing Test COMPILER_HAS_DEPRECATED_ENUMVAL_ATTR - Success
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Could NOT find FLEX (missing: FLEX_EXECUTABLE)
-- Could NOT find BISON (missing: BISON_EXECUTABLE)
-- Looking for stpcpy
-- Looking for stpcpy - found
-- Looking for strcasecmp
-- Looking for strcasecmp - found
-- Looking for strdup
-- Looking for strdup - found
-- Looking for strndup
-- Looking for strndup - found
-- Looking for _stricmp
-- Looking for _stricmp - not found
CMake Error at src/CMakeLists.txt:24 (bison_target):
Unknown CMake command "bison_target".
-- Configuring incomplete, errors occurred!
See also "/tmp/pip-req-build-91k96vz1/vendor/build/igraph/CMakeFiles/CMakeOutput.log".
See also "/tmp/pip-req-build-91k96vz1/vendor/build/igraph/CMakeFiles/CMakeError.log".
Build failed for the C core of igraph.
----------------------------------------
ERROR: Failed building wheel for igraph
Running setup.py clean for igraph
Failed to build igraph
Installing collected packages: texttable, igraph
Attempting uninstall: texttable
Found existing installation: texttable 1.6.4
Uninstalling texttable-1.6.4:
Successfully uninstalled texttable-1.6.4
Attempting uninstall: igraph
Found existing installation: igraph 0.9.10
Uninstalling igraph-0.9.10:
Successfully uninstalled igraph-0.9.10
Running setup.py install for igraph ... error
ERROR: Command errored out with exit status 1:
command: /opt/conda/envs/scanpy_py3pt9_igraph0pt10/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-91k96vz1/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-91k96vz1/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-_596473v/install-record.txt --single-version-externally-managed --compile --install-headers /opt/conda/envs/scanpy_py3pt9_igraph0pt10/include/python3.9/igraph
cwd: /tmp/pip-req-build-91k96vz1/
Complete output (109 lines):
running install
/opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/igraph
copying src/igraph/matching.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/sparse_matrix.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/summary.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/version.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/__init__.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/adjacency.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/automorphisms.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/basic.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/bipartite.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/clustering.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/community.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/configuration.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/cut.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/datatypes.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/formula.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/layout.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/seq.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/statistics.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/structural.py -> build/lib.linux-x86_64-3.9/igraph
copying src/igraph/utils.py -> build/lib.linux-x86_64-3.9/igraph
creating build/lib.linux-x86_64-3.9/igraph/app
copying src/igraph/app/__init__.py -> build/lib.linux-x86_64-3.9/igraph/app
copying src/igraph/app/shell.py -> build/lib.linux-x86_64-3.9/igraph/app
creating build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/metamagic.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/baseclasses.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/colors.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/shapes.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/text.py -> build/lib.linux-x86_64-3.9/igraph/drawing
copying src/igraph/drawing/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing
creating build/lib.linux-x86_64-3.9/igraph/remote
copying src/igraph/remote/__init__.py -> build/lib.linux-x86_64-3.9/igraph/remote
copying src/igraph/remote/gephi.py -> build/lib.linux-x86_64-3.9/igraph/remote
creating build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/__init__.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/adjacency.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/bipartite.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/files.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/images.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/libraries.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/objects.py -> build/lib.linux-x86_64-3.9/igraph/io
copying src/igraph/io/random.py -> build/lib.linux-x86_64-3.9/igraph/io
creating build/lib.linux-x86_64-3.9/igraph/operators
copying src/igraph/operators/__init__.py -> build/lib.linux-x86_64-3.9/igraph/operators
copying src/igraph/operators/functions.py -> build/lib.linux-x86_64-3.9/igraph/operators
copying src/igraph/operators/methods.py -> build/lib.linux-x86_64-3.9/igraph/operators
creating build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/base.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/coord.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/dendrogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/edge.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/histogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/matrix.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/palette.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/plot.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/polygon.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/text.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
copying src/igraph/drawing/cairo/vertex.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
creating build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/dendrogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/edge.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/histogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/matrix.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/palette.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/polygon.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
copying src/igraph/drawing/matplotlib/vertex.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
creating build/lib.linux-x86_64-3.9/igraph/drawing/plotly
copying src/igraph/drawing/plotly/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
copying src/igraph/drawing/plotly/edge.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
copying src/igraph/drawing/plotly/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
copying src/igraph/drawing/plotly/polygon.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
copying src/igraph/drawing/plotly/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
copying src/igraph/drawing/plotly/vertex.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
running build_ext
running build_c_core
We are going to build the C core of igraph.
Source folder: vendor/source/igraph
Build folder: vendor/build/igraph
Install folder: vendor/install/igraph
Configuring build...
-- Version number from Git: 0.10.0-dev+c9ed1c6b
-- Could NOT find FLEX (missing: FLEX_EXECUTABLE)
-- Could NOT find BISON (missing: BISON_EXECUTABLE)
CMake Error at src/CMakeLists.txt:24 (bison_target):
Unknown CMake command "bison_target".
-- Configuring incomplete, errors occurred!
See also "/tmp/pip-req-build-91k96vz1/vendor/build/igraph/CMakeFiles/CMakeOutput.log".
See also "/tmp/pip-req-build-91k96vz1/vendor/build/igraph/CMakeFiles/CMakeError.log".
Build failed for the C core of igraph.
----------------------------------------
Rolling back uninstall of igraph
Moving to /opt/conda/envs/scanpy_py3pt9_igraph0pt10/bin/igraph
from /tmp/pip-uninstall-insy6qii/igraph
Moving to /opt/conda/envs/scanpy_py3pt9_igraph0pt10/include/python3.9/igraph/
from /opt/conda/envs/scanpy_py3pt9_igraph0pt10/include/python3.9/~graph
Moving to /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages/igraph-0.9.10.dist-info/
from /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages/~graph-0.9.10.dist-info
Moving to /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages/igraph/
from /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages/~graph
ERROR: Command errored out with exit status 1: /opt/conda/envs/scanpy_py3pt9_igraph0pt10/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-91k96vz1/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-91k96vz1/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-_596473v/install-record.txt --single-version-externally-managed --compile --install-headers /opt/conda/envs/scanpy_py3pt9_igraph0pt10/include/python3.9/igraph Check the logs for full command output.
flex and bison via conda install seems to be working...
These appear to require sudo install permissions, which i don't have.
Why is pip complaining that you are running as root if you do not have root access? See:
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Ok. So it seems i do have sudo permissions, I just needed to run
apt update && apt upgrade
and
apt install sudo
Apologies for the confusion.
I tried
You might need to run
pip install --upgrade --force-reinstall git+https://github.com/igraph/python-igraph@develop Collecting git+https://github.com/igraph/python-igraph@develop
in casepip
is trying to be smart and considers the new version being identical to the existing one.
but got the following
(scanpy_py3pt9_igraph0pt10) root@dcd19bf51c7b:/opt/git/python-igraph# pip install --upgrade --force-reinstall git+https://github.com/igraph/python-igraph@develop Collecting git+https://github.com/igraph
/python-igraph@develop
Collecting git+https://github.com/igraph/python-igraph@develop
Cloning https://github.com/igraph/python-igraph (to revision develop) to /tmp/pip-req-build-7kkjy0vx
Running command git clone -q https://github.com/igraph/python-igraph /tmp/pip-req-build-7kkjy0vx
Running command git checkout -b develop --track origin/develop
Switched to a new branch 'develop'
Branch 'develop' set up to track remote branch 'develop' from 'origin'.
Resolved https://github.com/igraph/python-igraph to commit 64a9df3f768ba62346e99d397ef9181a56317e27
Running command git submodule update --init --recursive -q
ERROR: Could not find a version that satisfies the requirement Collecting (from versions: none)
ERROR: No matching distribution found for Collecting
So instead ran,
(scanpy_py3pt9_igraph0pt10) root@dcd19bf51c7b:/opt/git/python-igraph# pip install --upgrade --force-reinstall git+https://github.com/igraph/python-igraph@developCollecting git+https://github.com/igraph/python-igraph@develop
Cloning https://github.com/igraph/python-igraph (to revision develop) to /tmp/pip-req-build-gpm2j_q9
Running command git clone -q https://github.com/igraph/python-igraph /tmp/pip-req-build-gpm2j_q9
Running command git checkout -b develop --track origin/develop
Switched to a new branch 'develop'
Branch 'develop' set up to track remote branch 'develop' from 'origin'.
Resolved https://github.com/igraph/python-igraph to commit 64a9df3f768ba62346e99d397ef9181a56317e27
Running command git submodule update --init --recursive -q
Collecting texttable>=1.6.2
Using cached texttable-1.6.4-py2.py3-none-any.whl (10 kB)
Building wheels for collected packages: igraph
Building wheel for igraph (setup.py) ... done
Created wheel for igraph: filename=igraph-0.9.10-cp39-cp39-linux_x86_64.whl size=2076664 sha256=1d5da72fd5c86a7dbe172eb04c8e10e22542948c39e50d1edaf5498e337ad4c7
Stored in directory: /tmp/pip-ephem-wheel-cache-dz_rhlmk/wheels/fd/36/d1/e31f410d36dbdef5d45386d89324b9fc71fc00d1dd7313c3b7
Successfully built igraph
Installing collected packages: texttable, igraph
Attempting uninstall: texttable
Found existing installation: texttable 1.6.4
Uninstalling texttable-1.6.4:
Successfully uninstalled texttable-1.6.4
Attempting uninstall: igraph
Found existing installation: igraph 0.9.10
Uninstalling igraph-0.9.10:
Successfully uninstalled igraph-0.9.10
Successfully installed igraph-0.9.10 texttable-1.6.4
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
(scanpy_py3pt9_igraph0pt10) root@dcd19bf51c7b:/opt/git/python-igraph#
However, it still seems my igraph and python-graph version numbers have not updated?
(scanpy_py3pt9_igraph0pt10) root@dcd19bf51c7b:/opt/git/python-igraph# conda list igraph
# packages in environment at /opt/conda/envs/scanpy_py3pt9_igraph0pt10:
#
# Name Version Build Channel
igraph 0.9.8 hf5496dd_0 conda-forge
python-igraph 0.9.10 py39hf6748bb_0 conda-forge
(scanpy_py3pt9_igraph0pt10) root@dcd19bf51c7b:/opt/git/python-igraph#
I think the compilation went fine; you will not see an updated version number in python-igraph
because you are on the develop
branch and no new version has been assigned to the develop
branch yet. The version number is taken from src/igraph/version.py
, which still shows 0.9.10
on the development branch.
fantastic. thanks for your help both.
i have hit run on my large graph script in the new environment.... will let you know if it runs!
Happy to help fix the 32 -> 64 bit conversion for this if needed, since I know that glue code decently and @ntamas might be too busy.
But let's see if it's necessary at all
64-bit version of igraph looks to still be running fine so far.
from my log...
my script has 2,355,675,670 edges (no longer a negative edge count)
edge weights were calculated and added to the graph just fine,
igraph is now proceeding to leiden cluster (with igraph g.community_leiden()
)....
2022-04-19 16:31:04 21197 INFO adata read finished
2022-04-19 16:31:04 21197 INFO adata full length, 18,543,700
2022-04-19 16:31:04 21197 INFO adjacency extracted
2022-04-19 16:31:52 21197 INFO adjacency sources, targets extracted
2022-04-19 16:32:59 21197 INFO weights calculated
2022-04-19 16:32:59 21197 INFO graph initiated
2022-04-19 16:33:00 21197 INFO graph vertices added
2022-04-19 16:33:00 21197 INFO vcount: 18543700
2022-04-19 17:18:52 21197 INFO graph edges added
2022-04-19 17:18:52 21197 INFO ecount: 2355675670
2022-04-19 17:21:24 21197 INFO graph weights added
2022-04-19 17:21:24 21197 INFO leiden clustering begun with 0.1 resolution
Sounds promising! Maybe my in64 PR wasn't totally broken.
Btw we'll have UMAP in igraph within a day or two, so if you want to embed the graph you can use igraph for both clustering and embedding.
I'll close this now, since this is effectively fixed on develop
(and the issue itself has turned into a support thread).
Well, at least the loading part is fixed. @mezwick if you find any issues with processing a graph this large, please file another issue with the details
Thankyou all for your help. I appreciate your time.
happy to say g.community_leiden()
runs on my 64bit igraph with 18.5 million cells :) @vtraag
How long did it take to complete?
Leiden run time: 5h 26 mins 23.798647 secs
On a cluster with 3892 GB mem, 16384 GB storage, 128 cores.
PID took ~9% of memory... so ~350 GB RAM needed
g.community_leiden()
just seems to run on a single core (100 % CPU in top)
Great, good to hear you managed to run everything @mezwick! Sounds like quite a reasonable run time. Sorry for my late reply, I only get to reading this now. Thanks all for helping out!
g.community_leiden() just seems to run on a single core (100 % CPU in top)
Yes, that's correct; igraph is currently not guaranteed to be thread-safe and the algorithms use single-core implementations.
Describe the bug I am trying to generate an edge weighted graph
g
from a large anndataadata
object (18.5 million cells/rows). I can do this for a subset of data, but when i try to do this for the full dataset i get the followingSystemError
To reproduce I am trying to generate the graph like so...
Apologies if this is in the wrong place, it may be a problem for the c igraph library.
I have also posted this problem on the scanpy github issues page, though think the issue may be more to do with loading weights into igraph so am bringing it here https://github.com/scverse/scanpy/issues/1053#issuecomment-1099288285 .
Version information conda = 4.10.3 python = 3.9.12 python-igraph = 0.9.10 igraph = 0.9.8