Open ulupo opened 4 years ago
There seem to be another issue with nightly wheels: I get a segfault when running tests in continuumio/miniconda3
container,
# needed to allow running GDB inside docker https://stackoverflow.com/a/46676907/1791279
$ docker run --rm -it --cap-add=SYS_PTRACE --security-opt seccomp=unconfined continuumio/miniconda3 /bin/bash
apt update
apt install gdb
conda create -n gtda-env python=3.7
conda activate gtda-env
pip install giotto-tda-nightly
pip install hypothesis pytest
For instance,
$ python -c "import gtda.externals.python"
Illegal instruction (core dumped)
and with a full backtrace,
$ gdb run --args python -c "import gtda.externals.python"
[...]
Reading symbols from python...done.
(gdb) r
Starting program: /opt/conda/envs/gtda-env/bin/python -c import\ gtda.externals.python
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff5178700 (LWP 596)]
[New Thread 0x7ffff4977700 (LWP 597)]
...
Thread 1 "python" received signal SIGILL, Illegal instruction.
0x00007fffad9188b0 in ?? () from /opt/conda/envs/gtda-env/lib/python3.7/site-packages/giotto_ripser.cpython-37m-x86_64-linux-gnu.so
(gdb) bt
#0 0x00007fffad9188b0 in ?? ()
from /opt/conda/envs/gtda-env/lib/python3.7/site-packages/giotto_ripser.cpython-37m-x86_64-linux-gnu.so
#1 0x00007fffad92bac0 in ?? ()
from /opt/conda/envs/gtda-env/lib/python3.7/site-packages/giotto_ripser.cpython-37m-x86_64-linux-gnu.so
#2 0x00007fffad92ce7e in PyInit_giotto_ripser ()
from /opt/conda/envs/gtda-env/lib/python3.7/site-packages/giotto_ripser.cpython-37m-x86_64-linux-gnu.so
#3 0x0000555555773695 in _PyImport_LoadDynamicModuleWithSpec (spec=0x7fffae9a7e10, fp=<optimized out>)
at /tmp/build/80754af9/python_1578510683607/work/Python/importdl.c:159
#4 0x00005555557738a9 in _imp_create_dynamic_impl.isra.15 (file=0x0, spec=0x7fffae9a7e10)
at /tmp/build/80754af9/python_1578510683607/work/Python/import.c:2170
#5 _imp_create_dynamic (module=<optimized out>, args=<optimized out>, nargs=<optimized out>)
at /tmp/build/80754af9/python_1578510683607/work/Python/clinic/import.c.h:289
#6 0x000055555568cab2 in _PyMethodDef_RawFastCallDict (method=0x55555586f800 <imp_methods+320>, self=0x7ffff7a32cb0,
args=0x7ffff78c31a8, nargs=1, kwargs=<optimized out>) at /tmp/build/80754af9/python_1578510683607/work/Objects/call.c:530
#7 0x000055555568cbd1 in _PyCFunction_FastCallDict (func=0x7ffff7a4b7d0, args=<optimized out>, nargs=<optimized out>,
kwargs=<optimized out>) at /tmp/build/80754af9/python_1578510683607/work/Objects/call.c:586
#8 0x000055555571a839 in do_call_core (kwdict=0x7fffb8e58c80, callargs=0x7ffff78c3190, func=0x7ffff7a4b7d0)
at /tmp/build/80754af9/python_1578510683607/work/Python/ceval.c:4641
#9 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
at /tmp/build/80754af9/python_1578510683607/work/Python/ceval.c:3191
#10 0x000055555566e6f9 in _PyEval_EvalCodeWithName (_co=0x7ffff7a3a150, globals=<optimized out>, locals=<optimized out>,
args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x7ffff7850ea0, kwcount=0, kwstep=1, defs=0x0, defcount=0,
kwdefs=0x0, closure=0x0, name=0x7ffff7a38300, qualname=0x7ffff7a38300)
at /tmp/build/80754af9/python_1578510683607/work/Python/ceval.c:3930
which is a Ubuntu based system,
$ uname -a
Linux d614d763e5fb 5.0.0-38-generic #41-Ubuntu SMP Tue Dec 3 00:27:35 UTC 2019 x86_64 GNU/Linux
Not much ideas what could be wrong so far.
@rth, thanks for the report!
For the error with .tar.gz I can confirm I get the same error on Linux with,
pip install https://files.pythonhosted.org/packages/be/b3/cbb2d3bf866f8f1cf820dbe32bfc69cf5bdb0265c26e875670f2bd20be3f/giotto-tda-nightly-20200214.9.tar.gz
which yields,
fatal: not a git repository (or any of the parent directories): .git [185/1905]
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-req-build-3_74n4r_/setup.py", line 178, in <module>
cmdclass=dict(build_ext=CMakeBuild))
File "/opt/conda/envs/gtda-env/lib/python3.7/site-packages/setuptools/__init__.py", line 144, in setup
return distutils.core.setup(**attrs)
File "/opt/conda/envs/gtda-env/lib/python3.7/distutils/core.py", line 148, in setup
dist.run_commands()
File "/opt/conda/envs/gtda-env/lib/python3.7/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/opt/conda/envs/gtda-env/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/opt/conda/envs/gtda-env/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 223, in run
self.run_command('build')
File "/opt/conda/envs/gtda-env/lib/python3.7/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/opt/conda/envs/gtda-env/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/opt/conda/envs/gtda-env/lib/python3.7/distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/opt/conda/envs/gtda-env/lib/python3.7/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/opt/conda/envs/gtda-env/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/tmp/pip-req-build-3_74n4r_/setup.py", line 94, in run
self.install_dependencies()
File "/tmp/pip-req-build-3_74n4r_/setup.py", line 129, in install_dependencies
'--init', '--recursive'])
File "/opt/conda/envs/gtda-env/lib/python3.7/subprocess.py", line 363, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'submodule', 'update', '--init', '--recursive']' returned non-zero exit status 128.
The issue is in this line https://github.com/giotto-ai/giotto-tda/blob/6b0411bb281303b96fc38711fd73d368efeea829/setup.py#L119 which assumes that gtda is a git repository, which is not the case in a .tar.gz
.
@rth thanks for diagnosing this and for opening #314! It seems that in the current approach we should simply not publish the tarballs at all as they are guaranteed to fail.
Hello,
So is there a workaround for this issue at the moment? For me, install via pip(3) fails with the same error regarding git, and when I try to install it directly as a wheel by saying
pip3 install https://github.com/giotto-ai/giotto-tda/releases/download/v0.1.4/giotto_tda-0.1.4-cp36-cp36m-manylinux2010_x86_64.whl
I get an error saying that xxxxx is not a supported wheel on this platform. My Python version is 3.6.8 and my pip3 version is 9.0.1
Thanks for the report @madarax64! The message seems to suggest you are running on a potentially exotic (for giotto-tda
at least) Linux version. Could you tell us what that is? Notice that the meaning of manylinux2010
is explained in https://www.python.org/dev/peps/pep-0571/.
Finally, what you mean by "fails with the same error regarding git"? Which error exactly?
Hey @ulupo Weird - I'm on Ubuntu 18.04.
This one:
subprocess.CalledProcessError: Command '['git', 'submodule', 'update', '--init', '--recursive']' returned non-zero exit status 128.
Ubuntu 18.04 should definitely be supported by the manylinux wheels. So just to be clear, what does a run of
pip install giotto-tda
on a fresh environment, not in the repository root, yield?
So I upgraded my pip3, and then simply did:
pip3 install giotto-tda
And that worked off the bat. But when I actually run code, I get an error reading:
Illegal Instruction (core dumped)
Running with gdb like above gives the following error:
Thread 1 "python3" received signal SIGILL, Illegal instruction. 0x00007fffd3b24870 in ?? () from /usr/local/lib/python3.6/dist-packages/giotto_ripser.cpython-36m-x86_64-linux-gnu.so
But when I actually run code, I get an error reading: Illegal Instruction (core dumped)
Could you give us the code you ran, to reproduce?
Running with gdb like above gives the following error: Thread 1 "python3" received signal SIGILL, Illegal instruction. 0x00007fffd3b24870 in ?? () from /usr/local/lib/python3.6/dist-packages/giotto_ripser.cpython-36m-x86_64-linux-gnu.so
So this is installing from a clone of the repository? By wich I mean
git clone https://github.com/giotto-ai/giotto-tda
pip install -e .
The error looks like an esoteric C++-related error, which @MonkeyBreaker may be in a better position to diagnose.
I'm not working in Anaconda or a Virtualenv, so this is directly on my machine. Running pip3 install giotto-tda
in say, an empty folder works perfectly without any errors, after I upgraded my pip3 to version 20.
To reproduce the error, you can run the following:
python3 -c "import gtda.homology as hl"
No, I never installed from the clone of the repository. This came from the version I got from pip after I upgraded pip.
Hi @madarax64,
Sorry that you encounter Illegal Instruction (core dumped)
, this was fixed yesterday in PR #360.
This was caused by illegal instructions running on the CPU, in this case some AVX instructions that your CPU does not support.
I don't know when will the wheels integrate this change, but I think that giotto-tda-nightly
should now integrate them, or am I wrong @ulupo ?
@MonkeyBreaker @madarax64, I don't think a nightly with these changes has been pushed yet. But the new stable version 0.2.0 which will include these fixes will be out today or Monday.
Hi @ulupo @MonkeyBreaker Thanks, I look forward to it. I really appreciate it.
Description
I believe that all installs from tarballs on PyPI currently fail. The reason seems to be that we do not package the
CMakeLists.txt
file needed byCMake
.Steps/Code to Reproduce
giotto-tda-nightly
, wheels only support macOS 10.14 or above following updates in the Azure VM images https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops, so one can test this on macOS 10.13.pip install giotto-tda-nightly
to download the package from PyPI.pip
will skip the wheels as no supported wheels will be found, and will attempt to install from tarball.Expected Results
Package is installed as if it were being installed from source (
pip install .
from the root of a cloned repo).Actual Results
On my macOS 10.13 machine and running
pip install giotto-tda-nightly
: https://gist.github.com/ulupo/8bdf6f97a55cf86688a62e9728348409