mmslindner
commented
5 years ago Having the exactly same issue on a fresh Ubuntu 19.04 with python 3.8 installed from source
Closed myourshaw closed 4 years ago
mmslindner
commented
5 years ago Having the exactly same issue on a fresh Ubuntu 19.04 with python 3.8 installed from source
jmarshall
commented
5 years ago What version of Cython are you using?
This appears to be cython/cython#2938 and it seems (PR cython/cython#2971) you will need at least cython 0.29.9.
mmslindner
commented
5 years ago Thanks for the suggestions, @jmarshall! I just installed Cython 0.29.14 via python3.8 -m pip install Cython, but python3.8 -m pip install pysam is still failing.
In a way, it looks like they were solving the same issue in PR cython/cython#2971, but it does not seem like having the fixed Cython verson installed helps with compiling pysam.
jmarshall
commented
5 years ago The usual reason for that is that your pysam build is using the bundled pre-built libchtslib.c etc (which were likely built with an earlier Cython) rather than re-running Cython.
It appears that @myourshaw's build is rebuilding these from libchtslib.pyx etc:
…
# pysam: cython is available - using cythonize if necessary
…
running build_ext
cythoning pysam/libchtslib.pyx to pysam/libchtslib.c
…Are you certain yours is as well?
mmslindner
commented
5 years ago Yes, confirming that
Running it with no Cython installed fails at during compilation of pysam/libchtslib.c (with similar errror).
myourshaw
commented
5 years ago I, too, have Cython 0.29.14 installed.
caaespin
commented
5 years ago I see the same error.
jmarshall
commented
5 years ago ¯\_(ツ)_/¯ Works for me with Python 3.8.0 (3.8.0-3~18.04 from bionic-updates) and Cython 0.29.14, building pysam from within a tarball with python3.8 setup.py build.
There were some false starts with this PyCode_New in the Python 3.8.0 alphas. So I suppose the next question is what exact version of Python 3.8 are you using? (What are the contents of /home/myourshaw/.local/include/python3.8/patchlevel.h between start/end constants?)
wanyuac
commented
5 years ago I had exactly the same issue when I was running the command python setup.py build with Python 3.8.0 and Cython 0.29.14.
mmslindner
commented
5 years ago I installed Python 3.8.0 from source a few weeks ago on my Ubuntu 19.04. Here is the version information from patchlevel.h:
/* Version parsed out into numeric values */
/*--start constants--*/
#define PY_MAJOR_VERSION 3
#define PY_MINOR_VERSION 8
#define PY_MICRO_VERSION 0
#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_FINAL
#define PY_RELEASE_SERIAL 0
/* Version as a string */
#define PY_VERSION "3.8.0"
/*--end constants--*/@jmarshall did you also try to pip install? I still need to check if the tarball works for me
myourshaw
commented
4 years ago /home/myourshaw/.local/include/python3.8/patchlevel.h contains
/* Python version identification scheme.
When the major or minor version changes, the VERSION variable in
configure.ac must also be changed.
There is also (independent) API version information in modsupport.h.
*/
/* Values for PY_RELEASE_LEVEL */
#define PY_RELEASE_LEVEL_ALPHA 0xA
#define PY_RELEASE_LEVEL_BETA 0xB
#define PY_RELEASE_LEVEL_GAMMA 0xC /* For release candidates */
#define PY_RELEASE_LEVEL_FINAL 0xF /* Serial should be 0 here */
/* Higher for patch releases */
/* Version parsed out into numeric values */
/*--start constants--*/
#define PY_MAJOR_VERSION 3
#define PY_MINOR_VERSION 8
#define PY_MICRO_VERSION 0
#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_FINAL
#define PY_RELEASE_SERIAL 0
/* Version as a string */
#define PY_VERSION "3.8.0"
/*--end constants--*/
/* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2.
Use this for numeric comparisons, e.g. #if PY_VERSION_HEX >= ... */
#define PY_VERSION_HEX ((PY_MAJOR_VERSION << 24) | \
(PY_MINOR_VERSION << 16) | \
(PY_MICRO_VERSION << 8) | \
(PY_RELEASE_LEVEL << 4) | \
(PY_RELEASE_SERIAL << 0))Still have the same issue with Python 3.8.1.
kemin711
commented
4 years ago Anyone solved the problem. 3.8.0 failed, and upgraded to python 3.8.1 still having the same problem. It is a pain in the neck. I am going to try to change the source code and see that will fix the problem.
kemin711
commented
4 years ago Looks the latest version of pysam fixed this problem. I used git clone of this project and this problem was gone. I looked at the problem location it was a simple #define directive problem. Someone should close this issue.
jmarshall
commented
4 years ago @kemin711: What exact #define change do you think has solved the problem?
kemin711
commented
4 years ago I did not fix the problem. The crashed line is a C directive #define the function with one extra argument
kemin711
commented
4 years ago I don't know the exact fix. But it should be some minor mismatch of function signature.
jmarshall
commented
4 years ago Yes. That's PyCode_New, which is between Cython and Python — it does not appear in the pysam code. The mystery here is how some people have a mismatch between the two. One other possibility is that they are using a Cython built for Python 3.7 with their newly-installed Python 3.8… I am not familiar enough with Cython to know whether this could cause a problem.
ExplodingCabbage
commented
4 years ago I have similarly observed that if I do pip3 install pysam it fails with this compilation error, but if I clone this repo and do cd pysam && pip3 install . it installs fine. I'm not clever enough to use this information to diagnose the problem, but surely it's a big clue!
ExplodingCabbage
commented
4 years ago Of further note: doing git checkout v0.15.3 && pip3 install pysam still works. So the issue here is not a change that's happened to pysam since the last release. It's some sort of difference between the behaviour (or code?) we get when installing from source.
However, all is not well for me even after "successfully" installing from source...
mark@fractal:~/pysam$ sudo pip3.8 install .
Processing /home/mark/pysam
Building wheels for collected packages: pysam
Building wheel for pysam (setup.py) ... done
Created wheel for pysam: filename=pysam-0.15.3-cp38-cp38-linux_x86_64.whl size=12767271 sha256=990d98be4dafb5e430ad291e92f3ba0bc2554aae08a454d10c605fb398e7450d
Stored in directory: /tmp/pip-ephem-wheel-cache-1vt1fhah/wheels/72/ea/0f/e3c1f98648e6a1596e7a9ab228c2c08f8f273aafbd12a33dd4
Successfully built pysam
Installing collected packages: pysam
Successfully installed pysam-0.15.3
mark@fractal:~/pysam$ python3.8
Python 3.8.0 (default, Oct 28 2019, 16:14:01)
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pysam
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/mark/pysam/pysam/__init__.py", line 5, in <module>
from pysam.libchtslib import *
ModuleNotFoundError: No module named 'pysam.libchtslib'I guess that installing this way doesn't actually work!
pm67nz
commented
4 years ago This is caused by some .c files in pysam 0.15.3 which were made by Cython 0.29.10. It can be fixed by regenerating them with Cython 0.29.12 or later. I successfully used Cython 0.29.14 together with this addition to setup.cfg:
[build_ext]
force=1Installing with pip3 install pysam still fails.
As @ExplodingCabbage says, the following works. I guess PyPI needs to be updated.
git clone https://github.com/pysam-developers/pysam.git
cd pysam
pip3 install .Inspired by @pm67nz's post, I tried simply rebuilding with a newer version of Cython (and with my fix from https://github.com/pysam-developers/pysam/pull/874) and uploading to PyPI, and everything worked.
You can install my version with:
pip3.8 install pysam-cabbageand you'll be good to go.
@AndreasHeger, if you can spare the time to simply:
git clean -xdf)setup.py sdistthen I think everything should be fixed for the official pysam package.
jmarshall
commented
4 years ago Indeed. The question for the maintainers is whether to make a point release just with regenerated Cython output files, or to do that as part of a larger release updating pysam to htslib/samtools/bcftools 1.10.x.
kemin711
commented
4 years ago I got it fixed. Kemin Zhou 858 771-3269
On Tuesday, January 14, 2020, 06:14:07 AM PST, John Marshall <notifications@github.com> wrote: Indeed. The question for the maintainers is whether to make a point release just with regenerated Cython output files, or to do that as part of a larger release updating pysam to htslib/samtools/bcftools 1.10.x.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.
jmarshall
commented
4 years ago Thinking about this further, it probably is worthwhile making a point release update just regenerating the Cython outputs.
There is a soversion bump between htslib 1.9 and 1.10 and some datatype size changes. So when pysam is updated to include the 1.10.x htslib/samtools/bcftools it would be beneficial to update pysam/* as well to use the htslib 1.10 facilities, including using some new functions. So pysam would then require htslib 1.10.
Hence it would be good to have a final htslib-1.9-using pysam release that works nicely with Python 3.8.
AndreasHeger
commented
4 years ago Makes sense. I have merged some outstanding pull requests and will build a release before moving on to htslib-1.10.
jmarshall
commented
4 years ago @AndreasHeger: Where pysam-0.15.3.tar.gz contained libc*.c files that were generated by a version of Cython that turned out to be a little bit too old, pysam-0.15.4.tar.gz (i.e., the source tarball listed at https://pypi.org/project/pysam/#files) does not contain any Cython-generated *.c files at all. Is this an intended change?
ExplodingCabbage
commented
4 years ago Regardless of the point above by @jmarshall (which Andreas may yet want to answer), I can confirm that I can now do pip3.8 install pysam and then import pysam successfully. I thus think this issue can be closed.
AndreasHeger
commented
4 years ago Thanks, I will close. @jmarshall ,thanks, it was indeed an oversight. I think I will wait if it actually causes a problem. The advice has been in the past to package cython generated .c files, but that might also generate issues as this ticket showed.
I get the following error when trying to install pysam with pip in Python 3.8.0 ( I also posted this on the user group):