Installation via miniconda & native python breaks at build step for cffi

abiddanda54gene commented 2 years ago

Hello PheWeb developers, I've been trying for a day to install this software for around an hour or two (on both linux with miniconda and OSX).

I have made the YAML file I have used for setting up the conda environment prior to installation (to avoid issues with blist as seen on OSX for fresh installation).

name: pheweb 
channels:
  - conda-forge
dependencies:
  - blist
  - gevent

To reproduce the error:

conda env create -f pheweb.yaml
conda activate pheweb
pip install pheweb --no-cache 2>&1 | tee pheweb_install_Linux.log

Any suggestions would be helpful here - as I've run this in as sanitized an environment as I possibly could imagine and cannot parse the potential error here or if there is a tagged stable version that you would recommend restricting to.

pheweb_install_Linux.log pheweb_install_OSX.log

abiddanda54gene commented 2 years ago

The conda environment definition above helps with the issue regarding blist in #179 to some degree.

pjvandehaar commented 2 years ago

Thanks, this is a good error-report. I appreciate that, because it makes these miserable installation issues at least a little more pleasant to work on. I'm busy today, so I'm sorry that this response isn't great, but I'll try to look into this more later.

On macOS:

Maybe adding wheel to your pheweb.yaml would help, so that it wouldn't have to compile blist?
If that doesn't fix it, maybe replace your third line with pip install blist and if that breaks, talk to the blist devs at https://github.com/DanielStutzbach/blist ? I see the issue https://github.com/DanielStutzbach/blist/issues/90 there, which looks like it might be the same problem. Maybe bump that issue, because the blist's author is still active.
I've been gradually replacing C libraries (like blist) with python code, because of these frustrating compilation errors, but I'd like to keep blist around if this can be resolved. And unfortunately I can't reproduce your error on my mac because I'm still on clang v11 on Big Sur and that's causing other errors. Are you on clang v12? What version of macOS?

On Linux: that error is puzzling. Does the file build/temp.linux-x86_64-3.9/pheweb.load.cffi._x.cpp actually not exist? Sorry, I can't think of any ways to figure out this problem. Maybe there's some command line option for pip that will change how it compiles cffi cpp? Or maybe there's a way to use strace or some file system logging to check why that file isn't being created?

abiddanda54gene commented 2 years ago

Is there a stable release version tagged @pjvandehaar? Or potentially to have a version installed on bioconda? That would certainly help with installation issues like this if possible. I'll fiddle around a little bit more, but let me know if there is anything else I could potentially help with.

pjvandehaar commented 2 years ago

I uploaded a new release, 1.3.16, without blist. Can you try it? Unfortunately I still have a problem with MACOSX_DEPLOYMENT_TARGET so I didn't upload wheels for this release.
The master branch should always work, but I'm more careful with the versioned releases. You can see them in the git history, because they're named like "1.3.16". I upload them to PyPI (which is why pip3 install pheweb works), but not bioconda.
Would bioconda work better than PyPI? I don't know anything about it. I can add that to my release script. Should I?
Would it be helpful if I tagged releases using git? I don't see the benefit to make up for the slight amount of work, but I'm probably just missing it.

abiddanda54gene commented 2 years ago

Hi @pjvandehaar,

The newest version (1.3.16) worked great via pip install pheweb using a python environment with 3.9.5 on Mac OSX :) So no complaints there.

However, within a conda environment on Linux there are still some problems with finding the cffi files (much more cryptic). Is this able to work on servers at UM quite well?

name: pheweb
channels:
  - conda-forge
dependencies:
  - python=3.9.5

and then:

conda env crate -f pheweb.yaml
conda activate pheweb
pip install pheweb --no-cache --verbose 2>&1 | tee pheweb_install_Linux.log

pheweb_install_Linux.log

I think that the eventual gcc command expects build/temp.linux-x86_64-3.9/pheweb.load.cffi._x.cpp to be around, but this is only ever copied in as copying pheweb/load/cffi/x.cpp -> build/lib.linux-x86_64-3.9/pheweb/load/cffi.

Regarding point 4 - I definitely think having the tagged releases on github would be very helpful (although one can also version lock on pip as well). It would be useful on github to have to download source-code that is version locked as a tarball and trying to perform a local pip install.

abiddanda54gene commented 2 years ago

Hi @pjvandehaar,

I've recently solved this on linux for an installation via conda that worked great! The yaml was:

name: pheweb
channels:
  - conda-forge
dependencies:
  - python
  - gxx_linux-64>9
  - pip
  - pip:
    - pheweb

and then:

conda env create -f pheweb.yaml
conda activate pheweb
pheweb -help

This worked well for me (and was system agnostic as well so could work as a general solution on linux as well). Feel free to include this in recommendations to others as well!

abiddanda54gene commented 2 years ago

Closing this issue as its not active any more and there is a solution. Thanks very much!

statgen / pheweb

Installation via miniconda & native python breaks at build step for cffi #187