labgem / PPanGGOLiN

Build a partitioned pangenome graph from microbial genomes
https://ppanggolin.readthedocs.io
Other
236 stars 27 forks source link

PPanGGOLiN version 2.1.1 - Failed building new version #275

Open seajane opened 3 weeks ago

seajane commented 3 weeks ago

I tried installing manually from the GitHub repo as I am unable to use conda from my work computers. I used a virtual environment (venv) instead. I get the following error: note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for PPanGGOLiN Failed to build PPanGGOLiN ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (PPanGGOLiN)

It looks like the python was copied in fine: ...copying ppanggolin/workflow/workflow.py -> build/lib.macosx-14.0-arm64-cpython-312/ppanggolin/workflow running build_ext building 'nem_stats' extension ...

This is where I start to see clang warnings in `clang -fno-strict-overflow -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -Ippanggolin/nem/NEM/ -I/Users/hkbouzek/Documents/GitHub/PPanGGOLiN/ppgg/include -I/opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/include/python3.12 -c ppanggolin/nem/NEM/lib_io.c -o build/temp.macosx-14.0-arm64-cpython-312/ppanggolin/nem/NEM/lib_io.o -fcommon -Wno-int-conversion ppanggolin/nem/NEM/lib_io.c:107:9: warning: 'gets' is deprecated: This function is provided for compatibility reasons only. Due to security concerns inherent in the design of gets(3), it is highly recommended that you use fgets(3) instead. [-Wdeprecated-declarations] gets( NameF ) ; ^ /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/stdio.h:162:1: note: 'gets' has been explicitly marked deprecated here __deprecated_msg("This function is provided for compatibility reasons only. Due to security concerns inherent in the design of gets(3), it is highly recommended that you use fgets(3) instead.") ^ /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/sys/cdefs.h:218:48: note: expanded from macro '__deprecated_msg'

define deprecated_msg(_msg) attribute((deprecated__(_msg)))

                                                        ^
  ppanggolin/nem/NEM/lib_io.c:158:9: warning: 'gets' is deprecated: This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of gets(3), it is highly recommended that you use fgets(3) instead. [-Wdeprecated-declarations]
          gets( NameF ) ;
          ^
  /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/stdio.h:162:1: note: 'gets' has been explicitly marked deprecated here
  __deprecated_msg("This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of gets(3), it is highly recommended that you use fgets(3) instead.")
  ^
  /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/sys/cdefs.h:218:48: note: expanded from macro '__deprecated_msg'
          #define __deprecated_msg(_msg) __attribute__((__deprecated__(_msg)))
                                                        ^
  ppanggolin/nem/NEM/lib_io.c:235:9: warning: 'gets' is deprecated: This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of gets(3), it is highly recommended that you use fgets(3) instead. [-Wdeprecated-declarations]
          gets( stringread ) ;
          ^
  /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/stdio.h:162:1: note: 'gets' has been explicitly marked deprecated here
  __deprecated_msg("This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of gets(3), it is highly recommended that you use fgets(3) instead.")
  ^
  /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/sys/cdefs.h:218:48: note: expanded from macro '__deprecated_msg'
          #define __deprecated_msg(_msg) __attribute__((__deprecated__(_msg)))
                                                        ^
  ppanggolin/nem/NEM/lib_io.c:283:9: warning: 'gets' is deprecated: This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of gets(3), it is highly recommended that you use fgets(3) instead. [-Wdeprecated-declarations]
          gets( stringread ) ;
          ^
  /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/stdio.h:162:1: note: 'gets' has been explicitly marked deprecated here
  __deprecated_msg("This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of gets(3), it is highly recommended that you use fgets(3) instead.")
  ^
  /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/sys/cdefs.h:218:48: note: expanded from macro '__deprecated_msg'
          #define __deprecated_msg(_msg) __attribute__((__deprecated__(_msg)))
                                                        ^
  4 warnings generated.
  clang -fno-strict-overflow -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -Ippanggolin/nem/NEM/ -I/Users/hkbouzek/Documents/GitHub/PPanGGOLiN/ppgg/include -I/opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/include/python3.12 -c ppanggolin/nem/NEM/nem_alg.c -o build/temp.macosx-14.0-arm64-cpython-312/ppanggolin/nem/NEM/nem_alg.o -fcommon -Wno-int-conversion
  ppanggolin/nem/NEM/nem_alg.c:1511:20: warning: incompatible pointer types passing 'FILE **' (aka 'struct __sFILE **') to parameter of type 'FILE *' (aka 'struct __sFILE *'); dereference with * [-Wincompatible-pointer-types]
              setbuf(FlogP, NULL);
                     ^~~~~
                     *
  /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/stdio.h:175:31: note: passing argument to parameter here
  void     setbuf(FILE * __restrict, char * __restrict);
                                   ^
  1 warning generated.
  clang -fno-strict-overflow -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -Ippanggolin/nem/NEM/ -I/Users/hkbouzek/Documents/GitHub/PPanGGOLiN/ppgg/include -I/opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/include/python3.12 -c ppanggolin/nem/NEM/nem_exe.c -o build/temp.macosx-14.0-arm64-cpython-312/ppanggolin/nem/NEM/nem_exe.o -fcommon -Wno-int-conversion
  ppanggolin/nem/NEM/nem_exe.c:516:38: warning: incompatible pointer types passing 'float *' to parameter of type 'float **'; take the address with & [-Wincompatible-pointer-types]
                                       ClassifM ) ) != STS_OK )
                                       ^~~~~~~~
                                       &
  ppanggolin/nem/NEM/nem_exe.c:138:28: note: passing argument to parameter 'MatP' here
               float       **MatP         /* O and allocated (size : Nl * Nc) */
                             ^
  ppanggolin/nem/NEM/nem_exe.c:542:35: warning: incompatible pointer types passing 'float *' to parameter of type 'float **'; take the address with & [-Wincompatible-pointer-types]
                                    ClassifM ) ) != STS_OK )
                                    ^~~~~~~~
                                    &
  ppanggolin/nem/NEM/nem_exe.c:147:28: note: passing argument to parameter 'ClassifMP' here
               float       **ClassifMP    /* O and allocated (Npt*Nk) */
                             ^
  ppanggolin/nem/NEM/nem_exe.c:373:23: warning: comparison of integers of different signs: 'AlgoET' and 'int' [-Wsign-compare]
      if ( NemPara.Algo == -1 )
           ~~~~~~~~~~~~ ^  ~~
  ppanggolin/nem/NEM/nem_exe.c:389:25: warning: comparison of integers of different signs: 'CvemET' and 'int' [-Wsign-compare]
      if ( NemPara.CvTest == -1 ) {
           ~~~~~~~~~~~~~~ ^  ~~
  ppanggolin/nem/NEM/nem_exe.c:402:25: warning: comparison of integers of different signs: 'FormET' and 'int' [-Wsign-compare]
      if ( NemPara.Format == -1 )
           ~~~~~~~~~~~~~~ ^  ~~
  ppanggolin/nem/NEM/nem_exe.c:421:37: warning: comparison of integers of different signs: 'FamilyET' and 'int' [-Wsign-compare]
      if ( StatModel.Spec.ClassFamily == -1 )
           ~~~~~~~~~~~~~~~~~~~~~~~~~~ ^  ~~
  ppanggolin/nem/NEM/nem_exe.c:428:37: warning: comparison of integers of different signs: 'ProporET' and 'int' [-Wsign-compare]
      if ( StatModel.Spec.ClassPropor == -1 )
           ~~~~~~~~~~~~~~~~~~~~~~~~~~ ^  ~~
  ppanggolin/nem/NEM/nem_exe.c:435:37: warning: comparison of integers of different signs: 'DisperET' and 'int' [-Wsign-compare]
      if ( StatModel.Spec.ClassDisper == -1 )
           ~~~~~~~~~~~~~~~~~~~~~~~~~~ ^  ~~
  8 warnings generated.
  clang -fno-strict-overflow -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -Ippanggolin/nem/NEM/ -I/Users/hkbouzek/Documents/GitHub/PPanGGOLiN/ppgg/include -I/opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/include/python3.12 -c ppanggolin/nem/NEM/nem_hlp.c -o build/temp.macosx-14.0-arm64-cpython-312/ppanggolin/nem/NEM/nem_hlp.o -fcommon -Wno-int-conversion
  clang -fno-strict-overflow -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -Ippanggolin/nem/NEM/ -I/Users/hkbouzek/Documents/GitHub/PPanGGOLiN/ppgg/include -I/opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/include/python3.12 -c ppanggolin/nem/NEM/nem_mod.c -o build/temp.macosx-14.0-arm64-cpython-312/ppanggolin/nem/NEM/nem_mod.o -fcommon -Wno-int-conversion
  clang -fno-strict-overflow -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -Ippanggolin/nem/NEM/ -I/Users/hkbouzek/Documents/GitHub/PPanGGOLiN/ppgg/include -I/opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/include/python3.12 -c ppanggolin/nem/NEM/nem_nei.c -o build/temp.macosx-14.0-arm64-cpython-312/ppanggolin/nem/NEM/nem_nei.o -fcommon -Wno-int-conversion
  clang -fno-strict-overflow -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -Ippanggolin/nem/NEM/ -I/Users/hkbouzek/Documents/GitHub/PPanGGOLiN/ppgg/include -I/opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/include/python3.12 -c ppanggolin/nem/NEM/nem_rnd.c -o build/temp.macosx-14.0-arm64-cpython-312/ppanggolin/nem/NEM/nem_rnd.o -fcommon -Wno-int-conversion
  clang -fno-strict-overflow -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -Ippanggolin/nem/NEM/ -I/Users/hkbouzek/Documents/GitHub/PPanGGOLiN/ppgg/include -I/opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/include/python3.12 -c ppanggolin/nem/NEM/nem_stats.c -o build/temp.macosx-14.0-arm64-cpython-312/ppanggolin/nem/NEM/nem_stats.o -fcommon -Wno-int-conversion
  clang -bundle -undefined dynamic_lookup -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -lomp build/temp.macosx-14.0-arm64-cpython-312/ppanggolin/nem/NEM/genmemo.o build/temp.macosx-14.0-arm64-cpython-312/ppanggolin/nem/NEM/lib_io.o build/temp.macosx-14.0-arm64-cpython-312/ppanggolin/nem/NEM/nem_alg.o build/temp.macosx-14.0-arm64-cpython-312/ppanggolin/nem/NEM/nem_exe.o build/temp.macosx-14.0-arm64-cpython-312/ppanggolin/nem/NEM/nem_hlp.o build/temp.macosx-14.0-arm64-cpython-312/ppanggolin/nem/NEM/nem_mod.o build/temp.macosx-14.0-arm64-cpython-312/ppanggolin/nem/NEM/nem_nei.o build/temp.macosx-14.0-arm64-cpython-312/ppanggolin/nem/NEM/nem_rnd.o build/temp.macosx-14.0-arm64-cpython-312/ppanggolin/nem/NEM/nem_stats.o -o build/lib.macosx-14.0-arm64-cpython-312/nem_stats.cpython-312-darwin.so
  ld: library 'omp' not found
  clang: error: linker command failed with exit code 1 (use -v to see invocation)
  error: command '/usr/bin/clang' failed with exit code 1
  [end of output]`
JeanMainguy commented 3 weeks ago

Hi,

It looks like this issue might be related to macOS. Since we don’t have access to a Mac, I tried installing PPanGGOLIN on a macOS 14 runner using GitHub Actions. I had to remove argorn and infernal from the dependencies, but the installation seems to have worked fine.

Could you share how you installed PPanGGOLIN manually? That might help us figure out what's going on.

Thanks!

seajane commented 3 weeks ago

Sure!

  1. I created a virtual environment python3 -m venv ppgg
  2. Activated the environment source ppgg/bin/activate
  3. Brew installed dependencies: brew update brew upgrade brew install mmseqs2 brew install mafft brew install infernal brew install aragorn
  4. Clone repo git clone https://github.com/labgem/PPanGGOLiN.git
  5. Change directory cd PPanGGOLiN
  6. Install with pip3 pip3 install .
JeanMainguy commented 2 weeks ago

Thanks for you installation instruction.

From the logs you shared, it looks like the problem might be related to the compilation of the C code in ppanggolin. The message library 'omp' not found clang: error: seems to indicate that the OpenMP library is missing.

It might be helpful to install OpenMP, as suggested there: OpenMP Simple C Example with CMake:

brew install libomp

and then export the OpenMP_ROOT variable :

export OpenMP_ROOT=$(brew --prefix)/opt/libomp
seajane commented 2 weeks ago

Thank you for your comments. Unfortunately it didn't work. I did get different messages in the error that may be of use however: check.warn(importable) /private/var/folders/cv/q414p25d6yv_4sfgsdft3b400000gq/T/pip-build-env-p4oyqhwq/overlay/lib/python3.12/site-packages/setuptools/command/build_py.py:218: _Warning: Package 'ppanggolin.formats' is absent from thepackages` configuration.


          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'ppanggolin.formats' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.

          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'ppanggolin.formats' is explicitly added
          to the `packages` configuration field.

          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).

          You can read more about "package discovery" on setuptools documentation page:

          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html

          If you don't want 'ppanggolin.formats' to be distributed and are
          already explicitly excluding 'ppanggolin.formats' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.

          You can read more about "package data files" on setuptools documentation page:

          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html

          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************`

I get the same sort of error for 'ppanggolin.formats', 'ppanggolin.graph', 'ppanggolin.info', 'ppanggolin.meta', 'ppanggolin.mod', 'ppanggolin.nem', 'ppanggolin.nem.NEM', 'ppanggolin.projection', 'ppanggolin.utility', 'ppanggolin.workflow'

seajane commented 2 weeks ago

I am no expert in this at all, but feel the problem may be related to OpenMP changes on macOS https://mac.r-project.org/openmp/

JeanMainguy commented 2 weeks ago

Hi, It looks like the issue might be related to Python 3.12. You could try installing ppanggolin in a Python 3.10 environment instead of 3.12 with :

python3.10 -m venv ppgg

I gave it a shot on a macOS 14 GitHub runner using your method and managed to install it with Python 3.10 (here are the log), but not with Python 3.12 (though I ran into a different error than you did).

Hope that helps!