DominikBuchner / BOLDigger2

An even better Python program to query .fasta files against the COI database of www.boldsystems.org
MIT License
11 stars 0 forks source link

Installing Boldigger2 #17

Closed partridc closed 2 months ago

partridc commented 2 months ago

Hi. Thanks so much for this great package. Our lab has been using the first version of boldigger with great success. We recently have been trying to update to boldigger2, and I'm coming across some issues when trying to install it. I'm sure it is something I'm doing wrong, but I'm not sure how to fix it. We just upgraded to python3.10 and when trying to install boldigger2, we are hitting an error when it tries to build the wheel. It states that it couldn't find the HDF5 runtime, the LZO 2 runtime,, or the BLOSC2_DIR. I've posted some of the output below that shows the error information. Do you have any suggestions as to what I could be doing wrong? Thanks so much.

Installing build dependencies ... done Getting requirements to build wheel ... error error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> [32 lines of output] /var/folders/rb/vvhcl8md0w7gqct9h41m92kc0000gn/T/blosc_list_compressors6u01n4jz.c:1:10: fatal error: 'blosc.h' file not found

include "blosc.h"

           ^~~~~~~~~
  1 error generated.
  ld: warning: ignoring file /usr/local/lib/libblosc2.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
  Undefined symbols for architecture arm64:
    "_blosc2_list_compressors", referenced from:
        _main in blosc2_list_compressorskeog9pdp.o
  ld: symbol(s) not found for architecture arm64
  clang: error: linker command failed with exit code 1 (use -v to see invocation)
  cpuinfo failed, assuming no CPU features: 'flags'
  * Using Python 3.10.0 (v3.10.0:b494f5935c, Oct  4 2021, 14:59:19) [Clang 12.0.5 (clang-1205.0.22.11)]
  * Found cython 3.0.10
  * USE_PKGCONFIG: True
  * Found HDF5 headers at ``/usr/local/include``, library at ``/usr/local/lib``.
  .. WARNING:: Could not find the HDF5 runtime.
     The HDF5 shared library was *not* found in the default library
     paths. In case of runtime problems, please remember to install it.
  * Found LZO 2 headers at ``/usr/local/include``, library at ``/usr/local/lib``.
  .. WARNING:: Could not find the LZO 2 runtime.
     The LZO 2 shared library was *not* found in the default library
     paths. In case of runtime problems, please remember to install it.
  * Skipping detection of LZO 1 since LZO 2 has already been found.
  * Could not find bzip2 headers and library; disabling support for it.
  * Could not find blosc headers and library; using internal sources.
  * pkg-config header dirs for blosc2: /Users/runner/work/python-blosc2/python-blosc2/_skbuild/macosx-10.9-x86_64-3.9/cmake-install/include
  * pkg-config library dirs for blosc2: /Users/runner/work/python-blosc2/python-blosc2/_skbuild/macosx-10.9-x86_64-3.9/cmake-install/lib
  * Run 'blosc2_find_directories_hook'
  .. ERROR:: Could not find a local blosc2 installation.
     You may need to explicitly state where your local blosc2 headers
     and library can be found by setting the ``BLOSC2_DIR`` environment
     variable or by using the ``--blosc2`` command-line option.
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> See above for output.

DominikBuchner commented 2 months ago

Hi Patrick, I'm sure we can fix this. Which os is this?

€: Okay, seems to be MacOS. Have you installed via brew? First thing I would try is to:

Additionally, check this issue which seems to be related to this problem but according to the author is fixed: https://github.com/PyTables/PyTables/issues/998

partridc @.***> schrieb am Di., 23. Juli 2024, 00:37:

Hi. Thanks so much for this great package. Our lab has been using the first version of boldigger with great success. We recently have been trying to update to boldigger2, and I'm coming across some issues when trying to install it. I'm sure it is something I'm doing wrong, but I'm not sure how to fix it. We just upgraded to python3.10 and when trying to install boldigger2, we are hitting an error when it tries to build the wheel. It states that it couldn't find the HDF5 runtime, the LZO 2 runtime,, or the BLOSC2_DIR. I've posted some of the output below that shows the error information. Do you have any suggestions as to what I could be doing wrong? Thanks so much.

Installing build dependencies ... done Getting requirements to build wheel ... error error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> [32 lines of output] /var/folders/rb/vvhcl8md0w7gqct9h41m92kc0000gn/T/blosc_list_compressors6u01n4jz.c:1:10: fatal error: 'blosc.h' file not found

include "blosc.h"

^~~~~ 1 error generated. ld: warning: ignoring file /usr/local/lib/libblosc2.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64 Undefined symbols for architecture arm64: "_blosc2_list_compressors", referenced from: _main in blosc2_list_compressorskeog9pdp.o ld: symbol(s) not found for architecture arm64 clang: error: linker command failed with exit code 1 (use -v to see invocation) cpuinfo failed, assuming no CPU features: 'flags'

  • Using Python 3.10.0 (v3.10.0:b494f5935c, Oct 4 2021, 14:59:19) [Clang 12.0.5 (clang-1205.0.22.11)]
  • Found cython 3.0.10
  • USE_PKGCONFIG: True
  • Found HDF5 headers at /usr/local/include, library at /usr/local/lib. .. WARNING:: Could not find the HDF5 runtime. The HDF5 shared library was not found in the default library paths. In case of runtime problems, please remember to install it.
  • Found LZO 2 headers at /usr/local/include, library at /usr/local/lib. .. WARNING:: Could not find the LZO 2 runtime. The LZO 2 shared library was not found in the default library paths. In case of runtime problems, please remember to install it.
  • Skipping detection of LZO 1 since LZO 2 has already been found.
  • Could not find bzip2 headers and library; disabling support for it.
  • Could not find blosc headers and library; using internal sources.
  • pkg-config header dirs for blosc2: /Users/runner/work/python-blosc2/python-blosc2/_skbuild/macosx-10.9-x86_64-3.9/cmake-install/include
  • pkg-config library dirs for blosc2: /Users/runner/work/python-blosc2/python-blosc2/_skbuild/macosx-10.9-x86_64-3.9/cmake-install/lib
  • Run 'blosc2_find_directories_hook' .. ERROR:: Could not find a local blosc2 installation. You may need to explicitly state where your local blosc2 headers and library can be found by setting the BLOSC2_DIR environment variable or by using the --blosc2 command-line option. [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> See above for output.

— Reply to this email directly, view it on GitHub https://github.com/DominikBuchner/BOLDigger2/issues/17, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJH6ILF6SXYGHQNB65B2NHDZNWCRLAVCNFSM6AAAAABLJF7PJOVHI2DSMVQWIX3LMV43ASLTON2WKOZSGQZDGOBYGAZDGOA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

partridc commented 2 months ago

Thanks so much for the quick reply! I was able to get both pytables and boldigger2 to install. However, when running the identify function for the fasta file, python is quitting before any of the download links begin. I should have enough memory on the computer, so I don't think that is the issue. I also don't see a spike in RAM when running the code that would indicate an problem. Do you happen to know what could be causing this?

The code and warning messages are below: MacBook-Pro:~ cpartridge$ boldigger2 identify ~/Dropbox/SpiderWeb.dada2.CO1.unwrapped.fasta -threshold 99 97 13:36:24: Default thresholds changed! 13:36:24: Species: 99, Genus: 97, Family: 90, Order: 85, Class: 50 13:36:24: Trying to log in. BOLD username:
BOLD password: 13:36:39: Login successful. 13:36:39: Starting to gather download links from species level database. Generating download links: 0%| | 0/3084 [00:00<?, ?it/s]Illegal instruction: 4 MacBook-Pro:~ cpartridge$ /usr/local/Cellar/python@3.10/3.10.14_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '

DominikBuchner commented 2 months ago

Can you try to pip install --upgrade tqdm to the newest version?

partridc @.***> schrieb am Di., 23. Juli 2024, 20:59:

Thanks so much for the quick reply! I was able to get both pytables and boldigger2 to install. However, when running the identify function for the fasta file, python is quitting before any of the download links begin. I should have enough memory on the computer, so I don't think that is the issue. I also don't see a spike in RAM when running the code that would indicate an problem. Do you happen to know what could be causing this?

The code and warning messages are below: MacBook-Pro:~ cpartridge$ boldigger2 identify /Dropbox/SpiderWeb.dada2.CO1.unwrapped.fasta -threshold 99 97 13:36:24: Default thresholds changed! 13:36:24: Species: 99, Genus: 97, Family: 90, Order: 85, Class: 50 13:36:24: Trying to log in. BOLD username: BOLD password: 13:36:39: Login successful. 13:36:39: Starting to gather download links from species level database. Generating download links: 0%| | 0/3084 [00:00<?, ?it/s]Illegal instruction: 4 MacBook-Pro: cpartridge$ @.***/3.10.14_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '

— Reply to this email directly, view it on GitHub https://github.com/DominikBuchner/BOLDigger2/issues/17#issuecomment-2246030500, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJH6ILABQ3LMZQV6GQTHU6LZN2RZFAVCNFSM6AAAAABLJF7PJOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBWGAZTANJQGA . You are receiving this because you commented.Message ID: @.***>

partridc commented 2 months ago

It states that the requirement is already satisfied.

Requirement already satisfied: tqdm in /usr/local/lib/python3.10/site-packages

DominikBuchner commented 2 months ago

Can you do pip freeze and am paste the output here?

partridc @.***> schrieb am Di., 23. Juli 2024, 21:17:

It states that the requirement is already satisfied.

Requirement already satisfied: tqdm in /usr/local/lib/python3.10/site-packages

— Reply to this email directly, view it on GitHub https://github.com/DominikBuchner/BOLDigger2/issues/17#issuecomment-2246104686, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJH6ILBVDBEVY3KHVGUUOOLZN2T4PAVCNFSM6AAAAABLJF7PJOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBWGEYDINRYGY . You are receiving this because you commented.Message ID: @.***>

DominikBuchner commented 2 months ago

And can you tell me how large your fasta is? I kind of doubt that it might be to much to put into RAM

Dominik Buchner @.***> schrieb am Di., 23. Juli 2024, 21:58:

Can you do pip freeze and am paste the output here?

partridc @.***> schrieb am Di., 23. Juli 2024, 21:17:

It states that the requirement is already satisfied.

Requirement already satisfied: tqdm in /usr/local/lib/python3.10/site-packages

— Reply to this email directly, view it on GitHub https://github.com/DominikBuchner/BOLDigger2/issues/17#issuecomment-2246104686, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJH6ILBVDBEVY3KHVGUUOOLZN2T4PAVCNFSM6AAAAABLJF7PJOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBWGEYDINRYGY . You are receiving this because you commented.Message ID: @.***>

partridc commented 2 months ago

The fasta file is 681 kb (3,048 sequences). The computer I'm trying to run it on 24Gb RAM. The output from pip freeze is below. Again, thanks for your help!

appdirs==1.4.4 beautifulsoup4==4.12.3 biopython==1.84 blosc2==2.7.0 boldigger2==1.2.4 bs4==0.0.2 certifi==2024.7.4 charset-normalizer==3.3.2 cssselect==1.2.0 et-xmlfile==1.1.0 fake-useragent==1.5.1 html5lib==1.1 idna==3.7 importlib_metadata==8.1.0 joblib==1.4.2 lxml==5.2.2 lxml_html_clean==0.1.1 more-itertools==10.3.0 msgpack==1.0.8 ndindex==1.8 numexpr==2.10.1 numpy==1.26.4 openpyxl==3.1.5 packaging==24.1 pandas==2.2.2 parse==1.20.2 py-cpuinfo==9.0.0 pyarrow==17.0.0 pyee==11.1.0 pyppeteer==2.0.0 pyquery==2.0.0 python-dateutil==2.9.0.post0 pytz==2024.1 requests==2.32.3 requests-html==0.10.0 six==1.16.0 soupsieve==2.5 tables==3.9.2 tqdm==4.66.4 tqdm_joblib==0.0.4 typing_extensions==4.12.2 tzdata==2024.1 urllib3==1.26.19 w3lib==2.2.1 webencodings==0.5.1 websockets==10.4 zipp==3.19.2

DominikBuchner commented 2 months ago

Can you also upload the fasta so I can try to reproduce the issue? Just want to make sure that it's not the file in the end.

partridc commented 2 months ago

SpiderWeb.dada2.CO1.unwrapped.txt

Of course, it is attached. I uploaded as a txt file because it wasn't allowing me to upload it as a .fasta file.

DominikBuchner commented 2 months ago

Okay, it seems to be a problem related to your system, I cannot reproduce it. Any chance you could upgrade to a newer python version and retry? e.g. 3.12?

DominikBuchner commented 2 months ago

Hopefully fixed with 2.0.0.