Closed zhaiting closed 2 months ago
Cloufield
commented
3 months ago Hi, sorry for the inconvenience. As you said, the installation on a device with Apple silicon is quite problematic. (I have received multiple feedbacks on this error) Today I just got an M1 chip device and will test/fix this soon. I will let you know when it is finished.
Cloufield
commented
3 months ago Hi,
I tested with a M1 chip device but could not replicate the error. The installation works well without any error.
Probably there was something wrong when you installed the liftover package?
Could you please check if the files in /opt/anaconda3/envs/gwas/lib/python3.9/site-packages/liftover/ are the same as:
(gwaslab) yunye@Yunyes-Air liftover % ls -lh
total 384
-rw-r--r-- 1 yunye staff 176B Mar 30 13:51 __init__.py
drwxr-xr-x 5 yunye staff 160B Mar 30 13:51 __pycache__
-rwxr-xr-x 1 yunye staff 179K Mar 30 13:51 chain_file.cpython-39-darwin.so
-rw-r--r-- 1 yunye staff 347B Mar 30 13:51 download_file.py
-rw-r--r-- 1 yunye staff 1.5K Mar 30 13:51 lifter.py
zhaiting
commented
3 months ago Hello, thanks so much for looking into this! I got same files in the directory, please see my output below, but the import error still persists... (base) ting@This-MBP liftover % ls -lh total 368 -rw-r--r-- 1 ting staff 176B Mar 26 12:04 init.py drwxr-xr-x 5 ting staff 160B Mar 26 12:04 pycache -rwxr-xr-x 1 ting staff 172K Mar 26 12:04 chain_file.cpython-39-darwin.so -rw-r--r-- 1 ting staff 347B Mar 26 12:04 download_file.py -rw-r--r-- 1 ting staff 1.5K Mar 26 12:04 lifter.py
Cloufield
commented
3 months ago Hi,
It seems that your chain_file.cpython-39-darwin.so file is unsigned. Probably the way you installed it omitted some flags.
You can run codesign -dv chain_file.cpython-39-darwin.so to check it:
Executable=/opt/anaconda3/envs/gwaslab/lib/python3.9/site-packages/liftover/chain_file.cpython-39-darwin.so
Identifier=chain_file.cpython-39-darwin.so
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=1560 flags=0x20002(adhoc,linker-signed) hashes=45+0 location=embedded
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements=noneSee if this part is the same.
CodeDirectory v=20400 size=1560 flags=0x20002(adhoc,linker-signed) hashes=45+0 location=embedded
Signature=adhocIf not signed, you can ad-hoc sign it using codesign -s - chain_file.cpython-39-darwin.so.
Please let me know if this works or not. Thanks!
zhaiting
commented
3 months ago Hi, this indeed solved the error for liftover (chain_file.cpython-39-darwin.so was unsigned so I ad-hoc signed it), but now when running import, another similar error popped up with pysam, I followed the same strategy to sign the file, but seems not super successful -- might this due to how I installed gwaslab? I followed the conda installation instruction that first create the environment then pip install gwaslab==3.4.41. Please see below the new error message during import:
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
Cell In[1], line 1
----> 1 import gwaslab as gl
File /opt/anaconda3/envs/gwas/lib/python3.9/site-packages/gwaslab/__init__.py:1
----> 1 from gwaslab.g_Sumstats import Sumstats
2 from gwaslab.g_SumstatsT import SumstatsT
3 from gwaslab.g_SumstatsPair import SumstatsPair
File /opt/anaconda3/envs/gwas/lib/python3.9/site-packages/gwaslab/g_Sumstats.py:8
6 from gwaslab.g_Sumstats_summary import lookupstatus
7 from gwaslab.io_preformat_input import preformat
----> 8 from gwaslab.io_to_formats import _to_format
9 from gwaslab.g_Log import Log
10 from gwaslab.qc_fix_sumstats import fixID
File /opt/anaconda3/envs/gwas/lib/python3.9/site-packages/gwaslab/io_to_formats.py:5
3 import hashlib
4 import copy
----> 5 from pysam import tabix_compress
6 from pysam import tabix_index
7 from datetime import datetime
File /opt/anaconda3/envs/gwas/lib/python3.9/site-packages/pysam/__init__.py:4
1 import os
2 import sysconfig
----> 4 from pysam.libchtslib import *
5 import pysam.libchtslib as libchtslib
6 from pysam.libcsamtools import *
ImportError: dlopen(/opt/anaconda3/envs/gwas/lib/python3.9/site-packages/pysam/libchtslib.cpython-39-darwin.so, 0x0002): tried: '/opt/anaconda3/envs/gwas/lib/python3.9/site-packages/pysam/libchtslib.cpython-39-darwin.so' (code signature in <E12FE5B5-2C61-34B8-B1BB-27D9C436D54B> '/opt/anaconda3/envs/gwas/lib/python3.9/site-packages/pysam/libchtslib.cpython-39-darwin.so' not valid for use in process: Trying to load an unsigned library), '/System/Volumes/Preboot/Cryptexes/OS/opt/anaconda3/envs/gwas/lib/python3.9/site-packages/pysam/libchtslib.cpython-39-darwin.so' (no such file), '/opt/anaconda3/envs/gwas/lib/python3.9/site-packages/pysam/libchtslib.cpython-39-darwin.so' (code signature in <E12FE5B5-2C61-34B8-B1BB-27D9C436D54B> '/opt/anaconda3/envs/gwas/lib/python3.9/site-packages/pysam/libchtslib.cpython-39-darwin.so' not valid for use in process: Trying to load an unsigned library)And after codesign -s - libchtslib.cpython-39-darwin.so, and run import gwaslab as gl, I got this in the ImportError line:
ImportError: dlopen(/opt/anaconda3/envs/gwas/lib/python3.9/site-packages/pysam/libcutils.cpython-39-darwin.so, 0x0002): tried: '/opt/anaconda3/envs/gwas/lib/python3.9/site-packages/pysam/libcutils.cpython-39-darwin.so' (code signature in <5082F320-FED8-39B0-9526-D2ECF6655172> '/opt/anaconda3/envs/gwas/lib/python3.9/site-packages/pysam/libcutils.cpython-39-darwin.so' not valid for use in process: Trying to load an unsigned library), '/System/Volumes/Preboot/Cryptexes/OS/opt/anaconda3/envs/gwas/lib/python3.9/site-packages/pysam/libcutils.cpython-39-darwin.so' (no such file), '/opt/anaconda3/envs/gwas/lib/python3.9/site-packages/pysam/libcutils.cpython-39-darwin.so' (code signature in <5082F320-FED8-39B0-9526-D2ECF6655172> '/opt/anaconda3/envs/gwas/lib/python3.9/site-packages/pysam/libcutils.cpython-39-darwin.so' not valid for use in process: Trying to load an unsigned library)Then codesign -s - libcutils.cpython-39-darwin.so, and got this:
ImportError: dlopen(/opt/anaconda3/envs/gwas/lib/python3.9/site-packages/pysam/libcutils.cpython-39-darwin.so, 0x0002): Library not loaded: @rpath/libcsamtools.cpython-39-darwin.so
Referenced from: <5082F320-FED8-39B0-9526-D2ECF6655172> /opt/anaconda3/envs/gwas/lib/python3.9/site-packages/pysam/libcutils.cpython-39-darwin.so
Reason: tried: '/opt/anaconda3/envs/gwas/lib/libcsamtools.cpython-39-darwin.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/anaconda3/envs/gwas/lib/libcsamtools.cpython-39-darwin.so' (no such file), '/opt/anaconda3/envs/gwas/lib/libcsamtools.cpython-39-darwin.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/anaconda3/envs/gwas/lib/libcsamtools.cpython-39-darwin.so' (no such file), '/opt/anaconda3/envs/gwas/lib/python3.9/site-packages/pysam/libcsamtools.cpython-39-darwin.so' (code signature in <ADD598D0-7267-31FB-BE65-CD6FE7D34BCB> '/opt/anaconda3/envs/gwas/lib/python3.9/site-packages/pysam/libcsamtools.cpython-39-darwin.so' not valid for use in process: Trying to load an unsigned library), '/opt/anaconda3/envs/gwas/lib/libcsamtools.cpython-39-darwin.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/anaconda3/envs/gwas/lib/libcsamtools.cpython-39-darwin.so' (no such file), '/opt/anaconda3/envs/gwas/lib/libcsamtools.cpython-39-darwin.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/anaconda3/envs/gwas/lib/libcsamtools.cpython-39-darwin.so' (no such file), '/opt/anaconda3/envs/gwas/lib/python3.9/site-packages/pysam/libcsamtools.cpython-39-darwin.so' (code signature in <ADD598D0-7267-31FB-BE65-CD6FE7D34BCB> '/opt/anaconda3/envs/gwas/lib/python3.9/site-packages/pysam/libcsamtools.cpython-39-darwin.so' not valid for use in process: Trying to load an unsigned library), '/opt/anaconda3/envs/gwas/bin/../lib/libcsamtools.cpython-39-darwin.so' (no such file), '/opt/anaconda3/envs/gwas/bin/../lib/libcsamtools.cpython-39-darwin.so' (no such file), '/usr/local/lib/libcsamtools.cpython-39-darwin.so' (no such file), '/usr/lib/libcsamtools.cpython-39-darwin.so' (no such file, not in dyld cache)So I am not sure if I did the correct thing... I deleted pysam from the environment and reinstalled it by running again pip install gwaslab==3.4.41 but same errors still exist. any thoughts would be appreciated! Thank you!
Cloufield
commented
3 months ago I think it is still the same error. The .so files are not signed. This error is not related to gwaslab itself, but probably it is due to some settings of the packages or your compiler or something. It is quite similar to the error mentioned here https://discourse.panda3d.org/t/pip-install-om-m1-mac-works-after-some-pain/27360.
A simple temporary solution I guess is to ad-hoc sign all those .so files in the packages that have the error as shown in the link above.
ls -1 *.so | awk '{print "codesign -s -", $1}' | sh
Cloufield
commented
3 months ago FYI, the version of packages in my test environment:
Python 3.9.19
conda 24.3.0# packages in environment at /opt/anaconda3/envs/gwaslab:
#
# Name Version Build Channel
adjusttext 0.8 pypi_0 pypi
appdirs 1.4.4 pypi_0 pypi
astroid 3.1.0 pypi_0 pypi
biopython 1.83 pypi_0 pypi
bzip2 1.0.8 h93a5062_5 conda-forge
ca-certificates 2024.2.2 hf0a4a13_0 conda-forge
certifi 2024.2.2 pypi_0 pypi
charset-normalizer 3.3.2 pypi_0 pypi
click 8.1.7 pypi_0 pypi
cloudpickle 3.0.0 pypi_0 pypi
contourpy 1.2.0 pypi_0 pypi
cycler 0.12.1 pypi_0 pypi
dask 2024.3.1 pypi_0 pypi
datacache 1.4.1 pypi_0 pypi
dill 0.3.8 pypi_0 pypi
fonttools 4.50.0 pypi_0 pypi
fsspec 2024.3.1 pypi_0 pypi
gtfparse 1.3.0 pypi_0 pypi
gwaslab 3.4.41 pypi_0 pypi
idna 3.6 pypi_0 pypi
importlib-metadata 7.1.0 pypi_0 pypi
importlib-resources 6.4.0 pypi_0 pypi
isort 5.13.2 pypi_0 pypi
kiwisolver 1.4.5 pypi_0 pypi
libffi 3.4.2 h3422bc3_5 conda-forge
libsqlite 3.45.2 h091b4b1_0 conda-forge
libzlib 1.2.13 h53f4e23_5 conda-forge
liftover 1.2.2 pypi_0 pypi
locket 1.0.0 pypi_0 pypi
matplotlib 3.8.3 pypi_0 pypi
mccabe 0.7.0 pypi_0 pypi
memoized-property 1.0.3 pypi_0 pypi
mock 5.1.0 pypi_0 pypi
ncurses 6.4.20240210 h078ce10_0 conda-forge
nose 1.3.7 pypi_0 pypi
numpy 1.26.4 pypi_0 pypi
openssl 3.2.1 h0d3ecfb_1 conda-forge
packaging 24.0 pypi_0 pypi
pandas 2.2.1 pypi_0 pypi
partd 1.4.1 pypi_0 pypi
pillow 10.2.0 pypi_0 pypi
pip 24.0 pyhd8ed1ab_0 conda-forge
platformdirs 4.2.0 pypi_0 pypi
progressbar33 2.4 pypi_0 pypi
pyensembl 2.2.3 pypi_0 pypi
pylint 3.1.0 pypi_0 pypi
pyparsing 3.1.2 pypi_0 pypi
pysam 0.19.1 pypi_0 pypi
python 3.9.19 hd7ebdb9_0_cpython conda-forge
python-dateutil 2.9.0.post0 pypi_0 pypi
pytz 2024.1 pypi_0 pypi
pyyaml 6.0.1 pypi_0 pypi
readline 8.2 h92ec313_1 conda-forge
requests 2.31.0 pypi_0 pypi
scikit-allel 1.3.8 pypi_0 pypi
scipy 1.12.0 pypi_0 pypi
seaborn 0.13.2 pypi_0 pypi
serializable 0.4.0 pypi_0 pypi
setuptools 69.2.0 pyhd8ed1ab_0 conda-forge
simplejson 3.19.2 pypi_0 pypi
six 1.16.0 pypi_0 pypi
tinytimer 0.0.0 pypi_0 pypi
tk 8.6.13 h5083fa2_1 conda-forge
tomli 2.0.1 pypi_0 pypi
tomlkit 0.12.4 pypi_0 pypi
toolz 0.12.1 pypi_0 pypi
typechecks 0.1.0 pypi_0 pypi
typing-extensions 4.10.0 pypi_0 pypi
tzdata 2024.1 pypi_0 pypi
urllib3 2.2.1 pypi_0 pypi
wheel 0.43.0 pyhd8ed1ab_1 conda-forge
xz 5.2.6 h57fd34a_0 conda-forge
zipp 3.18.1 pypi_0 pypiHello, thank you for all helpful insights and suggestions - they've been really helpful and finally solved my issue!
After I re-installed and run the ls -1 *.so | awk '{print "codesign -s -", $1}' | sh inside the lifeover and pysam directories, I was able to run import gwaslab as gl without any errors and finally able to use gwaslab on my mac. Thanks again!
Hello, thanks for this great package! I had an import error after installing the package 3.4.41. I am using Apple M2 Max chip, I saw you mentioned this package might not work well with M chips #82 when installing, but I think I managed to install it - just having trouble importing it? Any help would be appreciated! Please find the error log below:
ImportError Traceback (most recent call last) Cell In[1], line 1 ----> 1 import gwaslab as gl
File /opt/anaconda3/envs/gwas/lib/python3.9/site-packages/gwaslab/init.py:1 ----> 1 from gwaslab.g_Sumstats import Sumstats 2 from gwaslab.g_SumstatsT import SumstatsT 3 from gwaslab.g_SumstatsPair import SumstatsPair
File /opt/anaconda3/envs/gwas/lib/python3.9/site-packages/gwaslab/g_Sumstats.py:7 5 from gwaslab.g_Sumstats_summary import summarize 6 from gwaslab.g_Sumstats_summary import lookupstatus ----> 7 from gwaslab.io_preformat_input import preformat 8 from gwaslab.io_to_formats import _to_format 9 from gwaslab.g_Log import Log
File /opt/anaconda3/envs/gwas/lib/python3.9/site-packages/gwaslab/io_preformat_input.py:8 6 import gc 7 from gwaslab.bd_common_data import get_format_dict ----> 8 from gwaslab.qc_fix_sumstats import sortcolumn 9 from gwaslab.qc_fix_sumstats import _process_build 10 from gwaslab.qc_check_datatype import check_datatype
File /opt/anaconda3/envs/gwas/lib/python3.9/site-packages/gwaslab/qc_fix_sumstats.py:7 5 from itertools import repeat 6 from multiprocessing import Pool ----> 7 from liftover import get_lifter 8 from functools import partial 9 from gwaslab.g_vchange_status import vchange_status
File /opt/anaconda3/envs/gwas/lib/python3.9/site-packages/liftover/init.py:6 3 name = 'liftover' 4 version = version(name) ----> 6 from liftover.lifter import get_lifter 7 from liftover.chain_file import ChainFile
File /opt/anaconda3/envs/gwas/lib/python3.9/site-packages/liftover/lifter.py:4 2 import os ----> 4 from liftover.chain_file import ChainFile 5 from liftover.download_file import download_file 7 def get_lifter(target: str, query: str, cache: str=None, one_based=False, chain_server='https://hgdownload.soe.ucsc.edu'):
ImportError: dlopen(/opt/anaconda3/envs/gwas/lib/python3.9/site-packages/liftover/chain_file.cpython-39-darwin.so, 0x0002): tried: '/opt/anaconda3/envs/gwas/lib/python3.9/site-packages/liftover/chain_file.cpython-39-darwin.so' (code signature in <0E0CB2A7-2B51-3EBF-9824-2D3BCFB180BB> '/opt/anaconda3/envs/gwas/lib/python3.9/site-packages/liftover/chain_file.cpython-39-darwin.so' not valid for use in process: Trying to load an unsigned library), '/System/Volumes/Preboot/Cryptexes/OS/opt/anaconda3/envs/gwas/lib/python3.9/site-packages/liftover/chain_file.cpython-39-darwin.so' (no such file), '/opt/anaconda3/envs/gwas/lib/python3.9/site-packages/liftover/chain_file.cpython-39-darwin.so' (code signature in <0E0CB2A7-2B51-3EBF-9824-2D3BCFB180BB> '/opt/anaconda3/envs/gwas/lib/python3.9/site-packages/liftover/chain_file.cpython-39-darwin.so' not valid for use in process: Trying to load an unsigned library)
PS: I tried to update the liftover package based on the error above but liftover seems to be working: Requirement already satisfied: liftover in /opt/anaconda3/envs/gwas/lib/python3.9/site-packages (1.2.2) Requirement already satisfied: urllib3 in /opt/anaconda3/envs/gwas/lib/python3.9/site-packages (from liftover) (2.2.1) Note: you may need to restart the kernel to use updated packages.