susieR update - Githubissues

bschilder commented 4 years ago

Hey Omer,

It looks like susieR has been updated so that in the newest version susie_bhat() is no longer a function (it was merged into the new function susie_suff_stat (). Unfortunately this change causes an error in run_finemapper.py:

python ./echolocatoR/tools/polyfun/run_finemapper.py --ld Data/GWAS/Nalls23andMe_2019/LRRK2/plink/chr12_40000001_43000001 --sumstats Data/GWAS/Nalls23andMe_2019/_genome_wide/PolyFun/output/PD_GWAS.12.snpvar_constrained.gz --n 1474097 --chr 12 --start 40515388 --end 40713899 --method susie --max-num-causal 10 --out Data/GWAS/Nalls23andMe_2019/LRRK2/finemap.UKBB.LRRK2.gz
*********************************************************************
* Fine-mapping Wrapper
* Version 1.0.0
* (C) 2019 Omer Weissbrod
*********************************************************************

[INFO]  Loaded an LD matrix for 22522 SNPs from Data/GWAS/Nalls23andMe_2019/LRRK2/plink/chr12_40000001_43000001.npz
[INFO]  Loading sumstats file...
[INFO]  Loaded sumstats for 373589 SNPs
[1] "/usr/local/anaconda3/envs/polyfun_venv/lib/R"

[INFO]  Removing 21839 SNPs in the LD matrix that are not in the sumstats file
[WARNING]  Flipping the alleles of 380 SNPs in the LD matrix that are flipped compared to the sumstats file
[INFO]  Starting functionally-informed SuSiE fine-mapping for chromosome 12 BP 40515388-40713899 (683 SNPs)
Traceback (most recent call last):
  File "./echolocatoR/tools/polyfun/run_finemapper.py", line 118, in <module>
    verbose=args.verbose, ld=ld, df_ld_snps=df_ld_snps)
  File "/Users/schilder/Desktop/Fine_Mapping/echolocatoR/tools/polyfun/finemapper.py", line 492, in finemap
    susie_obj = self.susieR.susie_bhat(
AttributeError: module 'susieR' has no attribute 'susie_bhat'

To handle this, in my R scripts I wrote a little snippet of code that searches for the new function and if it can't find it, uses the old one. Maybe you could include a pythonic version of this in your script?:

susie_func <- ifelse(length(find("susie_bhat"))==0, 
                       susieR::susie_suff_stat, susieR::susie_bhat)  
  fitted_bhat <- susie_func(...)

Moreover, as a way of dealing with these little dependency/versioning issues, would it be possible for you to provide a yaml file that contains all of the python, R, and other packages necessary for running PolyFun? This would be a really great way to make the right Conda environment and get started with PolyFun right away. Here's one example from a related repo by OpenTargets.

Thanks, Brian

omerwe commented 4 years ago

Thanks for the heads-up Brian! I fixed the code to handle either the old or new SuSiE versions. I looked into creating a .yaml file but it doesn't carry over the R packages so I'm not sure how useful it will be...

bschilder commented 4 years ago

Thanks!

As for Conda, you can actually add R packages too and then export them to a yaml file using conda env export while your environment is activated.

Most of susieR's dependencies are available on Anaconda Cloud. susieR itself is the only one I haven't figured out yet (since it's only in a GitHub repo), but apparently there's ways to also add GitHub repos by downloading them from source. Hope this helps!

Here's the .yaml for the polyfun_venv I've been using:

name: polyfun_venv
channels:
  - r
  - conda-forge
  - defaults
dependencies:
  - _r-mutex=1.0.0=anacondar_1
  - arrow-cpp=0.13.0=py37h8cfbac2_0
  - asn1crypto=1.2.0=py37_0
  - bitarray=1.1.0=py37h1de35cc_0
  - blas=1.0=mkl
  - boost-cpp=1.67.0=h1de35cc_4
  - brotli=1.0.7=h0a44026_0
  - bwidget=1.9.11=1
  - bzip2=1.0.8=h1de35cc_0
  - ca-certificates=2019.11.27=0
  - cairo=1.14.12=hc4e6be7_4
  - cctools=895=1
  - certifi=2019.11.28=py37_0
  - cffi=1.13.2=py37hb5b8e2f_0
  - chardet=3.0.4=py37_1003
  - clang=4.0.1=1
  - clang_osx-64=4.0.1=h1ce6c1d_18
  - clangxx=4.0.1=1
  - clangxx_osx-64=4.0.1=h22b1bf0_18
  - compiler-rt=4.0.1=hcfea43d_1
  - cryptography=2.8=py37ha12b0ac_0
  - curl=7.67.0=ha441bb4_0
  - decorator=4.4.1=py_0
  - double-conversion=3.1.5=haf313ee_1
  - expat=2.2.6=h0a44026_0
  - fontconfig=2.13.0=h5d5b041_1
  - freetype=2.9.1=hb4e5f40_0
  - fribidi=1.0.5=h1de35cc_0
  - gettext=0.19.8.1=h15daf44_3
  - gflags=2.2.2=h0a44026_0
  - gfortran_osx-64=4.8.5=h22b1bf0_8
  - git=2.23.0=pl526h6951d83_0
  - glib=2.63.1=hd977a24_0
  - glog=0.4.0=h0a44026_0
  - graphite2=1.3.13=h2098e52_0
  - gsl=2.4=h1de35cc_4
  - harfbuzz=1.8.8=hb8d4a28_0
  - icu=58.2=h4b95b61_1
  - idna=2.8=py37_0
  - intel-openmp=2019.4=233
  - jinja2=2.10.3=py_0
  - joblib=0.14.1=py_0
  - jpeg=9b=he5867d9_2
  - krb5=1.16.4=hddcf347_0
  - ld64=274.2=1
  - libboost=1.67.0=hebc422b_4
  - libcurl=7.67.0=h051b688_0
  - libcxx=4.0.1=hcfea43d_1
  - libcxxabi=4.0.1=hcfea43d_1
  - libedit=3.1.20181209=hb402a30_0
  - libevent=2.1.8=ha12b0ac_0
  - libffi=3.2.1=h475c297_4
  - libgfortran=3.0.1=h93005f0_2
  - libiconv=1.15=hdd342a3_7
  - libpng=1.6.37=ha441bb4_0
  - libprotobuf=3.6.0=hd9629dc_0
  - libssh2=1.8.2=ha12b0ac_0
  - libtiff=4.1.0=hcb84e12_0
  - libxml2=2.9.9=hf6e021a_1
  - llvm=4.0.1=1
  - llvm-lto-tapi=4.0.1=1
  - llvm-openmp=4.0.1=hcfea43d_1
  - lz4-c=1.8.1.2=h1de35cc_0
  - make=4.2.1=h3efe00b_1
  - markupsafe=1.1.1=py37h1de35cc_0
  - mkl=2019.4=233
  - mkl-service=2.3.0=py37hfbe908c_0
  - mkl_fft=1.0.15=py37h5e564d8_0
  - mkl_random=1.1.0=py37ha771720_0
  - ncurses=6.1=h0a44026_1
  - networkx=2.4=py_0
  - numpy=1.17.3=py37h4174a10_0
  - numpy-base=1.17.3=py37h6575580_0
  - openssl=1.1.1d=h1de35cc_3
  - pandas=0.25.3=py37h0a44026_0
  - pango=1.42.4=h060686c_0
  - pcre=8.43=h0a44026_0
  - perl=5.26.2=h4e221da_0
  - pip=19.3.1=py37_0
  - pixman=0.38.0=h1de35cc_0
  - pyarrow=0.13.0=py37h0a44026_0
  - pycparser=2.19=py37_0
  - pyopenssl=19.1.0=py37_0
  - pysocks=1.7.1=py37_0
  - python=3.7.3=h359304d_0
  - python-dateutil=2.8.1=py_0
  - pytz=2019.3=py_0
  - r-askpass=1.0=r36h1de35cc_0
  - r-assertthat=0.2.1=r36h6115d3f_0
  - r-backports=1.1.4=r36h46e59ec_0
  - r-base=3.6.1=hcb44179_1
  - r-bh=1.69.0_1=r36h6115d3f_0
  - r-bit=1.1_14=r36h46e59ec_0
  - r-bit64=0.9_7=r36h46e59ec_0
  - r-blob=1.1.1=r36h6115d3f_0
  - r-callr=3.2.0=r36h6115d3f_0
  - r-cli=1.1.0=r36h6115d3f_0
  - r-clipr=0.6.0=r36h6115d3f_0
  - r-clisymbols=1.2.0=r36h6115d3f_0
  - r-crayon=1.3.4=r36h6115d3f_0
  - r-curl=3.3=r36h46e59ec_0
  - r-dbi=1.0.0=r36h6115d3f_0
  - r-dbplyr=1.4.0=r36h6115d3f_0
  - r-desc=1.2.0=r36h6115d3f_0
  - r-devtools=2.0.2=r36h6115d3f_0
  - r-digest=0.6.18=r36h46e59ec_0
  - r-dplyr=0.8.0.1=r36h466af19_0
  - r-expm=0.999_4=r36haf69682_2
  - r-fansi=0.4.0=r36h46e59ec_0
  - r-fs=1.2.7=r36h466af19_0
  - r-gh=1.0.1=r36h6115d3f_0
  - r-git2r=0.25.2=r36h46e59ec_0
  - r-glue=1.3.1=r36h46e59ec_0
  - r-httr=1.4.0=r36h6115d3f_0
  - r-ini=0.3.1=r36h6115d3f_0
  - r-jsonlite=1.6=r36h46e59ec_0
  - r-lattice=0.20_38=r36h46e59ec_0
  - r-magrittr=1.5=r36h6115d3f_4
  - r-mass=7.3_51.3=r36h46e59ec_0
  - r-matrix=1.2_17=r36h46e59ec_0
  - r-matrixstats=0.54.0=r36h46e59ec_0
  - r-memoise=1.1.0=r36h6115d3f_0
  - r-mime=0.6=r36h46e59ec_0
  - r-openssl=1.3=r36h46e59ec_0
  - r-pillar=1.3.1=r36h6115d3f_0
  - r-pkgbuild=1.0.3=r36h6115d3f_0
  - r-pkgconfig=2.0.2=r36h6115d3f_0
  - r-pkgload=1.0.2=r36h466af19_0
  - r-plogr=0.2.0=r36h6115d3f_0
  - r-prettyunits=1.0.2=r36h6115d3f_0
  - r-processx=3.3.0=r36h46e59ec_0
  - r-ps=1.3.0=r36h46e59ec_0
  - r-purrr=0.3.2=r36h46e59ec_0
  - r-r6=2.4.0=r36h6115d3f_0
  - r-rcmdcheck=1.3.2=r36h6115d3f_0
  - r-rcpp=1.0.1=r36h466af19_0
  - r-remotes=2.0.4=r36h6115d3f_0
  - r-rlang=0.3.4=r36h46e59ec_0
  - r-rprojroot=1.3_2=r36h6115d3f_0
  - r-rsqlite=2.1.1=r36h466af19_0
  - r-rstudioapi=0.10=r36h6115d3f_0
  - r-sessioninfo=1.1.1=r36h6115d3f_0
  - r-sys=3.2=r36h46e59ec_0
  - r-tibble=2.1.1=r36h46e59ec_0
  - r-tidyselect=0.2.5=r36h466af19_0
  - r-usethis=1.5.0=r36h6115d3f_0
  - r-utf8=1.1.4=r36h46e59ec_0
  - r-wavethresh=4.6.8=r36h46e59ec_0
  - r-whisker=0.3_2=r36h6115d3f_4
  - r-withr=2.1.2=r36h6115d3f_0
  - r-xopen=1.0.0=r36h6115d3f_0
  - r-yaml=2.2.0=r36h46e59ec_0
  - re2=2019.08.01=h0a44026_0
  - readline=7.0=h1de35cc_5
  - requests=2.22.0=py37_1
  - rpy2=2.9.4=py37r36h1d22016_0
  - scikit-learn=0.21.3=py37h27c97d8_0
  - scipy=1.3.1=py37h1410ff5_0
  - setuptools=42.0.2=py37_0
  - six=1.13.0=py37_0
  - snappy=1.1.7=he62c110_3
  - sqlite=3.30.1=ha441bb4_0
  - thrift-cpp=0.11.0=hd79cdb6_3
  - tk=8.6.8=ha441bb4_0
  - tktable=2.10=h1de35cc_0
  - tqdm=4.38.0=py_0
  - urllib3=1.25.7=py37_0
  - wheel=0.33.6=py37_0
  - xz=5.2.4=h1de35cc_4
  - zlib=1.2.11=h1de35cc_3
  - zstd=1.3.7=h5bba6e5_0

omerwe commented 4 years ago

Thanks Brian, that's a great idea!!! I did this and updated the installation instructions. Thanks so much!!!

On Wed, Dec 18, 2019 at 5:59 PM Brian M. Schilder notifications@github.com wrote:

Thanks!

As for Conda, you can actually add R packages too https://docs.anaconda.com/anaconda/packages/r-language-pkg-docs/ and then export them to a yaml file https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html using conda env export while your environment is activated.

Most of susieR's dependencies are available on Anaconda Cloud https://anaconda.org/r/r-matrix. susieR itself is the only one I haven't figured out yet (since it's only in a GitHub repo), but apparently there's ways to also add GitHub repos by downloading them from source https://stackoverflow.com/questions/19042389/conda-installing-upgrading-directly-from-github. Hope this helps!

Here's the .yaml for the polyfun_venv I've been using:

name: polyfun_venv channels:

r

conda-forge

defaults dependencies:

_r-mutex=1.0.0=anacondar_1

arrow-cpp=0.13.0=py37h8cfbac2_0

asn1crypto=1.2.0=py37_0

bitarray=1.1.0=py37h1de35cc_0

blas=1.0=mkl

boost-cpp=1.67.0=h1de35cc_4

brotli=1.0.7=h0a44026_0

bwidget=1.9.11=1

bzip2=1.0.8=h1de35cc_0

ca-certificates=2019.11.27=0

cairo=1.14.12=hc4e6be7_4

cctools=895=1

certifi=2019.11.28=py37_0

cffi=1.13.2=py37hb5b8e2f_0

chardet=3.0.4=py37_1003

clang=4.0.1=1

clang_osx-64=4.0.1=h1ce6c1d_18

clangxx=4.0.1=1

clangxx_osx-64=4.0.1=h22b1bf0_18

compiler-rt=4.0.1=hcfea43d_1

cryptography=2.8=py37ha12b0ac_0

curl=7.67.0=ha441bb4_0

decorator=4.4.1=py_0

double-conversion=3.1.5=haf313ee_1

expat=2.2.6=h0a44026_0

fontconfig=2.13.0=h5d5b041_1

freetype=2.9.1=hb4e5f40_0

fribidi=1.0.5=h1de35cc_0

gettext=0.19.8.1=h15daf44_3

gflags=2.2.2=h0a44026_0

gfortran_osx-64=4.8.5=h22b1bf0_8

git=2.23.0=pl526h6951d83_0

glib=2.63.1=hd977a24_0

glog=0.4.0=h0a44026_0

graphite2=1.3.13=h2098e52_0

gsl=2.4=h1de35cc_4

harfbuzz=1.8.8=hb8d4a28_0

icu=58.2=h4b95b61_1

idna=2.8=py37_0

intel-openmp=2019.4=233

jinja2=2.10.3=py_0

joblib=0.14.1=py_0

jpeg=9b=he5867d9_2

krb5=1.16.4=hddcf347_0

ld64=274.2=1

libboost=1.67.0=hebc422b_4

libcurl=7.67.0=h051b688_0

libcxx=4.0.1=hcfea43d_1

libcxxabi=4.0.1=hcfea43d_1

libedit=3.1.20181209=hb402a30_0

libevent=2.1.8=ha12b0ac_0

libffi=3.2.1=h475c297_4

libgfortran=3.0.1=h93005f0_2

libiconv=1.15=hdd342a3_7

libpng=1.6.37=ha441bb4_0

libprotobuf=3.6.0=hd9629dc_0

libssh2=1.8.2=ha12b0ac_0

libtiff=4.1.0=hcb84e12_0

libxml2=2.9.9=hf6e021a_1

llvm=4.0.1=1

llvm-lto-tapi=4.0.1=1

llvm-openmp=4.0.1=hcfea43d_1

lz4-c=1.8.1.2=h1de35cc_0

make=4.2.1=h3efe00b_1

markupsafe=1.1.1=py37h1de35cc_0

mkl=2019.4=233

mkl-service=2.3.0=py37hfbe908c_0

mkl_fft=1.0.15=py37h5e564d8_0

mkl_random=1.1.0=py37ha771720_0

ncurses=6.1=h0a44026_1

networkx=2.4=py_0

numpy=1.17.3=py37h4174a10_0

numpy-base=1.17.3=py37h6575580_0

openssl=1.1.1d=h1de35cc_3

pandas=0.25.3=py37h0a44026_0

pango=1.42.4=h060686c_0

pcre=8.43=h0a44026_0

perl=5.26.2=h4e221da_0

pip=19.3.1=py37_0

pixman=0.38.0=h1de35cc_0

pyarrow=0.13.0=py37h0a44026_0

pycparser=2.19=py37_0

pyopenssl=19.1.0=py37_0

pysocks=1.7.1=py37_0

python=3.7.3=h359304d_0

python-dateutil=2.8.1=py_0

pytz=2019.3=py_0

r-askpass=1.0=r36h1de35cc_0

r-assertthat=0.2.1=r36h6115d3f_0

r-backports=1.1.4=r36h46e59ec_0

r-base=3.6.1=hcb44179_1

r-bh=1.69.0_1=r36h6115d3f_0

r-bit=1.1_14=r36h46e59ec_0

r-bit64=0.9_7=r36h46e59ec_0

r-blob=1.1.1=r36h6115d3f_0

r-callr=3.2.0=r36h6115d3f_0

r-cli=1.1.0=r36h6115d3f_0

r-clipr=0.6.0=r36h6115d3f_0

r-clisymbols=1.2.0=r36h6115d3f_0

r-crayon=1.3.4=r36h6115d3f_0

r-curl=3.3=r36h46e59ec_0

r-dbi=1.0.0=r36h6115d3f_0

r-dbplyr=1.4.0=r36h6115d3f_0

r-desc=1.2.0=r36h6115d3f_0

r-devtools=2.0.2=r36h6115d3f_0

r-digest=0.6.18=r36h46e59ec_0

r-dplyr=0.8.0.1=r36h466af19_0

r-expm=0.999_4=r36haf69682_2

r-fansi=0.4.0=r36h46e59ec_0

r-fs=1.2.7=r36h466af19_0

r-gh=1.0.1=r36h6115d3f_0

r-git2r=0.25.2=r36h46e59ec_0

r-glue=1.3.1=r36h46e59ec_0

r-httr=1.4.0=r36h6115d3f_0

r-ini=0.3.1=r36h6115d3f_0

r-jsonlite=1.6=r36h46e59ec_0

r-lattice=0.20_38=r36h46e59ec_0

r-magrittr=1.5=r36h6115d3f_4

r-mass=7.3_51.3=r36h46e59ec_0

r-matrix=1.2_17=r36h46e59ec_0

r-matrixstats=0.54.0=r36h46e59ec_0

r-memoise=1.1.0=r36h6115d3f_0

r-mime=0.6=r36h46e59ec_0

r-openssl=1.3=r36h46e59ec_0

r-pillar=1.3.1=r36h6115d3f_0

r-pkgbuild=1.0.3=r36h6115d3f_0

r-pkgconfig=2.0.2=r36h6115d3f_0

r-pkgload=1.0.2=r36h466af19_0

r-plogr=0.2.0=r36h6115d3f_0

r-prettyunits=1.0.2=r36h6115d3f_0

r-processx=3.3.0=r36h46e59ec_0

r-ps=1.3.0=r36h46e59ec_0

r-purrr=0.3.2=r36h46e59ec_0

r-r6=2.4.0=r36h6115d3f_0

r-rcmdcheck=1.3.2=r36h6115d3f_0

r-rcpp=1.0.1=r36h466af19_0

r-remotes=2.0.4=r36h6115d3f_0

r-rlang=0.3.4=r36h46e59ec_0

r-rprojroot=1.3_2=r36h6115d3f_0

r-rsqlite=2.1.1=r36h466af19_0

r-rstudioapi=0.10=r36h6115d3f_0

r-sessioninfo=1.1.1=r36h6115d3f_0

r-sys=3.2=r36h46e59ec_0

r-tibble=2.1.1=r36h46e59ec_0

r-tidyselect=0.2.5=r36h466af19_0

r-usethis=1.5.0=r36h6115d3f_0

r-utf8=1.1.4=r36h46e59ec_0

r-wavethresh=4.6.8=r36h46e59ec_0

r-whisker=0.3_2=r36h6115d3f_4

r-withr=2.1.2=r36h6115d3f_0

r-xopen=1.0.0=r36h6115d3f_0

r-yaml=2.2.0=r36h46e59ec_0

re2=2019.08.01=h0a44026_0

readline=7.0=h1de35cc_5

requests=2.22.0=py37_1

rpy2=2.9.4=py37r36h1d22016_0

scikit-learn=0.21.3=py37h27c97d8_0

scipy=1.3.1=py37h1410ff5_0

setuptools=42.0.2=py37_0

six=1.13.0=py37_0

snappy=1.1.7=he62c110_3

sqlite=3.30.1=ha441bb4_0

thrift-cpp=0.11.0=hd79cdb6_3

tk=8.6.8=ha441bb4_0

tktable=2.10=h1de35cc_0

tqdm=4.38.0=py_0

urllib3=1.25.7=py37_0

wheel=0.33.6=py37_0

xz=5.2.4=h1de35cc_4

zlib=1.2.11=h1de35cc_3

zstd=1.3.7=h5bba6e5_0

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/omerwe/polyfun/issues/18?email_source=notifications&email_token=ACNCB45ULTDRTLI5SVXM7YDQZKTOZA5CNFSM4J4DP3H2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHHY5KQ#issuecomment-567250602, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACNCB4Z24PMQYPVX7CRDWYDQZKTOZANCNFSM4J4DP3HQ .

bschilder commented 4 years ago

No problem, glad it was helpful! Just checked out your .yml file. Though you might want to also specify the pandas version in the .yml file bc I know that was an issue earlier.

mkoromina commented 2 years ago

Hi @omerwe , sorry to re-open this, but I also get the same error message when running polyfun-susie. My .yaml file looks like this:

name: polyfun
channels:
  - conda-forge
  - nodefaults
dependencies:
  - tqdm
  - r-wavethresh
  - pyarrow>=3.0
  - python=3.8
  - scikit-learn
  - r-lattice
  - r-ckmeans.1d.dp
  - r-stringi
  - bitarray
  - r-base
  - r
  - r-matrixstats
  - networkx
  - rpy2
  - scipy
  - r-devtools
  - numpy>=1.20
  - pandas
  - r-expm
  - pandas-plink
  - pip
  - r-susier
  - pip:
    - bgen==1.2.10

Happy to listen to your thoughts! Many thanks!

jdblischak commented 2 years ago

Try updating your conda env with r-susier=0.11.92. For details see #119 and https://github.com/stephenslab/susieR/issues/169

omerwe commented 2 years ago

@jdblischak thanks for addressing these issues!

I have very limited bandwidth for PolyFun maintenance, so I'll try to save some time by asking: Do you think I should edit the polyfun.yml file to explicitly ask for SuSIE v.0.11.92?

jdblischak commented 2 years ago

I have very limited bandwidth for PolyFun maintenance

I totally understand. And in general I think you do an amazing job keeping on top of this previous project.

Do you think I should edit the polyfun.yml file to explicitly ask for SuSIE v.0.11.92?

Yes, that is the best short term solution. Longer term I think the plan should be to migrate to using susie_rss(), but you can wait until I or someone else with sufficient motivation/bandwidth sends a PR with that update.

omerwe / polyfun

susieR update #18