AttributeError: module 'susieR' has no attribute 'susie_bhat'

[bk11-lab]

Hi, I am getting following error while running finemapping. What might be the problem?

cat sumstats.txt

CHR  SNP          BP         A1  A2  SNPVAR      Z            N
1    rs186077422  10000006   A   G   1.8599e-08  -1.7100e+00  80000
1    rs6678176    100000827  T   C   2.2472e-08  1.1075e+00   80000
1    rs78286437   100000843  T   C   1.6348e-08  2.5414e-01   80000
1    rs144406489  100001138  A   G   1.6348e-08  1.9074e-02   80000
1    rs10875231   100000012  T   G   1.6348e-08  1.2727e+00   80000

UKB LD matrix

wget https://data.broadinstitute.org/alkesgroup/UKBB_LD/chr1_100000001_103000001.npz
wget https://data.broadinstitute.org/alkesgroup/UKBB_LD/chr1_100000001_103000001.gz

finemapping command

python run_finemapper.py \
    --ld chr1_100000001_103000001 \
    --sumstats sumstats.txt \
    --n 100000 \
    --chr 1 \
    --start 100000000 \
    --end 100000850 \
    --method susie \
    --max-num-causal 5 \
    --out temp/finemap.a.txt.gz

ERROR Message

[WARNING]  R[write to console]: Error in (function (XtX, Xty, yty, n, X_colmeans = NA, y_mean = NA, maf = NULL,  : 
  unused arguments (bhat = c(-1.1075, 0.25414, -1.2727), shat = c(1, 1, 1), R = c(1, 0.360047191381454, 0.881189942359924, 0.360047191381454, 1, -0.124853432178497, 0.8811899423
59924, -0.124853432178497, 1))

Traceback (most recent call last):
  File "../../polyfun/polyfun/finemapper.py", line 836, in finemap
    susie_obj = self.susieR.susie_suff_stat(
  File "/packages/miniconda3/lib/python3.9/site-packages/rpy2/robjects/functions.py", line 202, in __call__
    return (super(SignatureTranslatedFunction, self)
  File "/packages/miniconda3/lib/python3.9/site-packages/rpy2/robjects/functions.py", line 125, in __call__
    res = super(Function, self).__call__(*new_args, **new_kwargs)
  File "/packages/miniconda3/lib/python3.9/site-packages/rpy2/rinterface_lib/conversion.py", line 45, in _
    cdata = function(*args, **kwargs)
  File "/packages/miniconda3/lib/python3.9/site-packages/rpy2/rinterface.py", line 813, in __call__
    raise embedded.RRuntimeError(_rinterface._geterrmessage())
rpy2.rinterface_lib.embedded.RRuntimeError: Error in (function (XtX, Xty, yty, n, X_colmeans = NA, y_mean = NA, maf = NULL,  : 
  unused arguments (bhat = c(-1.1075, 0.25414, -1.2727), shat = c(1, 1, 1), R = c(1, 0.360047191381454, 0.881189942359924, 0.360047191381454, 1, -0.124853432178497, 0.881189942359924, -0.124853432178497, 1))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "../../polyfun/polyfun/finemapper.py", line 1267, in <module>
    df_finemap = finemap_obj.finemap(locus_start=args.start, locus_end=args.end, num_causal_snps=args.max_num_causal,
  File "../../polyfun/polyfun/finemapper.py", line 851, in finemap
    susie_obj = self.susieR.susie_bhat(
AttributeError: module 'susieR' has no attribute 'susie_bhat'

[jdblischak]

AttributeError: module 'susieR' has no attribute 'susie_bhat'

That error is from an exception. This is the error that stopped the execution of the script:

rpy2.rinterface_lib.embedded.RRuntimeError: Error in (function (XtX, Xty, yty, n, X_colmeans = NA, y_mean = NA, maf = NULL,  : 
  unused arguments (bhat = c(-1.1075, 0.25414, -1.2727), shat = c(1, 1, 1), R = c(1, 0.360047191381454, 0.881189942359924, 0.360047191381454, 1, -0.124853432178497, 0.881189942359924, -0.124853432178497, 1))

This is likely due to recent changes in susieR, see https://github.com/stephenslab/susieR/issues/169

Try installing susieR 0.11.92 and re-running:

# with conda/mamba (recommended)
mamba install -c conda-forge r-susier=0.11.92

# in R
remotes::install_version("susieR", "0.11.92")

[omerwe]

I changed SuSiE version to 0.11.92 (in the .yml file), I hope this fixes the issue!

[mkoromina]

Hi @omerwe ,sorry to re-open this. I just git pull the changes from that directory, but unfortunately the error is still persistent for me when I try to run finemapper.py after activating the environment. Could it be that the environment needs to be reinstalled?

Update: As soon as I re-installed from scratch the polyfun conda environment, it seemed to be working okay!

[jdblischak]

Update: As soon as I re-installed from scratch the polyfun conda environment, it seemed to be working okay!

@mkoromina A full re-installation works, but in the future you could save some time by updating your existing environment:

git pull
mamba env update --file polyfun.yml 
conda activate polyfun
# to confirm version that is installed
conda list r-susier

[mkoromina]

Many thanks @jdblischak ! that is very helpful!

[alhenry]

Reopening this issue in case anyone having the same problem. Looking through the release notes from susieR

Seems like the susie_bhat function is merged into susie_suff_stat since v0.9.0

The version that works for me is 0.7.1 This can be installed within R e.g. with

devtools::install_github("stephenslab/susieR@0.7.1")

Not sure if conda-forge has the equivalent version

[jdblischak]

@alhenry What version of susieR were you using when you got an error? Did you try version 0.11.92? By using susieR 0.7.1, you are removing 2+ years worth of active development.

[alhenry]

@jdblischak I did try 0.11.92, and got the following:

> library(susieR)
> packageVersion('susieR')
[1] ‘0.11.92’
> exists('susie_bhat', where='package:susieR', mode='function')
[1] FALSE

In a separate conda environment, I installed v0.7.1 and got the following

> library(susieR)
> packageVersion('susieR')
[1] ‘0.7.1’
> exists('susie_bhat', where='package:susieR', mode='function')
[1] TRUE

I'm not hugely familiar with the internals of susie, but my understanding is that the change mainly affects the function calls rather than the underlying model that is implemented in polyfun wrapper (in which case results are hopefully not affected)?

[jdblischak]

@alhenry You're correct that susie_bhat() exists in susieR version 0.7.1 and not version 0.11.92. However, I am confused what problem you are trying to solve. The latest version of the PolyFun scripts should work fine with susieR 0.11.92. Did you get an error when you tried to run them? If yes, could you please share the full error message? The solution to fix PolyFun/susieR related errors is to update both PolyFun and susieR, not downgrade susieR to a version from 2019.

[alhenry]

@jdblischak thanks for looking into this. I tried polyfun with susie v0.11.92 and got the following error:

WARNING]  R[write to console]: Error in (function (z, R, z_ld_weight = 0, prior_variance = 50, estimate_p
rior_variance = TRUE,  : 
  argument "z" is missing, with no default

Traceback (most recent call last):
  File "scripts/polyfun/finemapper.py", line 847, in finemap
    susie_obj = self.susieR.susie_rss(
  File "/home/rmhiaah/anaconda3/envs/polyfun/lib/python3.8/site-packages/rpy2/robjects/functions.py", line
 201, in __call__
    return (super(SignatureTranslatedFunction, self)
  File "/home/rmhiaah/anaconda3/envs/polyfun/lib/python3.8/site-packages/rpy2/robjects/functions.py", line
 124, in __call__
    res = super(Function, self).__call__(*new_args, **new_kwargs)
  File "/home/rmhiaah/anaconda3/envs/polyfun/lib/python3.8/site-packages/rpy2/rinterface_lib/conversion.py
", line 45, in _
    cdata = function(*args, **kwargs)
  File "/home/rmhiaah/anaconda3/envs/polyfun/lib/python3.8/site-packages/rpy2/rinterface.py", line 810, in
 __call__
    raise embedded.RRuntimeError(_rinterface._geterrmessage())
rpy2.rinterface_lib.embedded.RRuntimeError: Error in (function (z, R, z_ld_weight = 0, prior_variance = 50
, estimate_prior_variance = TRUE,  : 
  argument "z" is missing, with no default

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "scripts/polyfun/finemapper.py", line 1278, in <module>
    df_finemap = finemap_obj.finemap(locus_start=args.start, locus_end=args.end, num_causal_snps=args.max_
num_causal,
  File "scripts/polyfun/finemapper.py", line 862, in finemap
    susie_obj = self.susieR.susie_bhat(
AttributeError: module 'susieR' has no attribute 'susie_bhat'

Not sure if this is related, but I also noticed that despite calling from within a conda env, the log file suggests that the script search for shared R library path (I used this in a university cluster with shared env) so I wonder if this somehow messed up the susieR installation in my local conda env. The log file header was:

[INFO]  Loading sumstats file...                                                                          
[INFO]  Loaded sumstats for 303081 SNPs in 12.15 seconds                                                  
[INFO]  cffi mode is CFFI_MODE.ANY                                                                        
[DEBUG]  Looking for R home with: R RHOME                                                                 
[INFO]  R home found: /home/rmhiaah/anaconda3/envs/polyfun/lib/R                                          
[DEBUG]  Looking for LD_LIBRARY_PATH with: /home/rmhiaah/anaconda3/envs/polyfun/lib/R/bin/Rscript -e cat(S
ys.getenv("LD_LIBRARY_PATH"))                                                                             
[INFO]  R library path: /shared/ucl/apps/openmpi/4.0.5/gnu-10.2.0/lib:/shared/ucl/apps/gcc/10.2.0-p95889/l
ib64:/shared/ucl/apps/gcc/10.2.0-p95889/lib:/shared/ucl/apps/jq/1.5/gnu-4.9.2/lib:/shared/ucl/apps/protobu
f/3.17.3/gnu-10.2.0/lib:/shared/ucl/apps/v8/3.15/v8/out/x64.release/lib.target:/shared/ucl/apps/gmt/6.2.0/
gnu-10.2.0/lib64:/shared/ucl/apps/geos/3.9.1/gnu-10.2.0/lib:/shared/ucl/apps/gdal/3.3.2_proj-8/gnu-10.2.0/
lib:/shared/ucl/apps/PROJ.4//8.1.1/lib:/shared/ucl/apps/SQLite/3360000/lib:/shared/ucl/apps/python/bundles
/gnu-10.2.0/python39-6.0.0/venv/lib64:/shared/ucl/apps/python/bundles/gnu-10.2.0/python39-6.0.0/venv/lib:/
shared/ucl/apps/python/3.9.6/gnu-10.2.0/lib:/shared/ucl/apps/graphicsmagick/1.3.21/gnu-4.9.2/lib:/shared/u
cl/apps/freetype/2.8.1/gnu-4.9.2/lib:/shared/ucl/apps/libtool/2.4.6/lib:/shared/ucl/apps/ucx/1.9.0/gnu-10.
2.0/lib:/shared/ucl/apps/binutils/2.36.1/gnu-10.2.0/lib64:/shared/ucl/apps/binutils/2.36.1/gnu-10.2.0/lib:
/shared/ucl/apps/numactl/2.0.12/lib:/shared/ucl/apps/pcre2/10.37/gnu-10.2.0/lib:/shared/ucl/apps/NetCDF/4.
8.1-gnu-10.2.0/lib:/shared/ucl/apps/udunits/2.2.28/gnu-10.2.0/lib:/shared/ucl/apps/HDF/5-1.10.6/serial/gnu
-10.2.0/lib:/shared/ucl/apps/gsl/2.7/gnu-10.2.0/lib:/shared/ucl/apps/Ghostscript/9.19/lib:/shared/ucl/apps
/fftw/3.3.9/gnu-10.2.0/lib:/shared/ucl/apps/java/jdk1.8.0_92/lib:/shared/ucl/apps/openblas/0.3.13-serial/g
nu-10.2.0/lib:/shared/ucl/apps/R/R-4.2.0-OpenBLAS/lib64/R/lib:/shared/ucl/apps/R/R-4.2.0-OpenBLAS/lib64   
[INFO]  LD_LIBRARY_PATH: /shared/ucl/apps/openmpi/4.0.5/gnu-10.2.0/lib:/shared/ucl/apps/gcc/10.2.0-p95889/lib64:/shared/ucl/apps/gcc/10.2.0-p95889/lib:/shared/ucl/apps/jq/1.5/gnu-4.9.2/lib:/shared/ucl/apps/protobuf/3.17.3/gnu-10.2.0/lib:/shared/ucl/apps/v8/3.15/v8/out/x64.release/lib.target:/shared/ucl/apps/gmt/6.2.0/gnu-10.2.0/lib64:/shared/ucl/apps/geos/3.9.1/gnu-10.2.0/lib:/shared/ucl/apps/gdal/3.3.2_proj-8/gnu-10.2.0/lib:/shared/ucl/apps/PROJ.4//8.1.1/lib:/shared/ucl/apps/SQLite/3360000/lib:/shared/ucl/apps/python/bundles/gnu-10.2.0/python39-6.0.0/venv/lib64:/shared/ucl/apps/python/bundles/gnu-10.2.0/python39-6.0.0/venv/lib:/shared/ucl/apps/python/3.9.6/gnu-10.2.0/lib:/shared/ucl/apps/graphicsmagick/1.3.21/gnu-4.9.2/lib:/shared/ucl/apps/freetype/2.8.1/gnu-4.9.2/lib:/shared/ucl/apps/libtool/2.4.6/lib:/shared/ucl/apps/ucx/1.9.0/gnu-10.2.0/lib:/shared/ucl/apps/binutils/2.36.1/gnu-10.2.0/lib64:/shared/ucl/apps/binutils/2.36.1/gnu-10.2.0/lib:/shared/ucl/apps/numactl/2.0.12/lib:/shared/ucl/apps/pcre2/10.37/gnu-10.2.0/lib:/shared/ucl/apps/NetCDF/4.8.1-gnu-10.2.0/lib:/shared/ucl/apps/udunits/2.2.28/gnu-10.2.0/lib:/shared/ucl/apps/HDF/5-1.10.6/serial/gnu-10.2.0/lib:/shared/ucl/apps/gsl/2.7/gnu-10.2.0/lib:/shared/ucl/apps/Ghostscript/9.19/lib:/shared/ucl/apps/fftw/3.3.9/gnu-10.2.0/lib:/shared/ucl/apps/java/jdk1.8.0_92/lib:/shared/ucl/apps/openblas/0.3.13-serial/gnu-10.2.0/lib:/shared/ucl/apps/R/R-4.2.0-OpenBLAS/lib64/R/lib:/shared/ucl/apps/R/R-4.2.0-OpenBLAS/lib64

[alhenry]

Apologies - just realised that actually the error is related to

  File "scripts/polyfun/finemapper.py", line 847, in finemap
    susie_obj = self.susieR.susie_rss(

I wonder if it's better to open a separate issue for this (if hasn't already been reported)?

[jdblischak]

Apologies - just realised that actually the error is related to

Correct. This is the same thing that happened to the original poster in this thread. The message about susie_bhat is expected, since it is only there for backwards compatibility. Essentially, it tries using the old function, and if it's not there, then it uses the modern one. Unfortunately this extraneous (and expected) error is at the bottom of the message. We should probably remove it, since that version of susieR is so outdated at this point (and anyone that wants to use it could always use it with an outdated polyfun).

The correct error is:

 argument "z" is missing, with no default

This suggests to me that all the SNPs in the region are being removed prior to finemapping. Can you check if your GWAS has any SNPs in the region you are trying to finemap? And if there are SNPs present in the GWAS, the next thing to check is if the naming scheme matches between your GWAS and the LD matrix.

If you're comfortable with Python debugging, you could also drop a import pdb; pdb.set_trace() right before the call to self.susieR.susie_rss() so that you can directly inspect the objects that are passed to susieR.

[alhenry]

Hi @jdblischak thanks for getting back. I can confirm that there are SNPs within the region in both GWAS and LD reference files, and that the two have the same SNP ids. I did try the debugging step you recommended but not very familiar with it. Could you elaborate a bit more how can I inspect the objects that are passed to susieR? Thanks for your patience

[jdblischak]

I can confirm that there are SNPs within the region in both GWAS and LD reference files, and that the two have the same SNP ids.

Great! That's a good start

I did try the debugging step you recommended but not very familiar with it. Could you elaborate a bit more how can I inspect the objects that are passed to susieR?

What I had in mind was inserting import pdb; pdb.set_trace() right before the call to susie_suff_stat()

https://github.com/omerwe/polyfun/blob/f70dec66b096216f2360cf21352eafe4110d4455/finemapper.py#L846-L847

Then in the interactive debugger you can inspect the Python objects directly, eg

bhat
L
prior_weights
self.n
print(m) # safer to explicitly print all single-letter variables, since some are pdb commands

However, if you're more comfortable in another language, eg R, there is another option that I had forgotten about. If you pass a directory to --debug-dir, it will automatically export all the objects to files so that you can investigate them

--debug-dir DEBUG_DIR If specified, this is a path of a directory that will include files for debugging problems

https://github.com/omerwe/polyfun/blob/f70dec66b096216f2360cf21352eafe4110d4455/finemapper.py#L807-L826