search

precimed / mixer

Causal Mixture Model for GWAS summary statistics
GNU General Public License v3.0
62 stars 17 forks source link

OSError: libbgmg.so after RuntimeWarning #66

Closed tobiaswolfram closed 1 year ago

tobiaswolfram commented 1 year ago

Hi, really cool tool! I worked through the tutorial and everything was running fine, but now I tried my own GWAS and after successfully debugging a few issues myself, I eventually failed. When running test1 after a succesful run of fit, I get this log:


INFO:root:__init__(lib_name=src/build/lib/libbgmg.so, context_id=0)
INFO:root:init_log(femshare.test.rep1.log)
INFO:root:log_message(***********************************************************************
* mixer.py: Univariate and Bivariate Causal Mixture for GWAS
* Version 1.2.0
* (c) 2016-2020 Oleksandr Frei, Alexey A. Shadrin, Dominic Holland
* Norwegian Centre for Mental Disorders Research / University of Oslo
* Center for Multimodal Imaging and Genetics / UCSD
* GNU General Public License v3
***********************************************************************
Call: 
./mixer.py test1 \
    --out femshare.test.rep1 \
    --lib src/build/lib/libbgmg.so \
    --bim-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.bim \
    --ld-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.run4.ld \
    --trait1-file femshare.csv.gz \
    --load-params-file femshare.fit.rep1.json 
)
INFO:root:__init__(lib_name=src/build/lib/libbgmg.so, context_id=0)
INFO:root:log_message(--fit-sequence: ['load', 'inflation']...)
INFO:root:log_message(fit_type==load...)
INFO:root:log_message(fit_type==load: Done, UnivariateParams(_pi: 0.003964433474557695, _sig2_beta: 5.936502354294048e-06, _sig2_zero: 1.050428396730295))
INFO:root:log_message(fit_type==load done (UnivariateParams(_pi: 0.003964433474557695, _sig2_beta: 5.936502354294048e-06, _sig2_zero: 1.050428396730295), {'params': {'pi': 0.003964433474557695, 'sig2_beta': 5.936502354294048e-06, 'sig2_zero': 1.050428396730295}}))
INFO:root:log_message(fit_type==inflation...)
INFO:root:log_message(fit_type==inflation done (UnivariateParams(_pi: 0.003964433474557695, _sig2_beta: 5.936502354294048e-06, _sig2_zero: 1.0352772554264185),            AIC: 743816.169366054
           BIC: 743826.6273385806
          cost: 371907.084683027
       cost_df: 1
     cost_fast: 371907.08468302735
        cost_n: 257293.4
 final_simplex: (array([[0.03466927],
       [0.03466912]]), array([371907.08468303, 371907.0847105 ]))
           fun: 371907.084683027
       message: 'Optimization terminated successfully.'
          nfev: 45
           nit: 19
        params: {'pi': 0.003964433474557695, 'sig2_beta': 5.936502354294048e-06, 'sig2_zero': 1.0352772554264185}
        status: 0
       success: True
             x: array([0.03466927])))
INFO:root:log_message(Calculate AIC/BIC w.r.t. infinitesimal model (fast cost function)...)
INFO:root:log_message(fit_type==infinitesimal...)
INFO:root:log_message(fit_type==infinitesimal done (UnivariateParams(_pi: 1.0, _sig2_beta: 2.7956413992854812e-08, _sig2_zero: 1.031896260367893),            AIC: 743818.1471324271
           BIC: 743839.0630774802
          cost: 371907.07356621354
       cost_df: 2
     cost_fast: 371907.07356621366
        cost_n: 257293.4
 final_simplex: (array([[-17.39261918,   0.03139814],
       [-17.39261919,   0.03139814],
       [-17.39261936,   0.03139815]]), array([371907.07356621, 371907.07356621, 371907.07358108]))
           fun: 371907.07356621354
       message: 'Optimization terminated successfully.'
          nfev: 105
           nit: 45
        params: {'pi': 1.0, 'sig2_beta': 2.7956413992854812e-08, 'sig2_zero': 1.031896260367893}
        status: 0
       success: True
             x: array([-17.39261918,   0.03139814])))
/data/mixer/precimed/mixer/utils.py:894: RuntimeWarning: divide by zero encountered in log10
  data_x=-np.log10(np.flip(np.cumsum(np.flip(data_weights[si]))))     # step 2
/home/twolfram@ensae.fr/.local/lib/python3.8/site-packages/scipy/interpolate/_interpolate.py:651: RuntimeWarning: invalid value encountered in subtract
  slope = (y_hi - y_lo) / (x_hi - x_lo)[:, None]
INFO:root:log_message(Done)
/usr/bin/python3: can't find '__main__' module in 'precimed'
run_mixer_script.sh: 42: --trait1-file: not found
run_mixer_script.sh: 43: --out: not found
run_mixer_script.sh: 44: --extract: not found
run_mixer_script.sh: 45: --bim-file: not found
run_mixer_script.sh: 46: --ld-file: not found
run_mixer_script.sh: 47: --lib: not found
Traceback (most recent call last):
  File "precimed/mixer.py", line 18, in <module>
    libbgmg = LibBgmg(args.lib)
  File "/data/mixer/precimed/mixer/libbgmg.py", line 28, in __init__
    self.cdll, self._lib_name = self._load_cdll(lib_name)
  File "/data/mixer/precimed/mixer/libbgmg.py", line 464, in _load_cdll
    raise OSError(exception_message)
OSError: libbgmg.so: cannot open shared object file: No such file or directory
Failed to load BGMG shared library from `['libbgmg.so', 'libbgmg.so']`. Try to add the location of `libbgmg.so` file into your PATH system variable, or to set BGMG_SHARED_LIBRARY - the specific system variable which may point to `libbgmg.so` file, including the full path.
run_mixer_script.sh: 50: --trait1-file: not found
run_mixer_script.sh: 51: --load-params-file: not found
run_mixer_script.sh: 52: --out: not found
run_mixer_script.sh: 53: --bim-file: not found
run_mixer_script.sh: 54: --ld-file: not found
run_mixer_script.sh: 55: --lib: not found
run_mixer_script.sh: 57: Fit: not found
python3: can't open file 'python/mixer.py': [Errno 2] No such file or directory
run_mixer_script.sh: 59: --trait1-file: not found
run_mixer_script.sh: 60: --trait2-file: not found
run_mixer_script.sh: 61: --trait1-params-file: not found
run_mixer_script.sh: 62: --trait2-params-file: not found
run_mixer_script.sh: 63: --out: not found
run_mixer_script.sh: 64: --extract: not found
run_mixer_script.sh: 65: --bim-file: not found
run_mixer_script.sh: 66: --ld-file: not found
run_mixer_script.sh: 67: --lib: not found
Traceback (most recent call last):
  File "precimed/mixer.py", line 18, in <module>
    libbgmg = LibBgmg(args.lib)
  File "/data/mixer/precimed/mixer/libbgmg.py", line 28, in __init__
    self.cdll, self._lib_name = self._load_cdll(lib_name)
  File "/data/mixer/precimed/mixer/libbgmg.py", line 464, in _load_cdll
    raise OSError(exception_message)
OSError: libbgmg.so: cannot open shared object file: No such file or directory
Failed to load BGMG shared library from `['libbgmg.so', 'libbgmg.so']`. Try to add the location of `libbgmg.so` file into your PATH system variable, or to set BGMG_SHARED_LIBRARY - the specific system variable which may point to `libbgmg.so` file, including the full path.
run_mixer_script.sh: 70: --trait1-file: not found
run_mixer_script.sh: 71: --trait2-file: not found
run_mixer_script.sh: 72: --trait1-params-file: not found
run_mixer_script.sh: 73: --trait2-params-file: not found
run_mixer_script.sh: 74: --out: not found
run_mixer_script.sh: 75: --bim-file: not found
run_mixer_script.sh: 76: --ld-file: not found
run_mixer_script.sh: 77: --lib: not found
python3: can't open file 'python/mixer.py': [Errno 2] No such file or directory
run_mixer_script.sh: 80: --trait1-file: not found
run_mixer_script.sh: 81: --trait2-file: not found
run_mixer_script.sh: 82: --load-params-file: not found
run_mixer_script.sh: 83: --out: not found
run_mixer_script.sh: 84: --bim-file: not found
run_mixer_script.sh: 85: --ld-file: not found
run_mixer_script.sh: 86: --lib: not found

Not quite sure what is happening here, but libbgmg.so is definitely present. I did all the checks you proposed in an earlier issue and nothing out of the ordinary emerged. Thank you for your help!

ofrei commented 1 year ago

First indication that something isn't right is /usr/bin/python3: can't find '__main__' module in 'precimed'. Could you paste the entire script you're executing?

tobiaswolfram commented 1 year ago

Thank you for the fast reply! This is my script. Basically I just tried to write a shell-script around your tutorial in the most idiotic way possible.

!/bin/bash

TRAIT_1=$1 TRAIT_2=$2 N_1=$3 N_2=$4

python3 sumstats.py csv --sumstats ${TRAIT_1} --out ${TRAIT_1}1.csv --force --auto --head 5 --n-val ${N_1} python3 sumstats.py zscore --sumstats ${TRAIT_1}1.csv --out ${TRAIT_1}2.csv --force python3 sumstats.py qc --sumstats ${TRAIT_1}2.csv --exclude-ranges 6:26000000-34000000 --max-or 1e37 --out ${TRAIT_1}.csv --force gzip ${TRAIT_1}.csv

python3 sumstats.py csv --sumstats ${TRAIT_2} --out ${TRAIT_2}1.csv --force --auto --head 5 --n-val ${N_2} python3 sumstats.py zscore --sumstats ${TRAIT_2}1.csv --out ${TRAIT_2}2.csv --force python3 sumstats.py qc --sumstats ${TRAIT_2}2.csv --exclude-ranges 6:26000000-34000000 --max-or 1e37 --out ${TRAIT_2}.csv --force gzip ${TRAIT_2}.csv

for i in $(seq 1 20); do SLURM_ARRAY_TASK_ID=$i python3 precimed/mixer.py fit1 \ --trait1-file ${TRAIT_1}.csv.gz \ --out ${TRAIT_1}.fit.rep${SLURM_ARRAY_TASK_ID} \ --extract 1000G_EUR_Phase3_plink/1000G.EUR.QC.prune_maf0p05_rand2M_r2p8.rep${SLURM_ARRAY_TASK_ID}.snps \ --bim-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.bim \ --ld-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.run4.ld \ --lib src/build/lib/libbgmg.so

python3 precimed/mixer.py test1 \
--trait1-file ${TRAIT_1}.csv.gz \
--load-params-file ${TRAIT_1}.fit.rep${SLURM_ARRAY_TASK_ID}.json \
--out ${TRAIT_1}.test.rep${SLURM_ARRAY_TASK_ID} \
--bim-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.bim \
--ld-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.run4.ld \
--lib src/build/lib/libbgmg.so

python3 precimed /mixer.py fit1
--trait1-file ${TRAIT_2}.csv.gz
--out ${TRAIT_2}.fit.rep${SLURM_ARRAY_TASK_ID}
--extract 1000G_EUR_Phase3_plink/1000G.EUR.QC.prune_maf0p05_rand2M_r2p8.rep${SLURM_ARRAY_TASK_ID}.snps
--bim-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.bim
--ld-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.run4.ld
--lib src/build/lib/libbgmg.so

python3 precimed/mixer.py test1
--trait1-file ${TRAIT_2}.csv.gz
--load-params-file ${TRAIT_2}.fit.rep${SLURM_ARRAY_TASK_ID}.json
--out ${TRAIT_2}.test.rep${SLURM_ARRAY_TASK_ID}
--bim-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.bim
--ld-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.run4.ld
--lib src/build/lib/libbgmg.so

python3 python/mixer.py fit2
--trait1-file ${TRAIT_1}.csv.gz
--trait2-file ${TRAIT_2}.csv.gz
--trait1-params-file ${TRAIT_1}.fit.rep${SLURM_ARRAY_TASK_ID}.json
--trait2-params-file ${TRAIT_2}.fit.rep${SLURM_ARRAY_TASK_ID}.json
--out ${TRAIT_1}vs${TRAIT_2}.fit.rep${SLURM_ARRAY_TASK_ID}
--extract 1000G_EUR_Phase3_plink/1000G.EUR.QC.prune_maf0p05_rand2M_r2p8.rep${SLURM_ARRAY_TASK_ID}.snps
--bim-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.bim
--ld-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.run4.ld
--lib src/build/lib/libbgmg.so

python3 precimed/mixer.py test2
--trait1-file ${TRAIT_1}.csv.gz
--trait2-file ${TRAIT_2}.csv.gz
--trait1-params-file ${TRAIT_1}.fit.rep${SLURM_ARRAY_TASK_ID}.json
--trait2-params-file ${TRAIT_2}.fit.rep${SLURM_ARRAY_TASK_ID}.json
--out ${TRAIT_1}vs${TRAIT_2}.fit.rep${SLURM_ARRAY_TASK_ID}
--bim-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.bim
--ld-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.run4.ld
--lib src/build/lib/libbgmg.so

python3 python/mixer.py test2
--trait1-file ${TRAIT_1}.csv.gz
--trait2-file ${TRAIT_2}.csv.gz
--load-params-file ${TRAIT_1}vs${TRAIT_2}.fit.rep${SLURM_ARRAY_TASK_ID}.json
--out ${TRAIT_1}vs${TRAIT_2}.test.rep${SLURM_ARRAY_TASK_ID}
--bim-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.bim
--ld-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.run4.ld
--lib src/build/lib/libbgmg.so

done

python precimed/mixer_figures.py combine --json ${TRAIT_1}vs${TRAIT_2}.fit.rep@.json --out ${TRAIT_1}vs${TRAIT_2}.fit python precimed/mixer_figures.py combine --json ${TRAIT_1}vs${TRAIT_2}.test.rep@.json --out ${TRAIT_1}vs${TRAIT_2}.test python precimed/mixer_figures.py two --json-fit ${TRAIT_1}vs${TRAIT_2}.fit.json --json-test ${TRAIT_1}vs${TRAIT_2}.test.json --out ${TRAIT_1}vs${TRAIT_2}_out --statistic mean std

ofrei commented 1 year ago

Space after precimed ?

  # Fit and test the second trait
python3 precimed /mixer.py fit1
tobiaswolfram commented 1 year ago

Gosh, you are probably right. Let me try it.

tobiaswolfram commented 1 year ago

No, unfortunately the problem still occurs. It seems like it has to be something that happens inside the first call to test1.

ofrei commented 1 year ago

ok, but then it's likely to be a missing backslash after fit1. python3 precimed /mixer.py fit1 --trait1-file ${TRAIT_2}.csv.gz

tobiaswolfram commented 1 year ago

For some reason it was not that as well. Even after fixing the backslash issues (sorry for that), it still fails after it has already started to run test1.

ofrei commented 1 year ago

Ok, then please execute one command at a time and let me know the exact one that fails, and what error message it produces

tobiaswolfram commented 1 year ago

Well, it seems like it runs through if I execute it step by step (which it did not before), so I suppose your solution indeed fixed it and it was truly just my lack of ability to properly code a loop that caused this commotion. Thank you so much for your help!

ofrei commented 1 year ago

No worries, good it's solved!