Binary logistic regression no longer converging in step 1 v2.2

[mcolosimo-p4]

Hi,

I built v2.2 (15b858d3fb) and ran our internal tests. It is now failing to converge in step 1 using our binary phenotypes. It is the same data and parameters so I'm a little confused what is different.

v2.2 run

              |===========================|
              |      REGENIE v2.2.gz      |
              |===========================|

Copyright (c) 2020-2021 Joelle Mbatchou, Andrey Ziyatdinov and Jonathan Marchini.
Distributed under the MIT License.
Compiled with Boost Iostream library.
Using Intel MKL with Eigen.

Log of output saved in file : /docker/fit_bin_out.log

Options in effect:
  --step 1 \
  --bed /pruned_variants/ref \
  --nauto 22 \
  --covarFile /docker/covariates.txt \
  --phenoFile /docker/phenotypes.txt \
  --bsize 1000 \
  --extract /docker/reg_step1_snplist.txt \
  --out /docker/fit_bin_out \
  --bt \
  --firth \
  --approx 

Fitting null model
 * bim              : [/pruned_variants/ref.bim] n_snps = 410000
   -keeping variants specified by --extract
     +number of variants remaining in the analysis = 61634
 * fam              : [/pruned_variants/ref.fam] n_samples = 408183
 * bed              : [/pruned_variants/ref.bed]
 * phenotypes       : [/docker/phenotypes.txt] n_pheno = 2
   -number of phenotyped individuals  = 337147
 * covariates       : [/docker/covariates.txt] n_cov = 12
   -number of individuals with covariate data = 337147
 * number of individuals used in analysis = 337147
   -fitting null logistic regression on binary phenotypes...done (469ms) 
   -residualizing and scaling phenotypes...done (5ms) 
 * # threads        : [95]
 * block size       : [1000]
 * # blocks         : [71] for 61634 variants
 * # CV folds       : [5]
 * ridge data_l0    : [5 : 0.01 0.25 0.5 0.75 0.99 ]
 * ridge data_l1    : [5 : 0.01 0.25 0.5 0.75 0.99 ]
 * approximate memory usage : 6GB
 * setting memory...done

  Level 1 ridge with logistic regression...
   -on phenotype 1 (angina_i20)...WARNING: Penalized logistic regression did not converge! (Increase --niter)
done (395478ms) 
   -on phenotype 2 (high_cholest_e78)...WARNING: Penalized logistic regression did not converge! (Increase --niter)
done (378967ms) 

v2.0.1 run

              |=============================|
              |      REGENIE v2.0.1.gz      |
              |=============================|

Copyright (c) 2020 Joelle Mbatchou and Jonathan Marchini.
Distributed under the MIT License.
Compiled with Boost Iostream library.
Using Intel MKL with Eigen.

Log of output saved in file : /docker/fit_bin_out.log

Options in effect:
  --step 1 \
  --bed /pruned_variants/ref \
  --nauto 22 \
  --covarFile /docker/covariates.txt \
  --phenoFile /docker/phenotypes.txt \
  --bsize 1000 \
  --extract /docker/reg_step1_snplist.txt \
  --out /docker/fit_bin_out \
  --bt \
  --firth \
  --approx 

Fitting null model
 * bim              : [/pruned_variants/ref.bim] n_snps = 410000
   -keeping only variants specified in [/docker/reg_step1_snplist.txt]
     +number of variants remaining in the analysis = 61634
 * fam              : [/pruned_variants/ref.fam] n_samples = 408183
 * bed              : [/pruned_variants/ref.bed]
 * phenotypes       : [/docker/phenotypes.txt] n_pheno = 2
   -number of phenotyped individuals = 337147
 * covariates       : [/docker/covariates.txt] n_cov = 12
   -number of individuals with covariate data = 337147
 * number of individuals used in analysis = 337147
   -fitting null logistic regression on binary phenotypes...done (589ms) 
   -residualizing and scaling phenotypes...done (6ms) 
 * # threads        : [95]
 * block size       : [1000]
 * # blocks         : [71]
 * # CV folds       : [5]
 * ridge data_l0    : [5 : 0.01 0.25 0.5 0.75 0.99 ]
 * ridge data_l1    : [5 : 0.01 0.25 0.5 0.75 0.99 ]
 * approximate memory usage : 6GB
 * setting memory...done

Level 1 ridge with logistic regression...
   -on phenotype 1 (angina_i20)...done (58471ms) 
   -on phenotype 2 (high_cholest_e78)...done (68490ms) 

Output
------
phenotype 1 (angina_i20) : 
  0.01  : Rsq = 0.0313038, MSE = 0.0461434, -logLik/N = 0.183758
  0.25  : Rsq = 0.031368, MSE = 0.046143, -logLik/N = 0.183734
  0.5   : Rsq = 0.0313778, MSE = 0.0461434, -logLik/N = 0.183734
  0.75  : Rsq = 0.0314275, MSE = 0.0461418, -logLik/N = 0.183722
  0.99  : Rsq = 0.0316216, MSE = 0.0461356, -logLik/N = 0.183683<- min value
  * making predictions...writing LOCO predictions...done (8399ms) 

phenotype 2 (high_cholest_e78) : 
  0.01  : Rsq = 0.04933, MSE = 0.0886845, -logLik/N = 0.308851
  0.25  : Rsq = 0.0494913, MSE = 0.0886725, -logLik/N = 0.308801
  0.5   : Rsq = 0.0495492, MSE = 0.0886677, -logLik/N = 0.308783
  0.75  : Rsq = 0.0496161, MSE = 0.0886622, -logLik/N = 0.30876
  0.99  : Rsq = 0.0497057, MSE = 0.0886571, -logLik/N = 0.308732<- min value
  * making predictions...writing LOCO predictions...done (7114ms) 

[joellembatchou]

Thanks for catching this bug. Should be fixed with 3068fd10066329e13ff594a461ad71d05267fd5e.

Cheers, Joelle

[mcolosimo-p4]

Hi Joelle,

Thanks for the quick response. I did a clean rebuild using the new commit , retested, and it is still failing to converge.

Best, Marc

[DDochtermann]

Having the same issue as well on the newest commit. Regression models that converged in versions 1.0.6.7 and 2.0.2 did not converge on 2.2 for me, using the same options and inputs.

[joellembatchou]

Still checking on this issue as I can't seem to replicate it (the code for ridge logistic regression with K-fold CV was not changed between v2.0.2 and v2.2). Which OS are you using?

[mcolosimo-p4]

I'm using the docker build (well specifically a docker container that I converted to a singularity container) running under CentOS 7.

I've been trying to build it natively again (not this issue, I probably should start another issue for this), but I'm stuck at the last step:

/usr/bin/ld: cannot find -lboost_iostreams
/usr/bin/ld: cannot find -lbgen
/usr/bin/ld: cannot find -lzstd
/usr/bin/ld: cannot find -ldb
/usr/bin/ld: cannot find -lsqlite3
/usr/bin/ld: cannot find -lboost
/usr/bin/ld: cannot find -lz
/usr/bin/ld: cannot find -lstdc++

I had it build bgen and iostreams is installed:

/usr/lib64/libboost_iostreams-mt.so.1.53.0 /usr/lib64/libboost_iostreams.so.1.53.0 /usr/lib64/libboost_iostreams-mt.so /usr/lib64/libboost_iostreams.so

I did add -L/usr/lib64/ and still fails to find it.

Marc

[freeseek]

I am having the same issue with version v2.2

[came203]

Same for me with version v2.2, worked in all other versions before

[Shicheng-Guo]

okay. I meet the same issue. .ERROR: logistic regression did not converge for phenotype X254. Perhaps increase --niter?

phecode X612.3, X254 are failed while others are okay.

REGENIE v2.2.2 and 70b8450

  WARNING: Fitted probabilities numerically 0/1 occured (phenotype #X401.3).
     WARNING: Fitted probabilities numerically 0/1 occured (phenotype #X426.4).
     WARNING: Fitted probabilities numerically 0/1 occured (phenotype #X446.6).
     WARNING: Fitted probabilities numerically 0/1 occured (phenotype #X457).
     WARNING: Fitted probabilities numerically 0/1 occured (phenotype #X513.3).
     WARNING: Fitted probabilities numerically 0/1 occured (phenotype #X520).
     WARNING: Fitted probabilities numerically 0/1 occured (phenotype #X520.2).
     WARNING: Fitted probabilities numerically 0/1 occured (phenotype #X535.9).
     WARNING: Fitted probabilities numerically 0/1 occured (phenotype #X536).
     WARNING: Fitted probabilities numerically 0/1 occured (phenotype #X536.8).
     WARNING: Fitted probabilities numerically 0/1 occured (phenotype #X560.2).
     WARNING: Fitted probabilities numerically 0/1 occured (phenotype #X586.12).
     WARNING: Fitted probabilities numerically 0/1 occured (phenotype #X588).
     WARNING: Fitted probabilities numerically 0/1 occured (phenotype #X592.11).
     WARNING: Fitted probabilities numerically 0/1 occured (phenotype #X592.2).
     WARNING: Fitted probabilities numerically 0/1 occured (phenotype #X592.3).ERROR: logistic regression did not converge for phenotype X612.3. Perhaps increase --niter?
(END)

[ggstatgen]

Also having this issue - any updates on what might be going on? Eg

-number of individuals with covariate data = 250867
 * number of individuals used in analysis = 250860
 * case-control counts for each trait:
- 'Alzheimers_disease': 433 cases and 250427 controls

...

 Level 1 ridge with logistic regression...
   -on phenotype 1 (Alzheimers_disease)...WARNING: Penalized logistic regression did not converge! (Increase --niter)
done (23409355ms)

As shown, it's taking ages to try to converge and then fails.

I'm using

--niter 50

and the statically linked binary as follows

regenie_v2.2.2.gz_x86_64_Centos7_mkl

[Shicheng-Guo]

Hi ggstagen

use old version, for example 2.0.2 will not have this issue. So try to use old version.

Start time: Fri Aug 13 16:40:15 2021

              |============================|
              |        REGENIE v2.0.2      |
              |============================|

Copyright (c) 2020 Joelle Mbatchou and Jonathan Marchini.
Distributed under the MIT License.

Options in effect:

[Shicheng-Guo]

Just test 2.2.3, same issue is occurred. looks only 2.0.2 works well.

Start time: Fri Aug 13 17:42:27 2021

              |=============================|
              |      REGENIE v2.2.3.gz      |
              |=============================|

Copyright (c) 2020-2021 Joelle Mbatchou, Andrey Ziyatdinov and Jonathan Marchini.
Distributed under the MIT License.
Compiled with Boost Iostream library.
Using Intel MKL with Eigen.

     WARNING: Fitted probabilities numerically 0/1 occured (phenotype #X643.1).ERROR: logistic regression did not converge for phenotype X646. Perhaps increase --niter?

[jsture]

I had the same issue with regenie_v2.2.3.gz_x86_64_Centos7_mkl

Running on UKB data

--step 1 \ --bed ../ukb_cal/ukb_cal_allChrs \ --extract ../ukb_cal/qc_pass.snplist \ --keep ../ukb_cal/qc_pass.id \ --phenoFile phenos.final.tsv \ --phenoColList diabetes,etc \ --covarFile covariates.txt \ --bt \ --bsize 1000 \ --lowmem \ --lowmem-prefix regenie_tmp_preds \ --out ukb_step1_BT \ --verbose

Hangs on Level 1 for at least some hours. Pretty balanced trait, 13023/145969.

Level 1 ridge with logistic regression... -on phenotype 1 (diabetes)...

16 cores, 16G mem

Best,

Jakob

[krebs-kristi]

Hi.

I'm having same issue with several different phenotypes using regenie_v2.2.3. Are there any news or solutions for this problem? Is this problem fixed in version 2.2.4? Thanks.

Best wishes, Kristi

[jsture]

2.2.4 fixed it. Thanks a bunch!

[mcolosimo-p4]

Confirmed that 2.2.4 also fixed it for me. However, step 2 results are now slightly different, but that is for another issue.