Open DesmondCampbell opened 7 years ago
Hello again,
I've just found out one of the solutions I suggest stops it crashing. That is, I limit the number of steps taken by lar() to < p For instance, the following line completes successfully lfit <- NA; lfit = lar( mX, vY, intercept=F, normalize=F, maxsteps=100 )
The glmnet() option dfmax may similarly curtail the steps taken. I'm not sure.
regards Desmond
Hello,
I think I have found a bug in the larInf() function of the selectiveInference R package
I think whenever p > n, i.e the number of regressors exceeds the number of rows of data, a singular matrix is generated causeing the function to crash.
My Dataset consists of matrix 360 rows of 777 columns (regressors), plus a response vector (360 long) There are no missing values. All variables have been mean centred. Regressors have been scaled to sd =1.
I fitted a Least Angle Regression via lar() I then ran larInf() with verbose set to TRUE.
The function crashed while processing the 360th variable, Note 360 = number of rows of data.
After this happenned the first time, I dropped one of each pair of some highly correlated regressors (remaining max pearson corr < 0.95). So I don't think it is colinearity.
I would appreciate you updating me about this issue. Also in the interim have any workarounds - LAR, LASSO preferred. For instance perhaps I can only run the model for first 50 steps.
My email is - desmond.campbell@glasgow.ac.uk
Below is R console output fom a crash of larInf() Feel free to contact me.
regards Desmond
fit the lar model
estimate sigma
attempt inference - function crashes
Enter a frame number, or 0 to exit
1: larInf(lfit, sigma = sigmaEst$sigmahat, verbose = T) 2: covtest.pval(obj, sigma, j) 3: solve(crossprod(x[, c(A, j), drop = F]), c(sA, sj)) 4: solve.default(crossprod(x[, c(A, j), drop = F]), c(sA, sj))
Enter an item from the menu, or 0 to exit Selection: 2 Called from: covtest.pval(obj, sigma, j) Browse[1]> str(obj) List of 19 $ lambda : num [1:360] 802 706 666 620 610 ... $ action : num [1:360] 4 3 326 257 9 514 761 615 128 261 ... $ sign : num [1:360] 1 1 1 -1 1 -1 1 -1 1 1 ... $ df : num [1:360] 0 1 2 3 4 5 6 7 8 9 ... $ beta : num [1:777, 1:360] 0 0 0 0 0 0 0 0 0 0 ... ..- attr(, "dimnames")=List of 2 .. ..$ : NULL .. ..$ : chr [1:360] "801.501" "706.463" "666.388" "619.644" ... $ completepath: logi TRUE $ bls : num [1:777] 7.02 0 -41.17 50.22 0 ... $ Gamma : num [1:430199, 1:360] -1.24 -1.34 -2.24 -1.91 -2.02 ... $ nk : num [1:360] 1553 3105 4655 6203 7749 ... $ vreg : num [1:360, 1:360] -3.41e-03 -2.25e-03 3.01e-05 9.73e-04 6.04e-04 ... $ mp : num [1:360] 706 666 620 610 561 ... $ x : num [1:360, 1:777] -0.0126 0.7447 1.3939 0.5283 0.2038 ... ..- attr(, "dimnames")=List of 2 .. ..$ : NULL .. ..$ : chr [1:777] "age" "bmi" "iSexMale" "iRaceWhite" ... $ y : num [1:360] -12.22 11.78 -4.22 -2.22 -3.22 ... $ bx : num [1:777] 0 0 0 0 0 0 0 0 0 0 ... $ by : num 0 $ sx : num [1:777] 1 1 1 1 1 1 1 1 1 1 ... $ intercept : logi FALSE $ normalize : logi FALSE $ call : language lar(x = mX, y = vY, intercept = F, normalize = F)
Enter a frame number, or 0 to exit
1: larInf(lfit, sigma = sigmaEst$sigmahat, verbose = T) 2: covtest.pval(obj, sigma, j) 3: solve(crossprod(x[, c(A, j), drop = F]), c(sA, sj)) 4: solve.default(crossprod(x[, c(A, j), drop = F]), c(sA, sj))
Selection: 4 Browse[4]> str(A) Error during wrapup: object 'A' not found Browse[4]> str(sA) Error during wrapup: object 'sA' not found Browse[4]> sA Error during wrapup: object 'sA' not found Browse[4]> sj Error during wrapup: object 'sj' not found Browse[4]>
Enter a frame number, or 0 to exit
1: larInf(lfit, sigma = sigmaEst$sigmahat, verbose = T) 2: covtest.pval(obj, sigma, j) 3: solve(crossprod(x[, c(A, j), drop = F]), c(sA, sj)) 4: solve.default(crossprod(x[, c(A, j), drop = F]), c(sA, sj))
Selection: 3 Called from: solve(crossprod(x[, c(A, j), drop = F]), c(sA, sj)) Browse[3]> x [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [1,] -0.135924521 -0.53959038 1.19214893 -0.01421069 -1.041016541 0.326034267 0.26500638 -0.960023700 -0.51561099 -1.20493243 [2,] -0.040796970 -1.50080114 -0.12313786 0.13012750 -0.431907053 -0.398122516 1.38694989 -0.751053458 0.39568325 -2.07117872 [3,] 1.010539007 0.26789815 -0.57416484 0.18401390 -0.777816318 -0.395627017 -0.29149014 -0.936544143 -0.78301461 1.46689919 [4,] -0.158262445 0.03422677 -1.49031686 0.14435823 0.373681784 -1.159683959 -1.83027672 -1.493705888 0.88538861 -1.81137952 [5,] -2.156637503 -0.99527307 2.36382893 -0.58794149 0.312564581 -1.852945215 -1.62766978 -0.313418720 -0.38212891 -0.74986617 [6,] 0.498646835 -0.77030057 -2.95103621 0.38704784 0.071228643 -0.005530756 -0.01630346 -1.384058435 0.05333596 -0.43174933 [7,] -0.755244306 -0.57808092 0.16527818 -1.90768563 0.678440355 1.461921770 1.00335522 0.166683520 -0.55154057 -0.48930007 [8,] 0.778585191 -0.93296826 -0.83427349 0.54545178 -2.439386238 -1.279680283 1.09715982 0.394343957 -1.20007454 1.11291377 [9,] 0.754577954 -2.17699470 0.49191538 0.38998743 -0.309635615 0.333371417 -0.66300952 -1.513257952 0.32198376 1.64118771 [10,] -1.099545613 -0.51411027 -0.31836686 -0.93410691 0.679744123 -0.086414090 0.96810912 2.341575314 1.65470449 1.92244707 [11,] 0.167349961 0.08911827 2.00514227 -1.58162900 1.014932965 1.045619649 -1.68767450 -1.317586126 0.43220392 -0.16430612 [12,] -0.029283481 0.29449012 -0.42367291 0.08779191 1.538640017 1.207992289 -0.46206989 -0.482461635 -1.14869873 -0.69258761 [13,] 1.875845028 0.69219193 -0.21993684 0.61936957 -1.766977094 0.481993148 -0.75647305 1.164762118 0.84888363 -0.49852718 [14,] 0.244634467 -1.61165588 1.04970016 1.25773169 2.289596615 0.017207862 -0.36149330 -2.220113051 -2.09969534 0.75007988 [15,] 0.702153686 0.18790080 -0.13176235 -0.07475668 -3.031246490 0.195700113 -0.12897901 -0.760771901 0.63486366 0.59624472 [16,] -0.015085665 0.07739601 0.34877761 -1.37696708 1.697693381 -0.532431684 -1.90806483 0.113902749 1.33088703 -0.19282119 [17,] -0.143370080 -0.17381374 -0.33805463 2.58334531 1.716225099 -1.747889162 -1.33484260 0.307544658 1.31283814 -0.25875486 [18,] 0.320666578 -1.39938990 0.90388559 -0.34682734 0.632786360 0.776295312 -1.97481963 1.695464227 -0.21660778 -0.49068080 [19,] 0.122334080 -0.36917683 0.84678400 -0.68131382 -0.617648033 -1.011039297 -0.57179915 0.471684001 -0.34040911 0.85024124 [20,] -0.594665192 0.45122022 1.15815376 -0.15466483 0.819283675 -0.170147151 -0.43803850 -0.091859204 0.89581581 -0.37425964 [21,] -0.442394755 1.62599840 2.68699190 -1.05947874 -1.317948105 0.243743597 -1.08256786 1.239914587 -0.21308478 1.14866509 [22,] 0.290503585 1.92329780 0.49434109 -0.13113782 1.581622553 -0.049417265 1.34234670 0.665728403 1.03842731 -1.62377562 [23,] 0.723653379 -0.16308618 -0.06669678 -0.29460770 0.542882170 -0.488798968 -0.18147054 0.648302852 -1.20255449 0.30756340 [24,] 0.459573950 -0.43878404 0.45234730 -1.36093856 -1.124450875 -0.895214637 -0.72844405 0.509580868 -0.34449943 -1.93496164 [25,] 0.185105160 1.19149970 0.58797623 -0.72952160 -0.174150393 0.207755767 0.97336677 -1.271143267 0.40021085 -0.11758417 [26,] 0.233805005 1.66993913 2.35390936 -0.30308914 0.830663022 -0.803728605 0.28198861 -0.382541835 -0.68465177 0.50432419 [27,] 0.593338262 1.13319174 0.97410755 1.28481802 1.050625407 1.816677838 -1.00219170 0.641766036 1.11519590 0.66781248 [28,] 2.001166926 -1.51748309 3.47986354 0.25304566 -0.873195032 1.307126285 -2.17789210 0.809061912 -0.09256017 -0.49002282 [29,] -1.837197758 2.73116619 0.02218368 0.79971121 0.414821577 2.220612436 1.04578863 0.077064865 0.13630547 0.24127207 [30,] -0.862390941 0.49927385 0.22159352 -0.92523637 0.496264134 0.895634975 -1.09656178 -0.987741449 -0.20264900 -0.62741259 [31,] 1.582887396 -1.49192711 -1.10596642 -1.84019303 0.010476067 1.763370740 0.34219906 -0.617517489 0.05698383 -2.11222684 [32,] 0.155200689 0.92173907 -1.09526404 0.08390022 -0.069542105 -0.916548375 -0.91594460 -1.184800346 -0.68427970 -0.19521214 [33,] -0.275019336 -0.51671477 0.63095093 1.56749078 -1.358048321 0.574185163 -0.33869176 0.529320119 0.62316739 0.11255449 [34,] 0.787675758 2.10530277 -0.48174932 1.46943133 -0.718815197 0.414021504 0.53115492 0.620213159 1.49887045 -0.21075028 [35,] -0.222988089 -0.88697108 -0.50296748 -0.65801536 0.398364733 -2.253295882 -0.18876424 0.024531029 -0.81106140 0.05212177 [36,] 1.392023729 -0.16780993 0.44485833 -0.35756506 -1.503152999 -0.495622571 -0.36220336 -0.791666008 2.09897861 0.18689630 [37,] -0.488652910 0.90935102 -0.16208895 0.24534311 -0.006880948 0.117982213 0.26067733 0.349656408 -2.12973949 0.70279824 [38,] 0.137377602 -1.39461339 0.58773983 -1.50030833 0.192743969 0.768607719 0.35868153 0.723573485 -0.62370678 -0.23986249 [39,] 0.003508096 0.81374561 -0.33266209 -0.87705486 -1.718899822 0.371091953 0.09259231 -0.719597432 1.11478469 0.08818384 [40,] -0.726885772 -0.95730687 0.53980555 -0.91290569 -0.519845254 0.527382255 -0.64299329 1.742491823 0.51877834 1.64375486 [41,] -0.721432863 -0.36983033 0.19829531 1.11951450 -0.720731915 0.046013913 0.90831952 1.151706980 0.67539050 0.20732545 [42,] -0.190524592 1.47883722 -0.05916185 -0.74054927 -0.861262991 0.713252252 0.91436715 0.408624974 -0.83930861 0.05546456 [43,] 1.335229412 -0.60414215 1.80066109 0.37009155 0.703867683 1.070627575 -0.66908482 -0.791556143 -1.01869590 -1.50168851 [44,] 0.355995577 -0.29060017 0.24280906 0.35524485 0.483979524 0.941098404 0.72646992 -0.735756883 0.17971169 0.09992694 [45,] 0.842759563 -1.78442144 -1.01239335 1.83961657 0.268628353 -0.603922419 -0.50300216 -1.854904995 1.36773691 -1.20195591 [46,] 0.774532574 2.25806515 -0.03640304 -0.46668736 0.268219343 -0.417661088 -0.22235378 -0.253015529 -0.58629073 0.27760027 [47,] 0.080459477 -0.66697896 1.22664690 0.84007939 -1.814923955 -0.505385186 0.33449550 1.277028155 0.58206897 -0.43474862 [48,] -0.672642017 0.47793931 0.77365964 0.10818225 0.789873389 -0.687525294 0.04592375 -0.157158147 -0.98304011 0.50841786 [49,] 1.835522538 -1.16107581 -0.93221744 -0.94851528 1.134906928 -0.145932374 -1.93770334 0.334439067 0.01211304 0.83270525 [50,] -0.207207497 -0.38027931 0.65490057 -0.07096093 0.760204081 -0.675620176 -1.29488628 -0.008979237 -0.34374124 -0.78807061 Browse[3]> c(A, j) Error during wrapup: object 'A' not found Browse[3]> c(sA, sj) Error during wrapup: object 'sA' not found Browse[3]>
Enter a frame number, or 0 to exit
1: larInf(lfit, sigma = sigmaEst$sigmahat, verbose = T) 2: covtest.pval(obj, sigma, j) 3: solve(crossprod(x[, c(A, j), drop = F]), c(sA, sj)) 4: solve.default(crossprod(x[, c(A, j), drop = F]), c(sA, sj))
Selection: 0
this is the fitted LAR model
my dataset
session info
locale: [1] LC_COLLATE=English_United Kingdom.1252 LC_CTYPE=English_United Kingdom.1252 LC_MONETARY=English_United Kingdom.1252 LC_NUMERIC=C LC_TIME=English_United Kingdom.1252
attached base packages: [1] parallel stats graphics grDevices utils datasets methods base
other attached packages: [1] selectiveInference_1.2.2 intervals_0.15.1 covTest_1.02 MASS_7.3-45 glmpath_0.97 survival_2.40-1 lars_1.2 glmnet_2.0-5 Matrix_1.2-8
[10] visreg_2.3-0 RColorBrewer_1.1-2 tableone_0.8.1 abind_1.4-5 doParallel_1.0.10 iterators_1.0.8 foreach_1.4.3 gplots_3.0.1 tidyr_0.6.1
[19] broom_0.4.1 snowfall_1.84-6.1 snow_0.4-2 dplyr_0.5.0 NMF_0.20.6 Biobase_2.34.0 BiocGenerics_0.20.0 cluster_2.0.5 rngtools_1.2.4
[28] pkgmaker_0.22 registry_0.3 moments_0.14
loaded via a namespace (and not attached): [1] Rcpp_0.12.9 lattice_0.20-34 class_7.3-14 zoo_1.7-14 gtools_3.5.0 assertthat_0.1 digest_0.6.11 psych_1.6.12 gridBase_0.4-7 R6_2.2.0 plyr_1.8.4 survey_3.32-1
[13] e1071_1.6-7 ggplot2_2.2.1 lazyeval_0.2.0 gdata_2.17.0 splines_3.3.2 stringr_1.1.0 foreign_0.8-67 munsell_0.4.3 mnormt_1.5-5 tibble_1.2 codetools_0.2-15 bitops_1.0-6
[25] grid_3.3.2 nlme_3.1-128 xtable_1.8-2 gtable_0.2.0 DBI_0.5-1 magrittr_1.5 scales_0.4.1 KernSmooth_2.23-15 stringi_1.1.2 reshape2_1.4.2 tools_3.3.2 colorspace_1.3-2
[37] caTools_1.17.1