search

weizhouUMICH / SAIGE

GNU Lesser General Public License v3.0
188 stars 73 forks source link

SAIGE GENE requiring excessive memory #327

Closed jhryu closed 2 years ago

jhryu commented 3 years ago

Hi Wei,

I am running SAIGE-GENE and I am noticing that my job keeps getting killed at a specific gene, unless there is an excessive amount of memory. It seems to run through the first couple genes without issues, but then once it gets to this gene, I get the error message below with 300GB of memory. It seems to run without issues when I run this same job with 400GB of memory.

Thank you so much, Justine

geneID: gene name here std::size_t sample_size = marker_file.samples().size();200161 missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! cntMarker: 24 indexforMissing: 44827 68865 69853 169157 122727 94138 111956 40707 160879 177347 182750 34349 143974 136880 79041 116964 103648 63971 102174 20474 168358 178060 154905 70845 59814 173282 135540 138935 43406 188547 193273 193758 161861 103228 92178 85412 56845 150489 48271 23614 111303 6537 179607 110440 18846 127942 18753 31361 7360 102037 19426 11615 89788 67550 89819 88682 178170 148092 117508 99995 31211 156559 85775 168925 194779 126174 39544 113673 74535 144365 29797 38679 72029 176039 48275 24835 133252 64763 198660 2800 7043 88222 154676 174673 96231 13763 106031 85655 71468 133739 192251 20592 143356 111053 155312 133692 185312 98189 164706 64103 139793 48302 33481 131870 57733 191998 135509 75567 25661 35569 37728 104543 33125 162933 157994 25644 41055 34773 185366 144341 147185 4832 123780 104322 99753 41653 44864 160513 49863 134708 193942 125807 71902 87556 113777 53631 40372 142312 117668 136427 83784 162666 124641 46874 108514 114329 145794 2938 139770 35161 92906 196922 136952 169327 81725 142296 123406 168336 59319 87860 176684 189913 57916 174832 79029 79092 110345 128109 72811 54146 29800 199765 187306 115555 65312 80891 10609 101742 113461 135902 35356 192920 169329 120739 93528 163924 69257 39457 80678 73543 93345 133307 48318 194978 5507 83176 116243 119448 39803 141809 120348 36598 149003 33810 78631 68726 90499 140954 41532 65536 11484 108819 152268 37654 98785 116598 55889 73708 29722 14125 100571 114332 126987 152402 174074 103364 129066 50131 156949 126116 187637 173473 135388 51801 79730 151079 40935 113589 53232 127484 106094 34868 132405 158222 56048 197627 45621 153425 1563 55440 106442 190036 129723 101016 148990 59326 71401 72252 87350 104535 71187 34956 136655 100105 55204 185121 130473 20943 26908 46571 51636 170199 37260 30928 19741 76228 149963 138440 35192 194511 90513 156305 73193 133422 157917 120451 28798 102570 8089 13418 124708 65727 23714 78241 140357 157321 62882 145213 16200 200122 93849 61693 114331 82451 187055 28382 95639 150442 146990 36723 106211 93136 83701 74752 142025 61069 17286 13153 31807 79794 129538 66575 2332 110872 169165 11131 153720 2296 191366 77233 13029 198479 41459 794 11784 66673 111330 147678 48120 122485 39409 159777 198381 168889 144172 135246 127142 124715 26797 89008 15812 62201 65835 53986 47653 167133 142793 65235 15590 147457 134377 42521 96702 142308 127326 99688 119203 129578 109637 12137 26571 16791 50839 166804 196662 71336 171482 166466 69942 31370 147986 155939 151636 18132 111055 194473 53132 52391 121958 62177 77329 149091 191439 57180 136806 20019 71993 92138 157127 177692 19733 156300 118396 49859 97901 188593 108955 133598 106796 162821 40230 153147 131494 89777 84649 163463 198733 88062 183817 108123 39726 1885 34095 117034 22461 31772 105115 56183 38142 158793 550 85488 42806 113508 34618 65953 191238 76753 110434 188139 159295 86348 15244 125066 82229 60914 56656 71357 10298 89325 114826 8332 31129 77091 77450 15430 171182 52368 74108 188236 182105 84942 6136 196722 1120 29318 19021 176958 160264 114179 177113 187221 70787 144075 41462 21494 137419 14917 41504 87310 99888 57059 181268 11601 80041 99910 140878 32517 37899 35469 150361 48642 12791 80068 66209 166792 125771 51434 76263 190194 121588 90003 14509 60556 59047 59825 115278 152846 95700 62907 74584 179613 151830 129681 122625 102459 55291 45548 33936 108178 143680 100691 22853 58574 167009 113602 100098 73206 177647 183654 101008 55506 189471 73954 1044 194202 155923 129061 32695 158707 110102 147696 148354 105841 149819 28267 122480 56735 176900 57711 68073 112403 47562 163518 13894 29757 66734 77206 27409 13516 141543 86830 102439 73192 104970 182424 165795 13157 125478 177351 164410 46631 141376 112991 114989 77512 59653 177316 85480 16712 144053 69752 61389 78403 81217 20556 176521 176600 123480 192626 154618 30779 155194 33364 93293 54423 110700 112041 83067 71666 46010 148344 50389 175451 166576 127154 184747 196652 160266 121762 1513 134363 187922 158556 126888 183532 152750 55863 119968 97922 107032 18959 109372 153919 179450 171840 104269 129475 95840 91877 131598 113169 164722 100991 56345 176796 80604 52213 33614 128868 117945 149065 154745 148884 70270 157512 109610 122770 39728 169113 89080 73730 65205 180682 117074 132579 33484 148085 28529 91049 141548 60551 25639 8153 59813 91686 31322 1693 141189 18478 29256 187843 141136 184487 14944 158435 101422 34220 78446 190745 112000 49961 53753 119002 84533 120070 161244 48076 29341 166393 2622 40135 189789 137573 55556 177594 94316 85314 11028 56893 46910 126771 181176 94804 10742 94043 131058 102851 138465 81207 186296 69479 125402 91009 69561 75005 56057 21837 51336 116153 164817 11715 178614 31777 176457 49308 109764 47837 48200 103527 20539 102052 128328 186441 136700 54055 189063 93863 38441 186037 65489 108379 38014 143755 153197 185551 117690 165996 99842 12531 149305 23768 152998 16402 90782 170596 109822 44543 187495 171120 98447 60706 180007 135602 36007 110196 16038 28134 47195 3649 170364 96440 114580 77449 155308 172624 165560 184560 199563 23321 120519 9227 96215 60691 118755 36625 43449 101246 122612 103049 46747 127345 109156 167472 19524 18489 55840 124039 77595 22470 16802 7810 81695 18153 45750 154484 47394 133033 72644 89882 157973 151556 38057 4898 96548 111500 54054 101037 121673 144672 64209 41694 169298 164456 120047 145063 114900 193716 135196 169132 85587 6371 2304 189987 11764 198738 184255 119903 47705 124372 33824 28486 113424 189762 83785 28265 141610 14643 170835 53495 21712 117997 52852 137289 39828 134157 3434 34798 152835 72642 77525 46178 193549 186044 28568 186125 35978 9052 185919 145638 17706 415 8956 187874 43049 85310 152479 145164 34359 102010 119230 101078 105921 14284 26326 59956 54427 145614 85239 197990 53927 65108 199489 139909 46610 199616 170089 86443 114947 169242 134604 88436 8256 26664 186445 86521 49441 49622 145515 37384 178380 96719 124953 24039 16250 142160 32066 55908 81283 130577 171585 9984 12380 128285 31570 179994 58147 105515 76005 153065 74827 172503 43404 52989 161568 79396 23673 61890 61442 53416 163909 75157 24560 189454 5218 116387 69066 186907 108604 26610 156902 186878 119105 77737 68333 59897 142127 74236 96660 22280 42749 143195 41205 158439 66816 76892 70466 10962 193717 56534 180284 75017 120224 132205 145610 44411 126755 95116 25995 3679 77511 178177 145677 105972 163284 142454 198992 10375 39201 111602 164709 63668 199806 156694 9255 77796 170315 105779 68129 170597 83853 111356 184261 197672 177210 84007 193274 147785 113836 67714 91279 155866 95434 74080 160156 88269 149053 97230 107858 3906 137027 27430 101237 163889 190671 81712 83294 102972 132530 118191 168375 29134 36559 64804 16572 58553 194487 42900 99216 36018 172788 147893 146813 179162 34299 75006 100374 160342 182365 117335 154316 113900 47272 13279 87008 9743 52138 187934 127003 99036 26890 131266 149751 117345 61244 168180 140562 197950 174584 156428 171231 170381 17144 197536 77815 33917 82347 59140 182267 8697 100363 173884 137891 121965 113718 84669 115018 87105 193903 183471 156909 108539 8740 1401 12652 56042 29618 39239 66482 102768 178190 137622 6025 60777 197959 157989 158757 76252 163514 145916 154595 145533 174561 181422 71873 23817 152346 120818 132246 77750 6575 163900 89949 148335 156700 198538 84153 95445 45200 131660 108468 35992 43947 21362 104866 41753 142275 151683 121731 101682 170811 165870 29168 164458 165427 65960 177334 192040 152589 164505 161134 182830 166067 93611 58135 41829 7008 174261 37262 196482 26955 22479 183682 23417 85208 12297 187108 4473 78218 240 119118 190190 71498 42355 15667 193552 70029 157184 116189 24103 183498 7432 102169 142898 138325 124778 117556 144352 3560 195822 195232 195143 144781 26999 48941 179455 155323 150370 24980 158471 84741 113529 135208 78432 102526 90333 178908 57537 109511 7064 144830 76053 145700 16008 86947 153829 181997 173829 111780 69517 1493 50229 74049 178233 46990 175570 22317 66329 152717 158388 49224 71697 72321 147722 9777 184803 51493 40251 142268 30558 176263 115671 28153 73701 68669 78517 133770 154030 166669 144333 66004 23369 31878 190616 143281 199810 94824 84357 185341 131506 33234 11218 177743 49889 121220 98605 31421 185956 198110 101778 195850 184732 38335 26827 61146 42180 106822 41011 124164 190160 100227 141984 39094 126557 181827 59595 107185 80064 198654 92163 155400 120559 144462 150737 170466 89150 68159 99121 129904 165490 174381 160819 6374 88754 113951 98948 139548 182373 147284 53918 73892 2657 5103 165505 198158 24493 125638 127678 27491 188060 140566 113190 3532 116716 34841 85539 168899 134511 186466 153089 25080 30675 128676 118356 150040 48082 119710 144519 9405 185575 166658 15615 175567 185339 178492 9411 139670 29097 7907 28822 156113 93412 84292 156948 15696 156339 182923 171888 106225 15010 64962 53155 37799 55701 18490 27892 196540 12362 155064 152416 194177 195208 58167 29377 47079 117598 65245 189927 143235 104259 71543 3720 131187 145523 67334 74552 58073 92768 31472 169809 190910 180847 78500 181747 84849 108457 49787 59934 100141 127934 17376 173137 35360 16005 14410 19119 131370 149656 182349 144434 157631 152810 80479 116578 56832 129793 15396 40243 186239 100996 170345 181359 119550 128413 6128 199620 54534 17891 104254 171862 165861 126961 185623 53888 196132 181068 190685 156252 147660 129109 63358 136874 176038 181036 70099 148355 60986 55451 12192 115794 55814 24949 84054 197245 92627 141316 67893 146256 13384 48556 77839 23078 19804 10274 84816 47651 53972 81606 10729 46245 109245 71963 33328 34681 140261 156967 140259 38309 130452 194308 45539 162465 19415 74389 115019 104349 3312 165120 122467 30557 82623 8869 36237 100165 81813 5139 49518 139814 2044 114562 87849 108757 33188 112071 197278 101448 8707 63264 47500 73027 170734 107022 156702 150204 190813 162462 126150 101792 7383 171589 89177 1978 128614 70435 17520 13874 60584 115754 152944 131774 189384 133029 191024 164146 93661 38437 41122 161744 187557 22430 11669 63564 183330 148709 132933 64966 120465 154271 134869 146149 196181 102156 106849 6874 36973 141363 199876 102483 177167 135965 76242 71648 6377 71232 30255 41378 71044 139448 88137 137116 149833 56558 112862 172340 138438 155940 144950 10132 7865 9617 52817 107579 17788 15242 87983 106916 75007 94279 45737 165908 145403 156802 125870 105001 3278 10245 26146 199546 64367 115879 65293 127948 70321 37082 151659 51533 117523 191218 132332 171139 54233 90058 155579 185090 182434 192683 161789 166613 59085 166734 150363 192581 49148 140423 45922 122582 67090 116347 138566 72083 198467 121841 173451 31601 154750 79418 62855 157643 166600 5275 114544 167555 1350 91326 145595 165752 149804 162003 101745 140768 138167 79220 151421 49028 136404 84010 196946 139042 70671 185480 185961 126963 112955 148569 114164 102830 151572 183906 66494 162546 154421 149226 199979 69495 120841 165846 164305 76835 128822 103017 78045 32544 85328 63128 4704 64510 66377 57003 195666 174023 189267 22040 142899 69338 105119 138160 165455 133129 25876 82731 11685 144272 140461 116809 86376 15961 187804 99787 22388 85992 148768 74288 3024 140186 12308 39602 67287 147763 16691 99250 37061 5534 179813 104732 25407 164089 123057 38287 104875 106581 120550 149860 160411 isCondition is FALSE weights: 24.9985 24.9955 24.997 24.79398 24.9985 24.9985 24.99251 24.99401 24.9985 24.9955 24.9985 24.99101 24.9985 24.9985 24.8729 24.9985 24.994 24.991 24.99849 24.997 24.997 24.99251 24.9955 24.9985 /var/spool/uger-8.5.5/ugerbm-c002/job_scripts/23439903: line 63: 679 Killed

humanpaingeneticslab commented 3 years ago

Hi Justine, Could you check the genetic boundaries for that gene (chromosome start and end positions), and make sure that it is "reasonable". Let us know!

jhryu commented 3 years ago

Hi,

I was running the program for CHR22 and it is getting stuck at the gene CECR5, CHR22:17137511 - 17165287. Is this gene too large? And if it is, would you recommend splitting the gene for the analysis?

Thank you for the quick reply! Justine

humanpaingeneticslab commented 3 years ago

Hi Justine,

I was running the program for CHR22 and it is getting stuck at the gene CECR5, CHR22:17137511 - 17165287.

Looks typical - not overly long (30K nt).

Is this gene too large? And if it is, would you recommend splitting the gene for the analysis?

No I wouldn’t split it. If it’s the only gene giving you trouble, then drop it; there’s 30,000 other genes! OTOH, if IT is your gene of interest, then I’m not sure how to proceed next.

I would have expected a few genes in chr 6 in the MHC region to be more troublesome, as this region is quite dense in SNPs.

Good Luck1

weizhouUMICH commented 3 years ago

Hi Justin,

It seems the gene only contains 24 markers, so it's not a large gene, and it should not need that large memory. Was this error only occurring for this gene? Have other genes with more markers been successfully tested?

Can you check how large is the sparse Sigma matrix (input for Step 2)?

Thanks, Wei

jhryu commented 3 years ago

Hi Wei,

The sparse Sigma matrix is 14.7MB.

The error has unfortunately occurred for other genes in other chromosomes as well.

I have also received this error below with the exact same script. I think it may be this large vector (298.5 Gb) that is requiring so much memory to run the script. Do you know what this might be referring to?

Thank you again! Justine

geneID: CECR5 std::size_t sample_size = marker_file.samples().size();200161 missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! missing_cnt > 0! cntMarker: 24 indexforMissing: 44827 68865 69853 169157 122727 94138 111956 40707 160879 177347 182750 34349 143974 136880 79041 116964 103648 63971 102174 20474 168358 178060 154905 70845 59814 173282 135540 138935 43406 188547 193273 193758 161861 103228 92178 85412 56845 150489 48271 23614 111303 6537 179607 110440 18846 127942 18753 31361 7360 102037 19426 11615 89788 67550 89819 88682 178170 148092 117508 99995 31211 156559 85775 168925 194779 126174 39544 113673 74535 144365 29797 38679 72029 176039 48275 24835 133252 64763 198660 2800 7043 88222 154676 174673 96231 13763 106031 85655 71468 133739 192251 20592 143356 111053 155312 133692 185312 98189 164706 64103 139793 48302 33481 131870 57733 191998 135509 75567 25661 35569 37728 104543 33125 162933 157994 25644 41055 34773 185366 144341 147185 4832 123780 104322 99753 41653 44864 160513 49863 134708 193942 125807 71902 87556 113777 53631 40372 142312 117668 136427 83784 162666 124641 46874 108514 114329 145794 2938 139770 35161 92906 196922 136952 169327 81725 142296 123406 168336 59319 87860 176684 189913 57916 174832 79029 79092 110345 128109 72811 54146 29800 199765 187306 115555 65312 80891 10609 101742 113461 135902 35356 192920 169329 120739 93528 163924 69257 39457 80678 73543 93345 133307 48318 194978 5507 83176 116243 119448 39803 141809 120348 36598 149003 33810 78631 68726 90499 140954 41532 65536 11484 108819 152268 37654 98785 116598 55889 73708 29722 14125 100571 114332 126987 152402 174074 103364 129066 50131 156949 126116 187637 173473 135388 51801 79730 151079 40935 113589 53232 127484 106094 34868 132405 158222 56048 197627 45621 153425 1563 55440 106442 190036 129723 101016 148990 59326 71401 72252 87350 104535 71187 34956 136655 100105 55204 185121 130473 20943 26908 46571 51636 170199 37260 30928 19741 76228 149963 138440 35192 194511 90513 156305 73193 133422 157917 120451 28798 102570 8089 13418 124708 65727 23714 78241 140357 157321 62882 145213 16200 200122 93849 61693 114331 82451 187055 28382 95639 150442 146990 36723 106211 93136 83701 74752 142025 61069 17286 13153 31807 79794 129538 66575 2332 110872 169165 11131 153720 2296 191366 77233 13029 198479 41459 794 11784 66673 111330 147678 48120 122485 39409 159777 198381 168889 144172 135246 127142 124715 26797 89008 15812 62201 65835 53986 47653 167133 142793 65235 15590 147457 134377 42521 96702 142308 127326 99688 119203 129578 109637 12137 26571 16791 50839 166804 196662 71336 171482 166466 69942 31370 147986 155939 151636 18132 111055 194473 53132 52391 121958 62177 77329 149091 191439 57180 136806 20019 71993 92138 157127 177692 19733 156300 118396 49859 97901 188593 108955 133598 106796 162821 40230 153147 131494 89777 84649 163463 198733 88062 183817 108123 39726 1885 34095 117034 22461 31772 105115 56183 38142 158793 550 85488 42806 113508 34618 65953 191238 76753 110434 188139 159295 86348 15244 125066 82229 60914 56656 71357 10298 89325 114826 8332 31129 77091 77450 15430 171182 52368 74108 188236 182105 84942 6136 196722 1120 29318 19021 176958 160264 114179 177113 187221 70787 144075 41462 21494 137419 14917 41504 87310 99888 57059 181268 11601 80041 99910 140878 32517 37899 35469 150361 48642 12791 80068 66209 166792 125771 51434 76263 190194 121588 90003 14509 60556 59047 59825 115278 152846 95700 62907 74584 179613 151830 129681 122625 102459 55291 45548 33936 108178 143680 100691 22853 58574 167009 113602 100098 73206 177647 183654 101008 55506 189471 73954 1044 194202 155923 129061 32695 158707 110102 147696 148354 105841 149819 28267 122480 56735 176900 57711 68073 112403 47562 163518 13894 29757 66734 77206 27409 13516 141543 86830 102439 73192 104970 182424 165795 13157 125478 177351 164410 46631 141376 112991 114989 77512 59653 177316 85480 16712 144053 69752 61389 78403 81217 20556 176521 176600 123480 192626 154618 30779 155194 33364 93293 54423 110700 112041 83067 71666 46010 148344 50389 175451 166576 127154 184747 196652 160266 121762 1513 134363 187922 158556 126888 183532 152750 55863 119968 97922 107032 18959 109372 153919 179450 171840 104269 129475 95840 91877 131598 113169 164722 100991 56345 176796 80604 52213 33614 128868 117945 149065 154745 148884 70270 157512 109610 122770 39728 169113 89080 73730 65205 180682 117074 132579 33484 148085 28529 91049 141548 60551 25639 8153 59813 91686 31322 1693 141189 18478 29256 187843 141136 184487 14944 158435 101422 34220 78446 190745 112000 49961 53753 119002 84533 120070 161244 48076 29341 166393 2622 40135 189789 137573 55556 177594 94316 85314 11028 56893 46910 126771 181176 94804 10742 94043 131058 102851 138465 81207 186296 69479 125402 91009 69561 75005 56057 21837 51336 116153 164817 11715 178614 31777 176457 49308 109764 47837 48200 103527 20539 102052 128328 186441 136700 54055 189063 93863 38441 186037 65489 108379 38014 143755 153197 185551 117690 165996 99842 12531 149305 23768 152998 16402 90782 170596 109822 44543 187495 171120 98447 60706 180007 135602 36007 110196 16038 28134 47195 3649 170364 96440 114580 77449 155308 172624 165560 184560 199563 23321 120519 9227 96215 60691 118755 36625 43449 101246 122612 103049 46747 127345 109156 167472 19524 18489 55840 124039 77595 22470 16802 7810 81695 18153 45750 154484 47394 133033 72644 89882 157973 151556 38057 4898 96548 111500 54054 101037 121673 144672 64209 41694 169298 164456 120047 145063 114900 193716 135196 169132 85587 6371 2304 189987 11764 198738 184255 119903 47705 124372 33824 28486 113424 189762 83785 28265 141610 14643 170835 53495 21712 117997 52852 137289 39828 134157 3434 34798 152835 72642 77525 46178 193549 186044 28568 186125 35978 9052 185919 145638 17706 415 8956 187874 43049 85310 152479 145164 34359 102010 119230 101078 105921 14284 26326 59956 54427 145614 85239 197990 53927 65108 199489 139909 46610 199616 170089 86443 114947 169242 134604 88436 8256 26664 186445 86521 49441 49622 145515 37384 178380 96719 124953 24039 16250 142160 32066 55908 81283 130577 171585 9984 12380 128285 31570 179994 58147 105515 76005 153065 74827 172503 43404 52989 161568 79396 23673 61890 61442 53416 163909 75157 24560 189454 5218 116387 69066 186907 108604 26610 156902 186878 119105 77737 68333 59897 142127 74236 96660 22280 42749 143195 41205 158439 66816 76892 70466 10962 193717 56534 180284 75017 120224 132205 145610 44411 126755 95116 25995 3679 77511 178177 145677 105972 163284 142454 198992 10375 39201 111602 164709 63668 199806 156694 9255 77796 170315 105779 68129 170597 83853 111356 184261 197672 177210 84007 193274 147785 113836 67714 91279 155866 95434 74080 160156 88269 149053 97230 107858 3906 137027 27430 101237 163889 190671 81712 83294 102972 132530 118191 168375 29134 36559 64804 16572 58553 194487 42900 99216 36018 172788 147893 146813 179162 34299 75006 100374 160342 182365 117335 154316 113900 47272 13279 87008 9743 52138 187934 127003 99036 26890 131266 149751 117345 61244 168180 140562 197950 174584 156428 171231 170381 17144 197536 77815 33917 82347 59140 182267 8697 100363 173884 137891 121965 113718 84669 115018 87105 193903 183471 156909 108539 8740 1401 12652 56042 29618 39239 66482 102768 178190 137622 6025 60777 197959 157989 158757 76252 163514 145916 154595 145533 174561 181422 71873 23817 152346 120818 132246 77750 6575 163900 89949 148335 156700 198538 84153 95445 45200 131660 108468 35992 43947 21362 104866 41753 142275 151683 121731 101682 170811 165870 29168 164458 165427 65960 177334 192040 152589 164505 161134 182830 166067 93611 58135 41829 7008 174261 37262 196482 26955 22479 183682 23417 85208 12297 187108 4473 78218 240 119118 190190 71498 42355 15667 193552 70029 157184 116189 24103 183498 7432 102169 142898 138325 124778 117556 144352 3560 195822 195232 195143 144781 26999 48941 179455 155323 150370 24980 158471 84741 113529 135208 78432 102526 90333 178908 57537 109511 7064 144830 76053 145700 16008 86947 153829 181997 173829 111780 69517 1493 50229 74049 178233 46990 175570 22317 66329 152717 158388 49224 71697 72321 147722 9777 184803 51493 40251 142268 30558 176263 115671 28153 73701 68669 78517 133770 154030 166669 144333 66004 23369 31878 190616 143281 199810 94824 84357 185341 131506 33234 11218 177743 49889 121220 98605 31421 185956 198110 101778 195850 184732 38335 26827 61146 42180 106822 41011 124164 190160 100227 141984 39094 126557 181827 59595 107185 80064 198654 92163 155400 120559 144462 150737 170466 89150 68159 99121 129904 165490 174381 160819 6374 88754 113951 98948 139548 182373 147284 53918 73892 2657 5103 165505 198158 24493 125638 127678 27491 188060 140566 113190 3532 116716 34841 85539 168899 134511 186466 153089 25080 30675 128676 118356 150040 48082 119710 144519 9405 185575 166658 15615 175567 185339 178492 9411 139670 29097 7907 28822 156113 93412 84292 156948 15696 156339 182923 171888 106225 15010 64962 53155 37799 55701 18490 27892 196540 12362 155064 152416 194177 195208 58167 29377 47079 117598 65245 189927 143235 104259 71543 3720 131187 145523 67334 74552 58073 92768 31472 169809 190910 180847 78500 181747 84849 108457 49787 59934 100141 127934 17376 173137 35360 16005 14410 19119 131370 149656 182349 144434 157631 152810 80479 116578 56832 129793 15396 40243 186239 100996 170345 181359 119550 128413 6128 199620 54534 17891 104254 171862 165861 126961 185623 53888 196132 181068 190685 156252 147660 129109 63358 136874 176038 181036 70099 148355 60986 55451 12192 115794 55814 24949 84054 197245 92627 141316 67893 146256 13384 48556 77839 23078 19804 10274 84816 47651 53972 81606 10729 46245 109245 71963 33328 34681 140261 156967 140259 38309 130452 194308 45539 162465 19415 74389 115019 104349 3312 165120 122467 30557 82623 8869 36237 100165 81813 5139 49518 139814 2044 114562 87849 108757 33188 112071 197278 101448 8707 63264 47500 73027 170734 107022 156702 150204 190813 162462 126150 101792 7383 171589 89177 1978 128614 70435 17520 13874 60584 115754 152944 131774 189384 133029 191024 164146 93661 38437 41122 161744 187557 22430 11669 63564 183330 148709 132933 64966 120465 154271 134869 146149 196181 102156 106849 6874 36973 141363 199876 102483 177167 135965 76242 71648 6377 71232 30255 41378 71044 139448 88137 137116 149833 56558 112862 172340 138438 155940 144950 10132 7865 9617 52817 107579 17788 15242 87983 106916 75007 94279 45737 165908 145403 156802 125870 105001 3278 10245 26146 199546 64367 115879 65293 127948 70321 37082 151659 51533 117523 191218 132332 171139 54233 90058 155579 185090 182434 192683 161789 166613 59085 166734 150363 192581 49148 140423 45922 122582 67090 116347 138566 72083 198467 121841 173451 31601 154750 79418 62855 157643 166600 5275 114544 167555 1350 91326 145595 165752 149804 162003 101745 140768 138167 79220 151421 49028 136404 84010 196946 139042 70671 185480 185961 126963 112955 148569 114164 102830 151572 183906 66494 162546 154421 149226 199979 69495 120841 165846 164305 76835 128822 103017 78045 32544 85328 63128 4704 64510 66377 57003 195666 174023 189267 22040 142899 69338 105119 138160 165455 133129 25876 82731 11685 144272 140461 116809 86376 15961 187804 99787 22388 85992 148768 74288 3024 140186 12308 39602 67287 147763 16691 99250 37061 5534 179813 104732 25407 164089 123057 38287 104875 106581 120550 149860 160411 isCondition is FALSE weights: 24.9985 24.9955 24.997 24.79398 24.9985 24.9985 24.99251 24.99401 24.9985 24.9955 24.9985 24.99101 24.9985 24.9985 24.8729 24.9985 24.994 24.991 24.99849 24.997 24.997 24.99251 24.9955 24.9985 Error: cannot allocate vector of size 298.5 Gb Timing stopped at: 11.64 0.06 11.7 Execution halted

weizhouUMICH commented 3 years ago

Hi @jhryu,

This is a little bit strange. My benchmark job on the similar sample sizes for genes with > 2000 markers costs < 8Gb memory.

Which version of SAIGE-GENE are you using?

Thanks, Wei

jhryu commented 3 years ago

Hi Wei,

Sorry for the extremely late reply. But the version I am using is 0.36.3.2.

Thanks, Justine

On Wed, May 5, 2021 at 12:17 PM weizhouUMICH @.***> wrote:

Hi @jhryu https://github.com/jhryu,

This is a little bit strange. My benchmark job on the similar sample sizes for genes with > 2000 markers costs < 8Gb memory.

Which version of SAIGE-GENE are you using?

Thanks, Wei

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/weizhouUMICH/SAIGE/issues/327#issuecomment-832824807, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB3UXW3Z5DWL7IVTGOPNO53TMFVQDANCNFSM42XU6EPQ .

weizhouUMICH commented 2 years ago

Sorry fo the late reply! We have just released a new version 1.0.0. It has substantial computational efficiency improvements for both Step 1 and Step 2 for single-variant and set-based tests and clearer log output. We have created a new program github page https://github.com/saigegit/SAIGE with the documentation provided https://saigegit.github.io/SAIGE-doc/ The program will be maintained by multiple SAIGE developers there. The docker image has been updated. Please feel free to try the version 1.0.0 and report issues if any.

Thanks! Wei