search

broadinstitute / BipolarCell2016

Other
22 stars 11 forks source link

No clusters merged by merge.clusters.DE when min.de.genes = 50 #2

Open edm1 opened 8 years ago

edm1 commented 8 years ago

Hi,

Thanks for including the code and data for the Drop-seq analysis of retinal bipolar cells. I've been going over the methods and have noticed that one step isn't working as expected.

After:

dsq.bip = merge.clusters.DE(dsq.bip, min.de.genes = 50, pcs.use = 1:37, TPM.mat = TPM.mat,
Count.mat = Count.mat)

I still have 30 clusters (rather than the 26 expected)

> table(dsq.bip@group)

   1    2    3    4    5    6    7    8    9   10   11   12   13   14   15   16   17   18   19   20   21   22 
7827 2945 2399 2237 1759 1655 1372 1091  817  792  558  553  535  479  398  377  313  285  252  232  173  100 
  23   24   25   26   27   28   29   30 
  91   53   48   47   46   38   14   13

Here is the DE_genes_matrix_2.txt output file which shows, for example, there are 66 DE genes between clusters 1 and 3. I was expecting fewer than 50.

Is this a fault with the code or was a different value of min.de.genes used to produce the clusters in the paper?

Thanks, Ed Mountjoy

carolynlou commented 7 years ago

Hi Ed,

Were you able to figure out the issue? I am also going through the methods right now and am getting the same results as you for this step.

Thanks, Carolyn

karthikshekhar commented 7 years ago

Hi both,

Ed, I am so sorry - I don't know how I missed this! I have not been following the github comments much, as I generally receive correspondence on my work email. Apologies again!

I am guessing both of you are using the GSE81904_class.R from the GEO website. Please use the class.R script on the github page. Essentially the difference arises due to one line (line 432 in GSE81904_class.R ):

num.de.genes = nrow(marker.pass)

This should be replaced with the following line, which is line 442 in the github class.R

num.de.genes = min(nrow(subset(marker.pass, log.effect > 0)), nrow(subset(marker.pass, log.effect < 0)))

Essentially the second condition is more stringent in that it looks for both +ve and -vely enriched genes between two clusters to decide whether to merge them.

I am copying below the stderr trail of the function that I ran in response to an email that I wrote (coincidentally on this day last year :) ) - so please let me know if this change does not solve your issue!

Karthik

> dsq.bip = merge.clusters.DE(dsq.bip, min.de.genes = 50, pcs.use=1:37, TPM.mat=TPM.mat, Count.mat=Count.mat)
[1] "Starting with 30 clusters"
[1] "Testing Cluster 1"
     X1   X2 X3  X4  X5  X6  X7  X8  X9 X10 X11 X12 X13 X14 X15 X16 X17 X18 X19 X20 X21 X22 X23 X24  X25 X26 X27  X28  X29 X30
1 1e+06 1274 40 384 264 244 314 494 488 402 308 414 398 328 472  12 404  14 476 118 154 536 764 462 1028 362  66 1116 1280 998
[1] "Testing Cluster 2"
    X1    X2   X3   X4   X5   X6   X7   X8   X9  X10  X11  X12  X13  X14  X15  X16  X17 X18  X19 X20  X21  X22 X23 X24  X25 X26 X27 X28
2 1274 1e+06 1440 1416 1432 1594 1522 1266 1308 1378 1424 1316 1544 1436 1514 1142 1592 916 1194  98 1530 1206 940 976 1412 994 924 714
  X29 X30
2 712 636
[1] "Testing Cluster 3"
  X1   X2    X3  X4  X5  X6  X7  X8  X9 X10 X11 X12 X13 X14 X15 X16 X17 X18 X19 X20 X21 X22 X23 X24 X25 X26 X27  X28  X29 X30
3 40 1440 1e+06 512 452 310 450 426 434 364 526 676 468 502 664  22 426   2 728 412 374 346 586 314 716 244 266 1954 1092 818
[1] "Testing Cluster 4"
   X1   X2  X3    X4  X5  X6  X7  X8  X9 X10 X11 X12 X13 X14 X15 X16 X17 X18 X19 X20 X21 X22 X23 X24 X25 X26 X27  X28  X29 X30
4 384 1416 512 1e+06 264 352 302 382 280 380 432 188 398 286 450 366 362 256 526 410 448 132 616 228 744 276 314 1586 1076 792
[1] "Testing Cluster 5"
   X1   X2  X3  X4    X5  X6  X7  X8  X9 X10 X11 X12 X13 X14 X15 X16 X17 X18 X19 X20 X21 X22 X23 X24 X25 X26 X27  X28  X29 X30
5 264 1432 452 264 1e+06 196 282 294 286 290 396 444 354 388 450 252 230 170 644 496 388  92 568 204 652 168 346 1844 1042 744
[1] "Testing Cluster 6"
   X1   X2  X3  X4  X5    X6  X7  X8  X9 X10 X11 X12 X13 X14 X15 X16 X17 X18 X19 X20 X21 X22 X23 X24 X25 X26 X27  X28  X29 X30
6 244 1594 310 352 196 1e+06 294 306 334 292 200 404 326 296 398 248 298 162 526 272 264 310 600 298 748  42 192 1462 1172 850
[1] "Testing Cluster 7"
   X1   X2  X3  X4  X5  X6    X7  X8  X9 X10 X11 X12 X13 X14 X15 X16 X17 X18 X19 X20 X21 X22 X23 X24 X25 X26 X27  X28  X29 X30
7 314 1522 450 302 282 294 1e+06 294 256 268 372 340 360 138 368 312 338 212 510 374 438 244 618  88 764 226 302 1616 1102 828
[1] "Testing Cluster 8"
   X1   X2  X3  X4  X5  X6  X7    X8  X9 X10 X11 X12 X13 X14 X15 X16 X17 X18 X19 X20 X21 X22 X23 X24 X25 X26 X27  X28  X29 X30
8 494 1266 426 382 294 306 294 1e+06 240 146 170 374 268 322 158 482 276 322 386 314 310 464 696 374 944 326 208 1248 1164 904
[1] "Testing Cluster 9"
   X1   X2  X3  X4  X5  X6  X7  X8    X9 X10 X11 X12 X13 X14 X15 X16 X17 X18 X19 X20 X21 X22 X23 X24 X25 X26 X27  X28  X29 X30
9 488 1308 434 280 286 334 256 240 1e+06 328 276 294 266 236  86 478 320 338 440 294 284 364 636 322 836 334 198 1242 1152 910
[1] "Testing Cluster 10"
    X1   X2  X3  X4  X5  X6  X7  X8  X9   X10 X11 X12 X13 X14 X15 X16 X17 X18 X19 X20 X21 X22 X23 X24 X25 X26 X27  X28  X29 X30
10 402 1378 364 380 290 292 268 146 328 1e+06 144 378 242 302 258 392 324 258 380 300 338 412 716 354 938 270 200 1300 1122 948
[1] "Testing Cluster 11"
    X1   X2  X3  X4  X5  X6  X7  X8  X9 X10   X11 X12 X13 X14 X15 X16 X17 X18 X19 X20 X21 X22 X23 X24 X25 X26 X27  X28 X29 X30
11 308 1424 526 432 396 200 372 170 276 144 1e+06 498 214 446 396 312 302 214 612 584 390 272 582 298 658 156 454 1912 984 776
[1] "Testing Cluster 12"
    X1   X2  X3  X4  X5  X6  X7  X8  X9 X10 X11   X12 X13 X14 X15 X16 X17 X18 X19 X20 X21 X22 X23 X24 X25 X26 X27  X28 X29 X30
12 414 1316 676 188 444 404 340 374 294 378 498 1e+06 384 308 492 402 360 268 566 638 472 214 546 232 622 292 596 1890 958 738
[1] "Testing Cluster 13"
    X1   X2  X3  X4  X5  X6  X7  X8  X9 X10 X11 X12   X13 X14 X15 X16 X17 X18 X19 X20 X21 X22 X23 X24 X25 X26 X27  X28  X29 X30
13 398 1544 468 398 354 326 360 268 266 242 214 384 1e+06 340 334 426 388 272 512 392 396 388 626 352 742 240 282 1504 1092 800
[1] "Testing Cluster 14"
    X1   X2  X3  X4  X5  X6  X7  X8  X9 X10 X11 X12 X13   X14 X15 X16 X17 X18 X19 X20 X21 X22 X23 X24 X25 X26 X27  X28  X29 X30
14 328 1436 502 286 388 296 138 322 236 302 446 308 340 1e+06 406 312 354 216 624 512 426 238 568 124 660 252 418 1926 1042 796
[1] "Testing Cluster 15"
    X1   X2  X3  X4  X5  X6  X7  X8 X9 X10 X11 X12 X13 X14   X15 X16 X17 X18 X19 X20 X21 X22 X23 X24 X25 X26 X27  X28  X29 X30
15 472 1514 664 450 450 398 368 158 86 258 396 492 334 406 1e+06 450 366 304 530 490 538 374 622 328 710 350 418 1624 1060 802
[1] "Testing Cluster 16"
   X1   X2 X3  X4  X5  X6  X7  X8  X9 X10 X11 X12 X13 X14 X15   X16 X17 X18 X19 X20 X21 X22  X23 X24 X25 X26 X27  X28  X29  X30
16 12 1142 22 366 252 248 312 482 478 392 312 402 426 312 450 1e+06 360 360 464 182 202 456 1310 390 944 320  22 1080 1022 1230
[1] "Testing Cluster 17"
    X1   X2  X3  X4  X5  X6  X7  X8  X9 X10 X11 X12 X13 X14 X15 X16   X17 X18 X19 X20 X21 X22 X23 X24 X25 X26 X27  X28  X29 X30
17 404 1592 426 362 230 298 338 276 320 324 302 360 388 354 366 360 1e+06 258 460 372 460 316 610 330 746 244 292 1562 1092 860
[1] "Testing Cluster 18"
   X1  X2 X3  X4  X5  X6  X7  X8  X9 X10 X11 X12 X13 X14 X15 X16 X17   X18 X19 X20 X21 X22 X23 X24 X25 X26 X27 X28 X29 X30
18 14 916  2 256 170 162 212 322 338 258 214 268 272 216 304 360 258 1e+06 318  40  78 972 928 748 668 636  52 868 838 944
[1] "Testing Cluster 19"
    X1   X2  X3  X4  X5  X6  X7  X8  X9 X10 X11 X12 X13 X14 X15 X16 X17 X18   X19  X20 X21 X22 X23 X24 X25 X26  X27  X28 X29 X30
19 476 1194 728 526 644 526 510 386 440 380 612 566 512 624 530 464 460 318 1e+06 1148 600 400 612 392 686 402 1318 1686 806 636
[1] "Testing Cluster 20"
    X1 X2  X3  X4  X5  X6  X7  X8  X9 X10 X11 X12 X13 X14 X15 X16 X17 X18  X19   X20 X21 X22 X23 X24 X25 X26 X27  X28 X29 X30
20 118 98 412 410 496 272 374 314 294 300 584 638 392 512 490 182 372  40 1148 1e+06 266 290 566 268 608 214 634 1082 570 394
[1] "Testing Cluster 21"
    X1   X2  X3  X4  X5  X6  X7  X8  X9 X10 X11 X12 X13 X14 X15 X16 X17 X18 X19 X20   X21 X22 X23 X24 X25 X26 X27  X28  X29 X30
21 154 1530 374 448 388 264 438 310 284 338 390 472 396 426 538 202 460  78 600 266 1e+06 320 320 280 540 188 236 1532 1092 742
[1] "Testing Cluster 22"
    X1   X2  X3  X4 X5  X6  X7  X8  X9 X10 X11 X12 X13 X14 X15 X16 X17 X18 X19 X20 X21   X22  X23 X24 X25 X26 X27  X28 X29 X30
22 536 1206 346 132 92 310 244 464 364 412 272 214 388 238 374 456 316 972 400 290 320 1e+06 1360 304 974 322  96 1132 732 900
[1] "Testing Cluster 23"
    X1  X2  X3  X4  X5  X6  X7  X8  X9 X10 X11 X12 X13 X14 X15  X16 X17 X18 X19 X20 X21  X22   X23  X24 X25  X26 X27 X28 X29 X30
23 764 940 586 616 568 600 618 696 636 716 582 546 626 568 622 1310 610 928 612 566 320 1360 1e+06 1354 354 1278 418 846 642 630
[1] "Testing Cluster 24"
    X1  X2  X3  X4  X5  X6 X7  X8  X9 X10 X11 X12 X13 X14 X15 X16 X17 X18 X19 X20 X21 X22  X23   X24 X25 X26 X27 X28 X29 X30
24 462 976 314 228 204 298 88 374 322 354 298 232 352 124 328 390 330 748 392 268 280 304 1354 1e+06 844 506 120 978 502 612
[1] "Testing Cluster 25"
     X1   X2  X3  X4  X5  X6  X7  X8  X9 X10 X11 X12 X13 X14 X15 X16 X17 X18 X19 X20 X21 X22 X23 X24   X25 X26 X27  X28 X29 X30
25 1028 1412 716 744 652 748 764 944 836 938 658 622 742 660 710 944 746 668 686 608 540 974 354 844 1e+06 774 554 1066 840 722
[1] "Testing Cluster 26"
    X1  X2  X3  X4  X5 X6  X7  X8  X9 X10 X11 X12 X13 X14 X15 X16 X17 X18 X19 X20 X21 X22  X23 X24 X25   X26 X27 X28 X29 X30
26 362 994 244 276 168 42 226 326 334 270 156 292 240 252 350 320 244 636 402 214 188 322 1278 506 774 1e+06  90 818 420 546
[1] "Testing Cluster 27"
   X1  X2  X3  X4  X5  X6  X7  X8  X9 X10 X11 X12 X13 X14 X15 X16 X17 X18  X19 X20 X21 X22 X23 X24 X25 X26   X27  X28 X29 X30
27 66 924 266 314 346 192 302 208 198 200 454 596 282 418 418  22 292  52 1318 634 236  96 418 120 554  90 1e+06 1380 622 436
[1] "Testing Cluster 28"
     X1  X2   X3   X4   X5   X6   X7   X8   X9  X10  X11  X12  X13  X14  X15  X16  X17 X18  X19  X20  X21  X22 X23 X24  X25 X26  X27   X28
28 1116 714 1954 1586 1844 1462 1616 1248 1242 1300 1912 1890 1504 1926 1624 1080 1562 868 1686 1082 1532 1132 846 978 1066 818 1380 1e+06
   X29 X30
28 336 246
[1] "Testing Cluster 29"
     X1  X2   X3   X4   X5   X6   X7   X8   X9  X10 X11 X12  X13  X14  X15  X16  X17 X18 X19 X20  X21 X22 X23 X24 X25 X26 X27 X28   X29
29 1280 712 1092 1076 1042 1172 1102 1164 1152 1122 984 958 1092 1042 1060 1022 1092 838 806 570 1092 732 642 502 840 420 622 336 1e+06
   X30
29 240
[1] "Testing Cluster 30"
    X1  X2  X3  X4  X5  X6  X7  X8  X9 X10 X11 X12 X13 X14 X15  X16 X17 X18 X19 X20 X21 X22 X23 X24 X25 X26 X27 X28 X29   X30
30 998 636 818 792 744 850 828 904 910 948 776 738 800 796 802 1230 860 944 636 394 742 900 630 612 722 546 436 246 240 1e+06
[1] "Recomputing pairwise markers for new clust 3"
[1] 1
[1] 2
[1] 4
[1] 5
[1] 6
[1] 7
[1] 8
[1] 9
[1] 10
[1] 11
[1] 12
[1] 13
[1] 14
[1] 15
[1] 16
[1] 17
[1] 18
[1] 19
[1] 20
[1] 21
[1] 22
[1] 23
[1] 24
[1] 25
[1] 26
[1] 27
[1] 28
[1] 29
[1] "Recomputing pairwise markers for new clust 1"
[1] 2
[1] 3
[1] 4
[1] 5
[1] 6
[1] 7
[1] 8
[1] 9
[1] 10
[1] 11
[1] 12
[1] 13
[1] 14
[1] 15
[1] 16
[1] 17
[1] 18
[1] 19
[1] 20
[1] 21
[1] 22
[1] 23
[1] 24
[1] 25
[1] 26
[1] 27
[1] 28
[1] "Recomputing pairwise markers for new clust 1"
[1] 2
[1] 3
[1] 4
[1] 5
[1] 6
[1] 7
[1] 8
[1] 9
[1] 10
[1] 11
[1] 12
[1] 13
[1] 14
[1] 15
[1] 16
[1] 17
[1] 18
[1] 19
[1] 20
[1] 21
[1] 22
[1] 23
[1] 24
[1] 25
[1] 26
[1] 27
[1] "Recomputing pairwise markers for new clust 5"
[1] 1
[1] 2
[1] 3
[1] 4
[1] 6
[1] 7
[1] 8
[1] 9
[1] 10
[1] 11
[1] 12
[1] 13
[1] 14
[1] 15
[1] 16
[1] 17
[1] 18
[1] 19
[1] 20
[1] 21
[1] 22
[1] 23
[1] 24
[1] 25
[1] 26
> table(dsq.bip@group)

    1     2     3     4     5     6     7     8     9    10    11    12    13    14    15    16    17    18    19    20    21    22    23 
10888  2945  2237  1759  1702  1372  1091   817   792   558   553   535   479   398   313   252   232   173   100    91    53    48    46 
   24    25    26 
   38    14    13