akundaje
commented
7 years ago You should talk to Joe about this. He spent some time looking into normalization.
Anshul
On Jul 19, 2017 2:47 PM, "Rajiv Movva" notifications@github.com wrote:
SuRE-seq, the previous dataset I was working with, is fundamentally a promoter assay, and therefore it was hard to find many biologically relevant features other than GC content (which affects nucleosome occupancy, and therefore accessibility of the promoter to Pol2 etc.) and some semblance of a TATA box. That said, there are still a few more things I'd like to try with that data, which I outlined here: #2 (comment) https://github.com/kundajelab/mpra/issues/2#issuecomment-316498512.
I've started working with the Sharpr dataset, which is an enhancer MPRA performed in HepG2 and K562 cells. An enhancer is placed upstream of two promoter types - minimal (minP) or strong (SV40P) - and downstream of the promoter is a barcode sequence uniquely mapping to the enhancer. The authors selected DNase peak regions in HepG2, K562, HUVEC, and H1-hESCs (DNase peaks from the latter two cell types are included to create diversity in the types of regions tested, since most of the HepG2 DNase peaks will be activating enhancers in HepG2 cells). 295bp regions are selected centered at the peak summits, and these 295bp regions are tiled with 145bp MPRA constructs, starting at 1bp-145bp, 6bp-150bp, ..., 151bp-295bp (31 constructs per region). Each of these 145bp constructs are used in the MPRA - there are 15720 regions total, so 15720*31 = 487K tested fragments per (cell type, promoter) pair. Each experiment is done in replicate, so there are in total 8 experiments (K562_minP_rep1, K562_minP_rep2, K562_SV40P_rep1, HepG2_minP_rep1, etc.)
To start, I did some quality control to see if this data was much better than SuRE. Here are some figures showing replicate-replicate correlation for the four different experiments (2 cell types, 2 promoters).
For all of these plots, more red corresponds to more input plasmids for the fragment, and more blue corresponds to fewer. Generally, I'd expect red points to be more reproducible between replicates because there's more data corresponding to those fragments (this isn't obviously true, from the plots however).
First, correlation of RNA read counts between replicates (sorry for messy x-labels). The numbers in the top left of the plots are the Spearman correlations.
[image: rnacountsreplicates] https://user-images.githubusercontent.com/17256372/28390988-0c8f0910-6c91-11e7-9044-5fa8ccde1781.png
Looks pretty good, but a lot of that correlation is driven by the amount of plasmid DNA corresponding to each fragment -- i.e., in general, more input plasmids means more RNA output. So let's look at these correlations again, normalizing for the amount of input DNA for each fragment*.
[image: normsignalsreplicates] https://user-images.githubusercontent.com/17256372/28391000-15095ec4-6c91-11e7-9321-45ebf942b139.png
The correlations are significantly worse here, as expected. It varies by experiment, but the biologically reproducible signal is about ~0.40 on average. It's unclear how the amount of DNA plasmid affects the correlation between replicates from the colors of the points here, there's just too much going on. What happens to these correlations if we look at only the fragments with >20 plasmid DNA reads (i.e., they are nontrivially represented in the input library)?
[image: normsignalswith20plasmids] https://user-images.githubusercontent.com/17256372/28391007-1a9c3136-6c91-11e7-9581-983e861923e5.png
Interestingly, the correlations go down for both celltypes with the minimal promoter, but the correlations increase slightly for the SV40 promoter. Not sure why this is / if this is statistically significant, so let's look at the full curve of replicate Spearman correlation vs. the threshold of input plasmid DNA. I would expect these graphs to have a positive correlation; that is, reproducibility should generally increase with higher thresholds of plasmid counts. In this plot, point-size is proportional to the number of fragments considered in the correlation at each threshold.
[image: replicatecorrvsplasmidthresholds] https://user-images.githubusercontent.com/17256372/28391009-1f613d7e-6c91-11e7-9388-74fc4c2f42de.png
Ok, good these graphs have the positive correlation I expected. Not sure why the top two graphs have an initial dip, though.
Anyway, it's still unclear how we can use this information to potentially curate the training set. Let's instead plot the number of fragments corresponding to each point on the previous plot, so we can see if there's a plasmid threshold that allows for a reasonable number of training fragments with a significant boost to data quality (as measured by replicate Spearman correlation).
[image: replicatecorrvsnumfragments] https://user-images.githubusercontent.com/17256372/28391021-26888c9c-6c91-11e7-9a0d-3663db246a57.png
Looks like no. The relationship is basically linear, so any improvement in correlation will require a linearly proportional loss in the amount of training data. So there's no obvious curation to be done with this dataset. An approach to weight training examples based on replicate variance might help here? @akundaje https://github.com/akundaje @pgreenside https://github.com/pgreenside do you think this would help at all?
I've starting training without any curation / weighting / augmentation, and I'll post an update about how that's going later today as well.
*The exact transformation is log2(fold change), and to avoid divide by zero errors, a pseudo read count of 1 is added to both RNA reads and DNA reads. Also, the number of RNA and DNA reads are divided by the total number of reads for the relevant experiment, to get a fraction read count. So the actual quantity is log2((RNA + 1) / sum(RNA reads)) - log2((DNA+1) / sum(DNA reads)). [Note that this difference is equivalent to doing a log of the quotient of the values inside each of the logs, I just wrote it that way for readability.]
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kundajelab/mpra/issues/4, or mute the thread https://github.com/notifications/unsubscribe-auth/AAI7EXD2Ivh6EOfQBYCe33DuyjbiGhByks5sPnmKgaJpZM4OdV2o .
rmovva
SuRE-seq, the previous dataset I was working with, is fundamentally a promoter assay, and therefore it was hard to find many biologically relevant features other than GC content (which affects nucleosome occupancy, and therefore accessibility of the promoter to Pol2 etc.) and some semblance of a TATA box. That said, there are still a few more things I'd like to try with that data, which I outlined here: https://github.com/kundajelab/mpra/issues/2#issuecomment-316498512.
I've started working with the Sharpr dataset, which is an enhancer MPRA performed in HepG2 and K562 cells. An enhancer is placed upstream of two promoter types - minimal (minP) or strong (SV40P) - and downstream of the promoter is a barcode sequence uniquely mapping to the enhancer. The authors selected DNase peak regions in HepG2, K562, HUVEC, and H1-hESCs (DNase peaks from the latter two cell types are included to create diversity in the types of regions tested, since most of the HepG2 DNase peaks will be activating enhancers in HepG2 cells). 295bp regions are selected centered at the peak summits, and these 295bp regions are tiled with 145bp MPRA constructs, starting at 1bp-145bp, 6bp-150bp, ..., 151bp-295bp (31 constructs per region). Each of these 145bp constructs are used in the MPRA - there are 15720 regions total, so 15720*31 = 487K tested fragments per (cell type, promoter) pair. Each experiment is done in replicate, so there are in total 8 experiments (K562_minP_rep1, K562_minP_rep2, K562_SV40P_rep1, HepG2_minP_rep1, etc.)
To start, I did some quality control to see if this data was much better than SuRE. Here are some figures showing replicate-replicate correlation for the four different experiments (2 cell types, 2 promoters).
For all of these plots, more red corresponds to more input plasmids for the fragment, and more blue corresponds to fewer. Generally, I'd expect red points to be more reproducible between replicates because there's more data corresponding to those fragments (this isn't obviously true, from the plots however).
First, correlation of RNA read counts between replicates (sorry for messy x-labels). The numbers in the top left of the plots are the Spearman correlations.
Looks pretty good, but a lot of that correlation is driven by the amount of plasmid DNA corresponding to each fragment -- i.e., in general, more input plasmids means more RNA output. So let's look at these correlations again, normalizing for the amount of input DNA for each fragment*.
The correlations are significantly worse here, as expected. It varies by experiment, but the biologically reproducible signal is about ~0.40 on average. It's unclear how the amount of DNA plasmid affects the correlation between replicates from the colors of the points here, there's just too much going on. What happens to these correlations if we look at only the fragments with >20 plasmid DNA reads (i.e., they are nontrivially represented in the input library)?
Interestingly, the correlations go down for both celltypes with the minimal promoter, but the correlations increase slightly for the SV40 promoter. Not sure why this is / if this is statistically significant, so let's look at the full curve of replicate Spearman correlation vs. the threshold of input plasmid DNA. I would expect these graphs to have a positive correlation; that is, reproducibility should generally increase with higher thresholds of plasmid counts. In this plot, point-size is proportional to the number of fragments considered in the correlation at each threshold.
Ok, good these graphs have the positive correlation I expected. Not sure why the top two graphs have an initial dip, though.
Anyway, it's still unclear how we can use this information to potentially curate the training set. Let's instead plot the number of fragments corresponding to each point on the previous plot, so we can see if there's a plasmid threshold that allows for a reasonable number of training fragments with a significant boost to data quality (as measured by replicate Spearman correlation).
Looks like no. The relationship is basically linear, so any improvement in correlation will require a linearly proportional loss in the amount of training data. So there's no obvious curation to be done with this dataset. An approach to weight training examples based on replicate variance might help here? @akundaje @pgreenside do you think this would help at all?
I've starting training without any curation / weighting / augmentation, and I'll post an update about how that's going later today as well.
*The exact transformation is log2(fold change), and to avoid divide by zero errors, a pseudo read count of 1 is added to both RNA reads and DNA reads. Also, the number of RNA and DNA reads are divided by the total number of reads for the relevant experiment, to get a fraction read count. So the actual quantity is log2((RNA + 1) / sum(RNA reads)) - log2((DNA+1) / sum(DNA reads)). [Note that this difference is equivalent to doing a log of the quotient of the values inside each of the logs, I just wrote it that way for readability.]