kaillahs
commented
4 months ago ** The above issue was accidentally closed - help would still be appreciated with the above questions.
Hi,
I have run into some additional questions regarding the training of my bias model since I last posted this question. I am concerned about the results of the training reports, specifically the training validity loss curves. Below I have described 3 models that I have trained so far as well as the concerns I have about them.
Model 1: This model is training on the total scATAC-seq data set stemming from muscle lysate. I found that the total loss for both the training and the validity loss curves was generally very high (between 400 and 500) and that the separation between the two curves is also concerningly large (60). This makes me think that the model may not be generalizing the training data very well and that the training set size may need to be increased. However, the current (and suggested) training validation split has 3 testing chromosomes, 2 validation chromosomes, and 16 training chromosomes, making it hard to increase the training set.
Model 2: I have tried subsetting the original ATAC-seq data to a specific cell type of interest (while keeping the original split), and this seems to decrease the total loss for the training and validity loss curves (between 230 and 265) but maintains the gap at around 28.
Model 3: Lastly, I decreased model 2's training set to ensure that the model is not learning specific patterns (5 testing, 2 validation, 14 training), but this did not change the curves and worsened other metrics in the QC report.
It is my understanding that the definition of a "good" training validity loss curve varies depending on the project. My concern stems from the fact that the example QC report provided by chromBPNet features a graph with a training loss between 155 and 161 and a gap of only 4.
Though all other metrics adhere to the cut-offs, I am wondering if this may have to do with the fact that my pearsonr value for nonpeaks tends to veer towards the lower end (0.13, 0.29, and 0.00 respectively). If so, how would you suggest fixing this issue?
I have attached the QC reports below for your convenience.
Model_1_QC.pdf Model_2_QC.pdf Model_3_QC.pdf
Thank you!
akundaje
panushri25
Hi,
I am integrating ChromBPNet analysis to refine snATAC-seq data (10x genomics) derived from skeletal muscle lysate. I am running into the following questions when training chromeBPNet:
- Is there a minimum number of cells or read depth necessary for training chromBPNet? Our total number of cells pre-QC filtering is 6305 and 4705 post-QC filtering.
- Would you recommend training chromeBPNet on only our cell population of interest, or all nuclei derived from our tissue lysate? My QC filtered dataset has 3580 cells of my population of interest. I’m inclined to train with a subset of my population of interest only, but am unsure whether my sample is sufficiently large for model training and subsequent analysis.
- How should I handle peak calling files? I have treated (n = 2 bio replicates) and control (n = 1 bio replicates) available. Each of these files has a peak.bed files associated with it. According to Anushri in Issue #117 on github, I should not be using the peak.bed files generated by 10x. Am I correct in understanding that the recommendation is to take the merged.bam file created in the previous step to peak call manually using MACS2? In doing this, I would have given the following command:
!macs2 callpeak -t data/downloads/merged.bam -f BAMPE -n "MACS2Peaks" -g "mm" -p 0.01 --shift -75 --extsize 150 --nomodel -B --SPMR --keep-dup all --call-summits --outdir data/downloads/MACS2PeakCallingPE
as recommended by the ENCODE pipeline with the exception of changing -f input to "BAMPE" instead of "BAM" as we are working with paired-end data. However, Anshul advised against this change in issue #176. Does this mean that I should keep the -f command as "BAM" even though I am working with paired-end data?
- Multiple folds: I would like to confirm my understanding of the usage of multiple folds. As I understand it I should be creating multiple folds (is there a recommended number?) in the splits folder each of which contains a different combination of training and validation chromosomes. I would then train a bias model and chrombpnet model for each fold separately. Later on, when using the tools, I would have to average out the bigwig or h5 files before inputting them into a given tool. Please let me know if this sounds right.
Thank you!