annashcherbina
commented
6 years ago I have trained a model on V576 DNAse with the following approach:
-
Begin with Daniel's DNAse peak set from all ENCODE cell types. Remove peaks from colorectal cancer, intestines, and colorectal cancer cell lines.
-
Extract 1kb regions centered at the peak summits.
-
Extract 1kb regions centered at the peak summits in the V576 DNAse dataset. Label those as positive. Label the adjoining 1kb regions as ambiguous (i.e. remove from training).
-
Subtact the 3 kb regions (1kb positive + 2kb ambiguous) in (3) from the combined ENCODE regions in (2). The "difference" regions form the negative set.
-
Sample the negative set from (4) to achieve a ratio of 10 negatives : 1 positive. (Note to self: the dataset was generate using utils functions from this repository: https://github.com/kundajelab/anna_utils/tree/master/seq_utils)
-
Train Basset architecture, initializing from ENCODE weights:
The validation loss did not decrease.
- Train Basset architecture, initializing from random weights:
The validation loss again did not decrease
- Apply Dropout of 0.8 after both dense layers; lowered learning rate from 0.001 to 0.0001 (https://github.com/kundajelab/deeplearning/blob/annashch-branch/gecco/basset_architecture_single_task_RegularizedDropout.py)
-
Added strong L1=0.01 & L2 = 0.01 regularization after 1st and last convolution layer, after 1st dense layer (https://github.com/kundajelab/deeplearning/blob/annashch-branch/gecco/basset_architecture_single_task_RegularizedL1L2.py)
The model is no longer overfitting to the training data, but the performance on validation data was poorer than the original baseline (see table below)
- Combined Dropout of 0.8 after 1st and last convolution layer and Regularization L1=0.0001 L2=0.0001 on first and last convolution layer; used ELU activations in between convolution layers: .
Model performance was:
Next steps: Use GC & Dinucleotide balanced negatives hyperparameter search on optimal dropout and regularization values.
akundaje
The current negative sets are leading the model to overfit on training data and to generalize poorly to validation data. The current negative sets are generated by identifying peaks that are absent in a specific CRC sample but present in one or more other CRC samples. In addition, genome regions that are not accessible for any ENCODE cell type are used as negatives. This is not working, most likely because the CRC samples are too similar to one another so peaks present in one sample and absent in another are not answering our question of interest.
Next Steps:
All to be repeated in 3 -folds to get variance/ model stability information:
GC-matched negatives from the genome (per Akshay's code)
Dinucleotide -matched negatives from the genome (also per Akshay's code)
Universal negatives -- these are peaks that are present in non-CRC tissues in ENCODE/Roadmap, but absent in all CRC samples
Whole genome training w/ GenomeLake
balanced batches for each negative set
Combine (1-4) above via curriculum learning https://ronan.collobert.com/pub/matos/2009_curriculum_icml.pdf