PengNi
commented
2 years ago Hi @simonbrd , you can try something like the following steps to train a new model:
# demo cmds for generating training samples
# 1. deepsignal_plant extract (extract features from fast5s)
deepsignal_plant extract --fast5_dir fast5s/ [--corrected_group --basecall_subgroup --reference_path] --methy_label 1 --motifs CG --mod_loc 0 --write_path samples_CG.hc_poses_positive.tsv [--nproc] --positions /path/to/file/contatining/high_confidence/positive/sites.tsv
deepsignal_plant extract --fast5_dir fast5s/ [--corrected_group --basecall_subgroup --reference_path] --methy_label 0 --motifs CG --mod_loc 0 --write_path samples_CG.hc_poses_negative.tsv [--nproc] --positions /path/to/file/contatining/high_confidence/negative/sites.tsv
# 2. randomly select equally number (e.g., 10m) of positive and negative samples
# the selected positive and negative samples then can be combined and used for training, see step 3.
python /path/to/scripts/randsel_file_rows.py --ori_filepath samples_CG.hc_poses_positive.tsv --write_filepath samples_CG.hc_poses_positive.r10m.tsv --num_lines 10000000 --header false &
python /path/to/scripts/randsel_file_rows.py --ori_filepath samples_CG.hc_poses_negative.tsv --write_filepath samples_CG.hc_poses_negative.r10m.tsv --num_lines 10000000 --header false &
# 3. combine positive and negative samples for training
# after combining, the combined file can be splited into two files as training/validating set, see step 4.
python /path/to/scripts/concat_two_files.py --fp1 samples_CG.hc_poses_positive.r10m.tsv --fp2 samples_CG.hc_poses_negative.r10m.tsv --concated_fp samples_CG.hc_poses.r20m.tsv
# 4. split samples for training/validating
# suppose file "samples_CG.hc_poses.r20m.tsv" has 20000000 lines (samples), and we use 200k samples for validation
# the .train.tsv and .valid.tsv can be converted to .bin format to accelerate training (scripts/generate_binary_feature_file.py)
head -19800000 samples_CG.hc_poses.r20m.tsv > samples_CG.hc_poses.r20m.train.tsv
tail -200000 samples_CG.hc_poses.r20m.tsv > samples_CG.hc_poses.r20m.valid.tsv
# 5. train
CUDA_VISIBLE_DEVICES=0 deepsignal_plant train --train_file samples_CG.hc_poses.r20m.train.tsv --valid_file samples_CG.hc_poses.r20m.valid.tsv --model_dir model.deepsignal_plant.CG --display_step 2000We currently have no R10.3 data, so we haven't planned to train an R10.3 model yet.
Best, Peng
simonbrd
DelphIONe
WeipengMO
Hello again, I have been using your tools for some time and now I would like to know more about how you designed your model model.dp2.CNN.arabnrice2-1_120m_R9.4plus_tem. bn13_sn16.both_bilstm.epoch6.ckpt?
For information, I am working on microalgae data and I would also like to make my model on microalgae data ?
Besides, have you also planned to design a model like (model.dp2.CNN.arabnrice2-1_120m_R9.4plus_tem. bn13_sn16.both_bilstm.epoch6.ckpt) the one that is available but for R10.3 data?
Thank you in advance