Paper Revision

[lamtung16]

@tdhock this is the outline of my new paper, can you give me some feedbacks

Learning Penalty Parameters for Optimal Partitioning via Automatic Feature Extraction

Abstract

Changepoint detection is a technique used to identify significant shifts in data sequences, which is crucial in various fields such as finance, genomics, and medicine. The Optimal Partitioning (OPART) algorithm locates these changes within a sequence and uses a penalty parameter to control the number of detected changepoints. Traditionally, methods involved manually extracting statistical features from sequences to form feature vectors for predictive models that estimate the penalty value. This study introduces a novel approach that learns the penalty parameter directly from sequences by utilizing recurrent architecture networks to automatically extract relevant features that aid in determining the penalty.

Introduction

• Introduce the concept of changepoint detection and its applications.
• Note that the OPART algorithm operates with a fixed penalty parameter.
• Highlight that prior studies have concentrated on predicting the penalty value using statistical features derived from sequences.
• A significant limitation is that manually extracted features may not always be relevant for predicting the penalty parameter. Researchers have attempted to mitigate this by generating extensive feature sets through transformations such as logarithm, log-log, absolute value, square, and square root. Linear model apply L1 regularization to diminish the influence of irrelevant features, while tree-based methods can handle this automatically by constraining hyperparameters like tree depth or the minimum number of samples required to split.
• This study employs recurrent architecture networks to extract a fixed number of relevant features for penalty prediction, rather than depending on a static feature vector derived from sequences. This approach enhances flexibility and adaptability, enabling the model to better capture data nuances and improve the accuracy of penalty predictions.
• The most commonly utilized RNN architectures for univariate sequence data are Vanilla RNNs, Long Short-Term Memory (LSTM) networks, and Gated Recurrent Units (GRU).

Novelty

• A diagram will illustrate the two steps involved in predicting the penalty from sequences: Step 1 focuses on feature extraction, while Step 2 involves using a predictive model to learn the penalty from the extracted features.
• Instead of treating feature extraction and penalty learning as separate stages, this study integrates them to directly learn the penalty from raw sequences.
• The rationale is that the relevant features from the sequences for predicting the penalty are often unknown. Recurrent networks can automatically extract valuable hidden features that contribute to penalty prediction.

Experiments

• Base architectures: RNN, LSTM, and GRU.
• Configuration: 1 or 2 layers, with hidden sizes of 2, 4, 8, or 16.

[tdhock]

great

[lamtung16]

hi @tdhock, I update my study:

1. So far, I implemented:
   • methods (6 total):
   • use all features as the input (4 total): AFT_XGBoost, linear_L1reg, mmit, mlp
   • use raw sequence as the input (3 total): rnn, lstm, gru
   • datasets (3 total): detailed, systematic, cancer (dataset used in LOPART paper)
   • results: https://github.com/lamtung16/ML_ChangepointDetection/blob/experiment/figures/0.get_plot_acc_include_RNN.ipynb

summary:

• rnn is really bad (long term memory problem)
• lstm is not bad, but worse than the previous methods (worth to show)
• most of the case, gru performs better than the rest

2. I have been trying to experiment in more datasets (17 sub-datasets in https://archive.ics.uci.edu/dataset/439/chipseq)
   • one problem: the length of these sequences is too long (some of them is more than 11 minion), then implement rnn takes forever --> not applicable
   • idea: I think about compressing sequences (e.g., taking mean or median or random on each 1000-length segment --> the length of compressed one is 1000 times shorter than the original one, or something like that), then apply rnn, lstm, gru on compressed ones.

[tdhock]

yes the GRU results look good.

yes the sequences in UCI chipseq are very large.

yes you can try "compressing sequences" which I think would be the same as doing pooling with window size 1000 right?

[lamtung16]

@tdhock some updates today:

• What I meant by "compressing sequences" is, yes, pooling: apply a pooling operation on the raw sequence by segmenting it into non-overlapping windows of n (100, 1000, 2000) elements and computing the mean or median of each segment, producing a pooled sequence that is n times shorter.
• I’ve been training the RNN models on 1 sequence at a time due to their varying lengths. I did try padding shorter sequences with 0s to match the length of the longest sequence, but this approach altered the sequences (padded sequences are not the same as original sequences) and significantly worsened performance. Therefore, I plan to continue training the RNN models one sequence at a time.

Stat	Dataset	Min Seq Length	Max Seq Length	Min Value	Max Value	Mean Variance	Non-Inf Min Low Limit	Non-Inf Max Upper Limit
cancer (1)	39	43628	-6.41	0.075	0.063	-5.75	6.9
detailed (1)	25	5937	-7.67	9.87	0.029	-4.97	6.19
systematic (1)	66	5937	-7.67	9.87	0.027	-4.84	6.19
chipseq (17)	275	11499958	0	31488	11270.19	5.44	20.09

Achievement

• For the 3 datasets—cancer, detailed, and systematic—I have successfully trained RNNs (including original RNNs, LSTMs, and GRUs) from raw sequences to lambda without any issues, and the results have been great. I believe this is because the sequence lengths are relatively short, and the value range is small, fluctuating between negative and positive.

Problem

• For the 17 chipseq datasets:
  • The output of the RNN model is initially too large due to the long sequences and high sequence values, which occasionally leads to the loss value (squared hinge error) becoming NaN (even with the pooled sequences).
  • The cause from the high sequence values: I don't want to modify the values of the pooled sequences to maintain their original form, so I won’t take any further processing steps (further processing can alter the sequences features, I was thinking about shifting the sequence value having mean 0, but what if mean value is a good feature -- doesn't make lot of sense but who know).
  • The cause from the big output lambda from long sequence:
  • I replaced the squared hinge error with linear hinge error, and the issue of NaN values in the loss has nearly disappeared, although it’s not completely resolved.
  • Instead of taking a linear combination of the extracted features z to prevent excessively large outputs, I then constrained the model's output by returning 25 $\times$ sigmoid(z), because from the dataset, I think the output range from 0 to 25 is good enough (most of the censored-interval target have range from 8 to 12). What do you think about this modification, is it cheating? Or to be more fair, I can use 100 $\times$ tanh(z)?
  • After modifying these two aspects (replacing the loss function and applying the squeeze function to the output), I’ve achieved good results so far. The code is still running, but I’ll share the results with you as soon as possible.