HDD data segmentation

[Doha7430]

Hi! I noticed that your code is somewhat different from TRN's in HDD's data segmentation, this maybe because frames are classified in different way in two models. TRN:

for start, end in zip(
    range(seed, target.shape[0] - self.dec_steps, self.enc_steps),
    range(seed + self.enc_steps, target.shape[0] - self.dec_steps, self.enc_steps)):
    enc_target = target[start:end]
    dec_target = self.get_dec_target(target[start:end + self.dec_steps])
    self.inputs.append([
        session, start, end, sensor[start:end],
        enc_target, dec_target,
        ])

Yours:

for start, end in zip(
        range(seed, target.shape[0], 1),  # self.enc_steps
        range(seed + self.enc_steps, target.shape[0]-self.dec_steps, 1)):
    enc_target = target[start:end]
    # dec_target = self.get_dec_target(target[start:end + self.dec_steps])
    dec_target = target[end:end + self.dec_steps]
    distance_target, class_h_target = self.get_distance_target(target[start:end])
    if class_h_target.argmax() != 21:
        self.inputs.append([
            session, start, end, enc_target, distance_target, class_h_target, dec_target
        ])

Will the differences in data segmentation affect the model?

[wangxiang1230]

Hi! I noticed that your code is somewhat different from TRN's in HDD's data segmentation, this maybe because frames are classified in different way in two models. TRN:

for start, end in zip(
    range(seed, target.shape[0] - self.dec_steps, self.enc_steps),
    range(seed + self.enc_steps, target.shape[0] - self.dec_steps, self.enc_steps)):
    enc_target = target[start:end]
    dec_target = self.get_dec_target(target[start:end + self.dec_steps])
    self.inputs.append([
        session, start, end, sensor[start:end],
        enc_target, dec_target,
        ])

Yours:

for start, end in zip(
        range(seed, target.shape[0], 1),  # self.enc_steps
        range(seed + self.enc_steps, target.shape[0]-self.dec_steps, 1)):
    enc_target = target[start:end]
    # dec_target = self.get_dec_target(target[start:end + self.dec_steps])
    dec_target = target[end:end + self.dec_steps]
    distance_target, class_h_target = self.get_distance_target(target[start:end])
    if class_h_target.argmax() != 21:
        self.inputs.append([
            session, start, end, enc_target, distance_target, class_h_target, dec_target
        ])

Will the differences in data segmentation affect the model?

Hi, there are indeed some differences here, because our training style is different but has no effect on the final performance evaluation.

[Doha7430]

Thank you for reply. In fact, I have also tried data segmentation similar to yours on HDD and found that using this data segmentation method for training can get better performance evaluation, even if training on a simple model can be better than TRN. Does this seem unfair?

[wangxiang1230]

Thank you for reply. In fact, I have also tried data segmentation similar to yours on HDD and found that using this data segmentation method for training can get better performance evaluation, even if training on a simple model can be better than TRN. Does this seem unfair?

This is an interesting question that we haven't noticed before. We didn't reproduce TRN on HDD before, we used the same training manner as IDN, because this data processing is in line with the training of our model. Meanwhile, if TRN uses this split manner, it seems that training and validation will be very slow.