E1.3.2: encoder_depth=4 the best (surprisingly good sentence percentage)
E1.4.x / E1.5.x: increasing encoder_depth with decreasing hidden_dim as a possible approach to reduce parameters while maintaining okayish evaluation results
E2 (vs E1): optical flow helps when model depth is 1 (shallow)
E2.1 (vs E1.3.2): optical flow helps less or even not when model depth increases - perhaps the model learns to calculate optical flow?
E3.x (vs E1 / E1.3.2): hand normalization alone is not useful
E4.x (vs E2.1 / E1.3.2): hand normalization also does not help optical flow further - @AmitMY any way to dig into and improve it, e.g., make a handshape change more explicit?
E5 (vs E1.3.2): face keypoints do not help much (only difference in the sentence percentage)
E5.1 (vs E5): no need to use full face keypoints, instead use the reduced set
@AmitMY please add if you spot anything else.
Next steps (mainly on improving the BIO prediction):
implement and try auto-regressive models @J22Melody
smoothing layer on BIO probabilities / smooth loss function @AmitMY @J22Melody
advanced decoding strategy @AmitMY
bring back E0 with IO tagging and optical flow only @AmitMY - low priority since we can also state the intuition of BIO tagging in the paper
perhaps we can also have an experiments that uses only optical flow but BIO tagging to justify the use of optical flow
Systematically run many experiments, results saved at:
https://github.com/J22Melody/transcription/blob/experiments/pose_to_segments/src/summary.csv
Observations:
hidden_dimalone does not helpencoder_depthalone helpsencoder_depth=4the best (surprisingly good sentence percentage)encoder_depthwith decreasinghidden_dimas a possible approach to reduce parameters while maintaining okayish evaluation results@AmitMY please add if you spot anything else.
Next steps (mainly on improving the BIO prediction):
@AmitMY please also add more ideas if any.