search

gzerveas / mvts_transformer

Multivariate Time Series Transformer, public version
MIT License
752 stars 173 forks source link

Test result in multivariate dataset without pretrain #19

Closed Mingzhe-Han closed 2 years ago

Mingzhe-Han commented 2 years ago

Hi,

I'm trying to study your code for multivariate classification dataset without pretrain, and I choose Handwriting for an example.

In order to achieve the paper's performance , I used the hyperparameters which is shown in your paper.

So I train the model with command below.

python src/main.py --output_dir experiments --comment "classification from Scratch" --name HW --records_file Classification_records.xls --data_dir data/Multivariate_ts/Handwriting --data_class tsra --pattern TRAIN --epochs 400 --lr 0.001 --optimizer RAdam --pos_encoding learnable --task classification --key_metric accuracy --val_ratio 0.2 --num_layers 3 --num_heads 16 --d_model 128 --dim_feedforward 256 --batch_size 128

And then I test the model with command below.

python src/main.py --output_dir experiments --comment "classification from Scratch" --name HW --records_file Classification_records.xls --data_dir data/Multivariate_ts/Handwriting --data_class tsra --pattern TRAIN --epochs 400 --lr 0.001 --optimizer RAdam --pos_encoding learnable --task classification --key_metric accuracy --val_ratio 0 --num_layers 3 --num_heads 16 --d_model 128 --dim_feedforward 256 --batch_size 128 --test_pattern TEST --test_only testset --load_model experiments/HW_2022-07-27_20-01-05_axV/checkpoints/model_best.pth

I thought I use the same data split, same model and same hyperparameters, but finally I find the acc is 0.25882352941176473 and it's different from 0.3 in paper. Is there any step I missed?

Mingzhe-Han commented 2 years ago

After reading https://github.com/gzerveas/mvts_transformer/issues/11, I search best epoch and train the model with such epoch and the whole train set, the result can be 0.316 now.

Guanyunlph commented 1 year ago

After reading #11, I search best epoch and train the model with such epoch and the whole train set, the result can be 0.316 now.

I encountered the same problem as you did, that is, I couldn't get results similar to the paper. How did you solve it? If you could describe it in detail, it would greatly help me. Thank you.