search

gzerveas / mvts_transformer

Multivariate Time Series Transformer, public version
MIT License
750 stars 173 forks source link

inquiry on program being killed #63

Closed harrykwon0524 closed 10 months ago

harrykwon0524 commented 10 months ago

Hello i was currently working on the code with the customed dataset that i have and the dataset contains around 380000 samples. I did customize the csv dataset that i have to match the example dataset you used, such as SpokenArabicDigits and created ts files. And as i run the code for classification, the program stops running and says killed. Should i assume the dataset is too big for the code to handle?

kill
gzerveas commented 10 months ago

Hi Hyungtaek,

Can you post the exact command you are running? The code itself doesn't kill the process, but it is possible that e.g. the environment you are running it on (e.g. SLURM) limits the available memory and will kill it (unless you set it up by requesting more memory).

With the --limit_size 0.1 option, the code allows you to use only a subset of your dataset (here, 10%). You can try it and see if everything runs alright; if it does, then you'll have to configure your environment accordingly (e.g. -m 40G with SLURM).

harrykwon0524 commented 10 months ago

the command is as follows: python src/main.py --output_dir experiments --comment "classification from Scratch" --name orange_fromScratch --records_file Classification_records.xls --data_dir src/datasets/orange/ --data_class tsra --pattern TRAIN --val_pattern TEST --epochs 400 --lr 0.001 --optimizer RAdam --pos_encoding learnable --task classification --key_metric accuracy and this was run on WSL with 16gb ram and NVIDIA GeForce RTX 2050. Are you suggesting that the code itself should not create such errors, but the specifications may be an issue? And I should try to change the parameters accordingly?