search

epic-kitchens / epic-kitchens-slowfast

Other
28 stars 15 forks source link

Graphics card memory overflow during the test #6

Closed em0il closed 3 years ago

em0il commented 3 years ago

Hardware: System 1: Nvidia GTX 970 (4GB memory) System 2: Nvidia RTX 2080 (11GB memory) (these are two separate systems).

I tried running the tests with the following command:

python tools/run_net.py --cfg configs/EPIC-KITCHENS/SLOWFAST_8x8_R50.yaml NUM_GPUS 1 OUTPUT_DIR /path/to/experiment_dir EPICKITCHENS.VISUAL_DATA_DIR /path/to/dataset EPICKITCHENS.ANNOTATIONS_DIR /path/to/annotations TRAIN.ENABLE False TEST.ENABLE True TEST.CHECKPOINT_FILE_PATH /path/to/experiment_dir/checkpoints/checkpoint_best.pyth EPICKITCHENS.TEST_LIST EPIC_100_test_timestamps.pkl EPICKITCHENS.TEST_SPLIT test

(I have adapted the paths to my systems in each case. The Annotation repository, SlowFast repository and SlowFast model were all downloaded on 2021-05-18.)

When trying to run the test on system 1 it aborted with "Killed". Thereupon I shortened the EPIC_100_test_timestamps.pkl to one annotation. When running the test with the shortened pickle file I got this error (see picture). Screenshot from 2021-05-18 09-35-51 I thought I needed more graphics card memory so I switched to the System 2. When running the test with the original data, it was also terminated with "Killed". When running the test with the shortened Pickel file, I received the same error as on system 1 (see picture). Bildschirmfoto vom 2021-05-18 13-07-50

What is the reason for this? Is there a problem loading the data?

ekazakos commented 3 years ago

Hi,

Thanks for using our code. Can you try setting TEST.BATCH_SIZE 1 and let me know if it solves your issue?

em0il commented 3 years ago

Thank you, the test can now be run on system 2. system 1 seems to be too weak for this task. Thank you for your quick reply. I wish you good luck for the challenge.