Training to slow

[zhr01]

training too slow

According to tutorial，I use command lines to train lanegcn model， below is how I setup the exp:

    python nuplan/planning/script/run_training.py \
        experiment_name=vector_experiment \
        py_func=train \
        +training=training_vector_model \
        scenario_builder=nuplan \
        scenario_filter.limit_total_scenarios=0.1 \
        lightning.trainer.params.max_epochs=100 \
        data_loader.params.batch_size=8 \
        data_loader.params.num_workers=4

It took me almost 2 DAY to run 11 batch,

Setup

• 64 core CPU
• 8 Tesla V100
• 250GB memory

[JingyuQian]

Hello, Did you cache training data? If you always compute features on the go, it'll take you a very long time.

[zhr01]

Thanks for your reply. I think the mem of my server is not enough for data preprocessing. I reduce the value of batch_size, num_workers，and 'Number of threads across all nodes', the code runs without problem. It's just too slow