search

huawei-noah / vega

AutoML tools chain
http://www.noahlab.com.hk/opensource/vega/
Other
842 stars 175 forks source link

Slow processing #122

Closed vanessasidrim closed 3 years ago

vanessasidrim commented 3 years ago

I have three GPUs with 48GB, but when it started running in parallel only 28% of the capacity is consumed from one of them and only 2% from the other two.

What would be the reason that the maximum amount of processing is not being used? Do I need to do some configuration to use all the processing?

zhangjiajin commented 3 years ago

@vanessasidrim

The CARS, Darts, and Modnas algorithms do not support parallelism. Only one GPU is running.

vanessasidrim commented 3 years ago

I'm running SP-NAS.

I defined the configuration in the spnas.yml file as follows:

general:
     parallel_search: True
     parallel_fully_train: True
     devices_per_trainer: 3
     cluster:
           master_ip: 0.0.0.0
           listen_port: 9999

But the execution doesn't use all my processing.

zhangjiajin commented 3 years ago

@vanessasidrim

Modify the configuration as follows:

general:
   parallel_search: True
   parallel_fully_train: True
#   devices_per_trainer: 3
#   cluster:
#         master_ip: 0.0.0.0
#         listen_port: 9999