why the experiment can not fully use the memory of GPU

[yangqiao33]

Describe the issue: Hi, I am trying to using NNI to optimize the hyperparameters of a training. When I run the experiment in my Linux server, I found the trials that the experiment run cannot fully use the memory of GPU. Normally if I run the python file, it can use full memory and the training is fast, but if I run the python file through nni, the training is slow. please tell me how to fix it. Like the screenshot, the trail by nni only occupy 159MB.

Environment:

• NNI version:
• Training service (local|remote|pai|aml|etc):
• Client OS:
• Server OS (for remote mode only):
• Python version:
• PyTorch/TensorFlow version:
• Is conda/virtualenv/venv used?:
• Is running in Docker?:

Configuration:

• Experiment config (remember to remove secrets!):
• Search space:

Log message:

• nnimanager.log:
• dispatcher.log:
• nnictl stdout and stderr:

How to reproduce it?:

[XDZxdz1]

Hello, I am also trying to use NNI to optimize the training hyperparameters. When I run NNI on the ubantu server, it always shows "failed". Then, I check the log file and it says "GPU context requested, but no GPUs found.". How can I solve this problem? I look forward to your reply very much, and I would greatly appreciate it.