I assigned two GPUs (48 GB) for the jobs. And found some job is skipped all the time even though the required memory is smaller than available memory of one GPU of two. Here is the output:
[__main__ WARNING] Skipping {'name': 'job_1', 'command': "python ~/train.py, 'num_gpus': 1, 'memory': 1024}, not enough GPUs available ({'6': 788, '7': 5381}).
[__main__ WARNING] Skipping {'name': 'job_2', 'command': "python ~/train.py, 'num_gpus': 1, 'memory': 1024}, not enough GPUs available ({'6': 788, '7': 5381}).
[__main__ WARNING] Skipping {'name': 'job_3', 'command': "python ~/train.py, 'num_gpus': 1, 'memory': 1024}, not enough GPUs available ({'6': 788, '7': 5381}).
I assigned two GPUs (48 GB) for the jobs. And found some job is skipped all the time even though the required memory is smaller than available memory of one GPU of two. Here is the output: