pynvml in GPU memory profiling error

Getting this error on Ubuntu 20.04 running on WSL2:

Issue originating in gpu_max_mem.py

Traceback (most recent call last):
  File "/home/jshier/anaconda3/envs/hear-eval/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/jshier/anaconda3/envs/hear-eval/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/jshier/development/neuralaudio/hear2021-eval-kit/heareval/embeddings/runner.py", line 131, in <module>
    runner()
  File "/home/jshier/anaconda3/envs/hear-eval/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/home/jshier/anaconda3/envs/hear-eval/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/home/jshier/anaconda3/envs/hear-eval/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/jshier/anaconda3/envs/hear-eval/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/home/jshier/development/neuralaudio/hear2021-eval-kit/heareval/embeddings/runner.py", line 110, in runner
    gpu_max_mem_used = gpu_max_mem.measure()
  File "/home/jshier/development/neuralaudio/hear2021-eval-kit/heareval/gpu_max_mem.py", line 24, in measure
    h = nvmlDeviceGetHandleByIndex(0)
  File "/home/jshier/anaconda3/envs/hear-eval/lib/python3.8/site-packages/pynvml/nvml.py", line 1576, in nvmlDeviceGetHandleByIndex
    _nvmlCheckReturn(ret)
  File "/home/jshier/anaconda3/envs/hear-eval/lib/python3.8/site-packages/pynvml/nvml.py", line 743, in _nvmlCheckReturn
    raise NVMLError(ret)
pynvml.nvml.NVMLError_Unknown: Unknown Error

Originally posted by @jorshi in https://github.com/neuralaudio/hear-eval-kit/issues/277#issuecomment-908917960

hearbenchmark / hear-eval-kit

pynvml in GPU memory profiling error #278