ValueError: invalid literal for int() with base 10

[TomTomTommi]

• Discribe the bug

  Traceback (most recent call last):
  File "/gpfs/home/jj323/mambaforge-pypy3/envs/infini/lib/python3.10/runpy.py", line 196, in _run_module_as_main
  return _run_code(code, main_globals, None,
  File "/gpfs/home/jj323/mambaforge-pypy3/envs/infini/lib/python3.10/runpy.py", line 86, in _run_code
  exec(code, run_globals)
  File "/gpfs/home/jj323/infinigen/infinigen/datagen/manage_jobs.py", line 812, in <module>
  main(args)
  File "/gpfs/home/jj323/mambaforge-pypy3/envs/infini/lib/python3.10/site-packages/gin/config.py", line 1605, in gin_wrapper
  utils.augment_exception_message_and_reraise(e, err_str)
  File "/gpfs/home/jj323/mambaforge-pypy3/envs/infini/lib/python3.10/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
  raise proxy.with_traceback(exception.__traceback__) from None
  File "/gpfs/home/jj323/mambaforge-pypy3/envs/infini/lib/python3.10/site-packages/gin/config.py", line 1582, in gin_wrapper
  return fn(*new_args, **new_kwargs)
  File "/gpfs/home/jj323/infinigen/infinigen/datagen/manage_jobs.py", line 675, in main
  manage_datagen_jobs(all_scenes, elapsed=(now-start_time).total_seconds())
  File "/gpfs/home/jj323/mambaforge-pypy3/envs/infini/lib/python3.10/site-packages/gin/config.py", line 1605, in gin_wrapper
  utils.augment_exception_message_and_reraise(e, err_str)
  File "/gpfs/home/jj323/mambaforge-pypy3/envs/infini/lib/python3.10/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
  raise proxy.with_traceback(exception.__traceback__) from None
  File "/gpfs/home/jj323/mambaforge-pypy3/envs/infini/lib/python3.10/site-packages/gin/config.py", line 1582, in gin_wrapper
  return fn(*new_args, **new_kwargs)
  File "/gpfs/home/jj323/infinigen/infinigen/datagen/manage_jobs.py", line 603, in manage_datagen_jobs
  LocalScheduleHandler.instance().poll()
  File "/gpfs/home/jj323/infinigen/infinigen/datagen/util/submitit_emulator.py", line 181, in poll
  total = self.total_resources()
  File "/gpfs/home/jj323/mambaforge-pypy3/envs/infini/lib/python3.10/site-packages/gin/config.py", line 1605, in gin_wrapper
  utils.augment_exception_message_and_reraise(e, err_str)
  File "/gpfs/home/jj323/mambaforge-pypy3/envs/infini/lib/python3.10/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
  raise proxy.with_traceback(exception.__traceback__) from None
  File "/gpfs/home/jj323/mambaforge-pypy3/envs/infini/lib/python3.10/site-packages/gin/config.py", line 1582, in gin_wrapper
  return fn(*new_args, **new_kwargs)
  File "/gpfs/home/jj323/infinigen/infinigen/datagen/util/submitit_emulator.py", line 158, in total_resources
  visible = [int(s.strip()) for s in os.environ[CUDA_VARNAME].split(',')]
  File "/gpfs/home/jj323/infinigen/infinigen/datagen/util/submitit_emulator.py", line 158, in <listcomp>
  visible = [int(s.strip()) for s in os.environ[CUDA_VARNAME].split(',')]
  ValueError: invalid literal for int() with base 10: 'GPU-c5aa4d0e-51b8-94f4-8799-3f17f14b64ea'
  In call to configurable 'total_resources' (<function LocalScheduleHandler.total_resources at 0x14f7115e6c20>)
  In call to configurable 'manage_datagen_jobs' (<function manage_datagen_jobs at 0x14f7113f7370>)
  In call to configurable 'main' (<function main at 0x14f7113f7250>)

• Steps to Reproduce python -m infinigen.datagen.manage_jobs --output_folder outputs/my_videos --num_scenes 50 --pipeline_config local_128GB.gin monocular_video cuda_terrain opengl_gt --cleanup big_files --warmup_sec 60000 --config video high_quality_terrain

I tried to comment the code in infinigen/infinigen/datagen/util/submitit_emulator.py from line 163~166 (link) and it fixed. I wonder if this is a correct solution.

[araistrick]

It seems your CUDA_VISIBLE_DEVICES contains long strings like GPU-c5aa4d0e-51b8-94f4-8799-3f17f14b64ea where I (incorrectly) had assumed it was only integers. Could you try removing the int(..) from int(s.strip()) so it just looks like [s.strip() for s in os.environ[CUDA_VARNAME].split(',')]? If this works for you I will patch it into main. I should probably find and follow some kind of specification for what these are allowed to look like.

[TomTomTommi]

Yes. This works.