How to specify GPUs?

[zitianlll]

Describe the feature you would like to see implemented

I want to run the generation job on the specified gpus(such as 4,5,6,7) of our 8*a100 sever, while the commdscreen python -m infinigen.datagen.manage_jobs --output_folder outputs/my_videos --num_scenes 100 \ --pipeline_config local_256GB monocular_video cuda_terrain blender_gt \ --cleanup big_files --warmup_sec 60000 --config video high_quality_terrain would assign each tasks to each GPU. I have tried the code such as CUDA_VISIBLE_DEVICES=4, but it seems to be useless.

[araistrick]

Hello,

local_256GB is intended to respect CUDA_VISIBLE_DEVICES, although the current implementation would expect a comma separated list of integers IE CUDA_VISIBLE_DEVICES=4,5,6,7. The code which parses this is here

The logs should also contain what setting of CUDA_VISIBLE_DEVICES each job actually observed, which gets printed by this line. Please check the results of this print statement in your coarse.out logs and let me know what it says so we can continue with debugging

[zitianlll]

I just ran this command CUDA_VISIBLE_DEVICES=4,5,6,7 screen python -m infinigen.datagen.manage_jobs --output_folder outputs/videos_20240405 --num_scenes 100 \ --pipeline_config local_256GB monocular_video cuda_terrain blender_gt \ --cleanup big_files --warmup_sec 60000 --config video high_quality_terrain

But it still used an unspecified GPU, while the specified GPUs were idle:

coarse.out:

The following command is sent: "nice -n 20 /home/zitian_he/zitian_he/anaconda3/envs/infinigen/bin/python -m infinigen_examples.generate_nature -- --output_folder /mnt1/public/zitian_he/infinigen/outputs/videos_20240405/cdf5cfd/coarse --seed cdf5cfd --task coarse --task_uniqname coarse -g river video high_quality_terrain -p LOG_DIR='/mnt1/public/zitian_he/infinigen/outputs/videos_20240405/cdf5cfd/logs' execute_tasks.frame_range=[1, 192] execute_tasks.camera_id=[0, 0]" keep_placeholder=False placeholder.name='TreeFlowerFactory(6065525).spawn_placeholder(0)' list(placeholder.children)=[] obj.name='TreeFlowerFactory(6065525).spawn_asset(0)' list(obj.children)=[] keep_placeholder=False placeholder.name='TreeFlowerFactory(6065525).spawn_placeholder(1)' list(placeholder.children)=[] obj.name='TreeFlowerFactory(6065525).spawn_asset(1)' list(obj.children)=[] keep_placeholder=False placeholder.name='TreeFlowerFactory(6065525).spawn_placeholder(2)' list(placeholder.children)=[] obj.name='TreeFlowerFactory(6065525).spawn_asset(2)' list(obj.children)=[] keep_placeholder=False placeholder.name='TreeFlowerFactory(6065525).spawn_placeholder(3)' list(placeholder.children)=[] obj.name='TreeFlowerFactory(6065525).spawn_asset(3)' list(obj.children)=[] keep_placeholder=False placeholder.name='TreeFlowerFactory(6065525).spawn_placeholder(4)' list(placeholder.children)=[] obj.name='TreeFlowerFactory(6065525).spawn_asset(4)' list(obj.children)=[] Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected keep_placeholder=False placeholder.name='GenericTreeFactory(6065525).spawn_placeholder(0)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(6065525).spawn_asset(0)']] obj.name='GenericTreeFactory(6065525).spawn_asset(0)' list(obj.children)=[] Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected keep_placeholder=False placeholder.name='GenericTreeFactory(6065525).spawn_placeholder(1)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(6065525).spawn_asset(1)']] obj.name='GenericTreeFactory(6065525).spawn_asset(1)' list(obj.children)=[] keep_placeholder=False placeholder.name='FruitFactoryDurian(6065525).spawn_placeholder(0)' list(placeholder.children)=[] obj.name='FruitFactoryDurian(6065525).spawn_asset(0)' list(obj.children)=[] keep_placeholder=False placeholder.name='FruitFactoryDurian(6065525).spawn_placeholder(1)' list(placeholder.children)=[] obj.name='FruitFactoryDurian(6065525).spawn_asset(1)' list(obj.children)=[] keep_placeholder=False placeholder.name='FruitFactoryDurian(6065525).spawn_placeholder(2)' list(placeholder.children)=[] obj.name='FruitFactoryDurian(6065525).spawn_asset(2)' list(obj.children)=[] keep_placeholder=False placeholder.name='FruitFactoryDurian(6065525).spawn_placeholder(3)' list(placeholder.children)=[] obj.name='FruitFactoryDurian(6065525).spawn_asset(3)' list(obj.children)=[] keep_placeholder=False placeholder.name='FruitFactoryDurian(6065525).spawn_placeholder(4)' list(placeholder.children)=[] obj.name='FruitFactoryDurian(6065525).spawn_asset(4)' list(obj.children)=[] keep_placeholder=False placeholder.name='BranchFactory(6065525).spawn_placeholder(0)' list(placeholder.children)=[] obj.name='BranchFactory(6065525).spawn_asset(0)' list(obj.children)=[] keep_placeholder=False placeholder.name='BranchFactory(6065525).spawn_placeholder(1)' list(placeholder.children)=[] obj.name='BranchFactory(6065525).spawn_asset(1)' list(obj.children)=[] keep_placeholder=False placeholder.name='LeafFactoryBroadleaf(2067777).spawn_placeholder(0)' list(placeholder.children)=[] obj.name='LeafFactoryBroadleaf(2067777).spawn_asset(0)' list(obj.children)=[] keep_placeholder=False placeholder.name='LeafFactoryBroadleaf(2067777).spawn_placeholder(1)' list(placeholder.children)=[] obj.name='LeafFactoryBroadleaf(2067777).spawn_asset(1)' list(obj.children)=[] keep_placeholder=False placeholder.name='LeafFactoryBroadleaf(2067777).spawn_placeholder(2)' list(placeholder.children)=[] obj.name='LeafFactoryBroadleaf(2067777).spawn_asset(2)' list(obj.children)=[] keep_placeholder=False placeholder.name='LeafFactoryBroadleaf(2067777).spawn_placeholder(3)' list(placeholder.children)=[] obj.name='LeafFactoryBroadleaf(2067777).spawn_asset(3)' list(obj.children)=[] keep_placeholder=False placeholder.name='LeafFactoryBroadleaf(2067777).spawn_placeholder(4)' list(placeholder.children)=[] obj.name='LeafFactoryBroadleaf(2067777).spawn_asset(4)' list(obj.children)=[] Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected Error: Evaluated geometry from modifier does not contain a mesh Warning: No mesh data to join keep_placeholder=False placeholder.name='GenericTreeFactory(2067777).spawn_placeholder(0)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(2067777).spawn_asset(0)']] obj.name='GenericTreeFactory(2067777).spawn_asset(0)' list(obj.children)=[] Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected Error: Evaluated geometry from modifier does not contain a mesh Warning: No mesh data to join keep_placeholder=False placeholder.name='GenericTreeFactory(2067777).spawn_placeholder(1)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(2067777).spawn_asset(1)']] obj.name='GenericTreeFactory(2067777).spawn_asset(1)' list(obj.children)=[] keep_placeholder=False placeholder.name='FruitFactoryCoconutgreen(2067777).spawn_placeholder(0)' list(placeholder.children)=[] obj.name='FruitFactoryCoconutgreen(2067777).spawn_asset(0)' list(obj.children)=[] keep_placeholder=False placeholder.name='FruitFactoryCoconutgreen(2067777).spawn_placeholder(1)' list(placeholder.children)=[] obj.name='FruitFactoryCoconutgreen(2067777).spawn_asset(1)' list(obj.children)=[] keep_placeholder=False placeholder.name='FruitFactoryCoconutgreen(2067777).spawn_placeholder(2)' list(placeholder.children)=[] obj.name='FruitFactoryCoconutgreen(2067777).spawn_asset(2)' list(obj.children)=[] keep_placeholder=False placeholder.name='FruitFactoryCoconutgreen(2067777).spawn_placeholder(3)' list(placeholder.children)=[] obj.name='FruitFactoryCoconutgreen(2067777).spawn_asset(3)' list(obj.children)=[] keep_placeholder=False placeholder.name='FruitFactoryCoconutgreen(2067777).spawn_placeholder(4)' list(placeholder.children)=[] obj.name='FruitFactoryCoconutgreen(2067777).spawn_asset(4)' list(obj.children)=[] keep_placeholder=False placeholder.name='BranchFactory(2067777).spawn_placeholder(0)' list(placeholder.children)=[] obj.name='BranchFactory(2067777).spawn_asset(0)' list(obj.children)=[] keep_placeholder=False placeholder.name='BranchFactory(2067777).spawn_placeholder(1)' list(placeholder.children)=[] obj.name='BranchFactory(2067777).spawn_asset(1)' list(obj.children)=[] keep_placeholder=False placeholder.name='LeafFactory(1751696839).spawn_placeholder(0)' list(placeholder.children)=[] obj.name='LeafFactory(1751696839).spawn_asset(0)' list(obj.children)=[] keep_placeholder=False placeholder.name='LeafFactory(1751696839).spawn_placeholder(1)' list(placeholder.children)=[] obj.name='LeafFactory(1751696839).spawn_asset(1)' list(obj.children)=[] keep_placeholder=False placeholder.name='LeafFactory(1751696839).spawn_placeholder(2)' list(placeholder.children)=[] obj.name='LeafFactory(1751696839).spawn_asset(2)' list(obj.children)=[] Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected keep_placeholder=False placeholder.name='GenericTreeFactory(1751696839).spawn_placeholder(0)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(1751696839).spawn_asset(0)']] obj.name='GenericTreeFactory(1751696839).spawn_asset(0)' list(obj.children)=[] Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected keep_placeholder=False placeholder.name='GenericTreeFactory(1751696839).spawn_placeholder(1)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(1751696839).spawn_asset(1)']] obj.name='GenericTreeFactory(1751696839).spawn_asset(1)' list(obj.children)=[] Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected keep_placeholder=False placeholder.name='GenericTreeFactory(1751696839).spawn_placeholder(2)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(1751696839).spawn_asset(2)']] obj.name='GenericTreeFactory(1751696839).spawn_asset(2)' list(obj.children)=[] split obj.name='OpaqueTerrain' into inview area 1090.53 and outofview area 0.00 split obj.name='OpaqueTerrain' into inview area 697.52 and outofview area 0.00 keep_placeholder=False placeholder.name='BlenderRockFactory(64661).spawn_placeholder(0)' list(placeholder.children)=[] obj.name='BlenderRockFactory(64661).spawn_asset(0)' list(obj.children)=[] keep_placeholder=False placeholder.name='BlenderRockFactory(64661).spawn_placeholder(1)' list(placeholder.children)=[] obj.name='BlenderRockFactory(64661).spawn_asset(1)' list(obj.children)=[] keep_placeholder=False placeholder.name='BlenderRockFactory(64661).spawn_placeholder(2)' list(placeholder.children)=[] obj.name='BlenderRockFactory(64661).spawn_asset(2)' list(obj.children)=[] keep_placeholder=False placeholder.name='BlenderRockFactory(64661).spawn_placeholder(3)' list(placeholder.children)=[] obj.name='BlenderRockFactory(64661).spawn_asset(3)' list(obj.children)=[] keep_placeholder=False placeholder.name='BlenderRockFactory(64661).spawn_placeholder(4)' list(placeholder.children)=[] obj.name='BlenderRockFactory(64661).spawn_asset(4)' list(obj.children)=[] 50 scatter:BlenderRockFactory(64661) instances created. keep_placeholder=False placeholder.name='LeafFactoryV2(25497).spawn_placeholder(0)' list(placeholder.children)=[] obj.name='LeafFactoryV2(25497).spawn_asset(0)' list(obj.children)=[] keep_placeholder=False placeholder.name='LeafFactoryV2(25497).spawn_placeholder(1)' list(placeholder.children)=[] obj.name='LeafFactoryV2(25497).spawn_asset(1)' list(obj.children)=[] keep_placeholder=False placeholder.name='LeafFactoryV2(25497).spawn_placeholder(2)' list(placeholder.children)=[] obj.name='LeafFactoryV2(25497).spawn_asset(2)' list(obj.children)=[] keep_placeholder=False placeholder.name='LeafFactoryV2(25497).spawn_placeholder(3)' list(placeholder.children)=[] obj.name='LeafFactoryV2(25497).spawn_asset(3)' list(obj.children)=[] keep_placeholder=False placeholder.name='LeafFactoryV2(25497).spawn_placeholder(4)' list(placeholder.children)=[] obj.name='LeafFactoryV2(25497).spawn_asset(4)' list(obj.children)=[] 2133 scatter:LeafFactoryV2(25497) instances created. Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected keep_placeholder=False placeholder.name='GenericTreeFactory(23779).spawn_placeholder(0)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(23779).spawn_asset(0)']] obj.name='GenericTreeFactory(23779).spawn_asset(0)' list(obj.children)=[] Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected keep_placeholder=False placeholder.name='GenericTreeFactory(23779).spawn_placeholder(1)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(23779).spawn_asset(1)']] obj.name='GenericTreeFactory(23779).spawn_asset(1)' list(obj.children)=[] Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected keep_placeholder=False placeholder.name='GenericTreeFactory(23779).spawn_placeholder(2)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(23779).spawn_asset(2)']] obj.name='GenericTreeFactory(23779).spawn_asset(2)' list(obj.children)=[] Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected keep_placeholder=False placeholder.name='GenericTreeFactory(23779).spawn_placeholder(3)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(23779).spawn_asset(3)']] obj.name='GenericTreeFactory(23779).spawn_asset(3)' list(obj.children)=[] Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected keep_placeholder=False placeholder.name='GenericTreeFactory(23779).spawn_placeholder(4)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(23779).spawn_asset(4)']] obj.name='GenericTreeFactory(23779).spawn_asset(4)' list(obj.children)=[] Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected keep_placeholder=False placeholder.name='GenericTreeFactory(23779).spawn_placeholder(5)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(23779).spawn_asset(5)']] obj.name='GenericTreeFactory(23779).spawn_asset(5)' list(obj.children)=[] Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected keep_placeholder=False placeholder.name='GenericTreeFactory(23779).spawn_placeholder(6)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(23779).spawn_asset(6)']] obj.name='GenericTreeFactory(23779).spawn_asset(6)' list(obj.children)=[] Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected keep_placeholder=False placeholder.name='GenericTreeFactory(23779).spawn_placeholder(7)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(23779).spawn_asset(7)']] obj.name='GenericTreeFactory(23779).spawn_asset(7)' list(obj.children)=[] Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected keep_placeholder=False placeholder.name='GenericTreeFactory(23779).spawn_placeholder(8)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(23779).spawn_asset(8)']] obj.name='GenericTreeFactory(23779).spawn_asset(8)' list(obj.children)=[] Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected Info: Applied modifier was not first, result may not be as expected keep_placeholder=False placeholder.name='GenericTreeFactory(23779).spawn_placeholder(9)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(23779).spawn_asset(9)']] obj.name='GenericTreeFactory(23779).spawn_asset(9)' list(obj.children)=[] 53 scatter:GenericTreeFactory(23779) instances created. 33 scatter:chopped_tree instances created. keep_placeholder=False placeholder.name='GrassTuftFactory(89086).spawn_placeholder(0)' list(placeholder.children)=[] obj.name='GrassTuftFactory(89086).spawn_asset(0)' list(obj.children)=[] keep_placeholder=False placeholder.name='GrassTuftFactory(89086).spawn_placeholder(1)' list(placeholder.children)=[] obj.name='GrassTuftFactory(89086).spawn_asset(1)' list(obj.children)=[] keep_placeholder=False placeholder.name='GrassTuftFactory(89086).spawn_placeholder(2)' list(placeholder.children)=[] obj.name='GrassTuftFactory(89086).spawn_asset(2)' list(obj.children)=[] keep_placeholder=False placeholder.name='GrassTuftFactory(89086).spawn_placeholder(3)' list(placeholder.children)=[] obj.name='GrassTuftFactory(89086).spawn_asset(3)' list(obj.children)=[] keep_placeholder=False placeholder.name='GrassTuftFactory(89086).spawn_placeholder(4)' list(placeholder.children)=[] obj.name='GrassTuftFactory(89086).spawn_asset(4)' list(obj.children)=[] keep_placeholder=False placeholder.name='GrassTuftFactory(89086).spawn_placeholder(5)' list(placeholder.children)=[] obj.name='GrassTuftFactory(89086).spawn_asset(5)' list(obj.children)=[] keep_placeholder=False placeholder.name='GrassTuftFactory(89086).spawn_placeholder(6)' list(placeholder.children)=[] obj.name='GrassTuftFactory(89086).spawn_asset(6)' list(obj.children)=[] keep_placeholder=False placeholder.name='GrassTuftFactory(89086).spawn_placeholder(7)' list(placeholder.children)=[] obj.name='GrassTuftFactory(89086).spawn_asset(7)' list(obj.children)=[] keep_placeholder=False placeholder.name='GrassTuftFactory(89086).spawn_placeholder(8)' list(placeholder.children)=[] obj.name='GrassTuftFactory(89086).spawn_asset(8)' list(obj.children)=[] keep_placeholder=False placeholder.name='GrassTuftFactory(89086).spawn_placeholder(9)' list(placeholder.children)=[] obj.name='GrassTuftFactory(89086).spawn_asset(9)' list(obj.children)=[] 121090 scatter:GrassTuftFactory(89086) instances created. Info: Saved "scene.blend" Error: Not freed memory blocks: 30, total unfreed memory 0.000973 MB

[araistrick]

Hello, apologies for the delay, I have a patch for this and will release it shortly.

[araistrick]

should be fixed.