I'm not sure if it's something I did or what, but for some reason my output has 0 vertices after 50k iterations. I set my save_iter to 25k. Running the first checkpoint (25k) I get "vertices: 3146952 faces: 6195326". Running the second checkpoint (50k) I get "vertices: 468872 faces: 924836". (which is already a huge jump down). Running any of the next outputs I get:
(Setting affinity with NVML failed, skipping...)
Running mesh extraction with 1 GPUs.
Setup trainer.
Using random seed 0
model parameter count: 99,166,504
Initialize model weights using type: none, gain: None
Using random seed 0
Allow TensorFloat32 operations on supported devices
Loading checkpoint (local): logs/example_group/example_name/75000.pt
Loading the model...
Done with loading the checkpoint.
Extracting surface at resolution 2153 819 1436
Traceback (most recent call last):
File "projects/neuralangelo/scripts/extract_mesh.py", line 105, in
main()
File "projects/neuralangelo/scripts/extract_mesh.py", line 93, in main
print(f"vertices: {len(mesh.vertices)}")
AttributeError: 'list' object has no attribute 'vertices'
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 4509) of binary: /root/miniconda3/envs/neuralangelo/bin/python
Traceback (most recent call last):
File "/root/miniconda3/envs/neuralangelo/bin/torchrun", line 10, in
sys.exit(main())
File "/root/miniconda3/envs/neuralangelo/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper
return f(*args, **kwargs)
File "/root/miniconda3/envs/neuralangelo/lib/python3.8/site-packages/torch/distributed/run.py", line 794, in main
run(args)
File "/root/miniconda3/envs/neuralangelo/lib/python3.8/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/root/miniconda3/envs/neuralangelo/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/root/miniconda3/envs/neuralangelo/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
Root Cause (first observed failure):
[0]:
time : 2023-10-28_12:49:21
host :
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 4509)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
From the line: AttributeError: 'list' object has no attribute 'vertices' I'm guessing that there just isn't any vertices to be turned into a mesh in the output.
I'm guessing my training just failed and I need to rerun it, but just to be sure I'm posting this here and incase anyone else stumbles across this issue.
I'm not sure if it's something I did or what, but for some reason my output has 0 vertices after 50k iterations. I set my save_iter to 25k. Running the first checkpoint (25k) I get "vertices: 3146952 faces: 6195326". Running the second checkpoint (50k) I get "vertices: 468872 faces: 924836". (which is already a huge jump down). Running any of the next outputs I get: (Setting affinity with NVML failed, skipping...) Running mesh extraction with 1 GPUs. Setup trainer. Using random seed 0 model parameter count: 99,166,504 Initialize model weights using type: none, gain: None Using random seed 0 Allow TensorFloat32 operations on supported devices Loading checkpoint (local): logs/example_group/example_name/75000.pt
projects/neuralangelo/scripts/extract_mesh.py FAILED
Failures: