I'm not sure if it's something I did or what, but for some reason my output has 0 vertices after 50k iterations. I set my save_iter to 25k. Running the first checkpoint (25k) I get "vertices: 3146952 faces: 6195326". Running the second checkpoint (50k) I get "vertices: 468872 faces: 924836". (which is already a huge jump down). Running any of the next outputs I get:
(Setting affinity with NVML failed, skipping...)
Running mesh extraction with 1 GPUs.
Setup trainer.
Using random seed 0
model parameter count: 99,166,504
Initialize model weights using type: none, gain: None
Using random seed 0
Allow TensorFloat32 operations on supported devices
Loading checkpoint (local): logs/example_group/example_name/75000.pt
Loading the model...
Done with loading the checkpoint.
Extracting surface at resolution 2153 819 1436
Traceback (most recent call last):
File "projects/neuralangelo/scripts/extract_mesh.py", line 105, in
main()
File "projects/neuralangelo/scripts/extract_mesh.py", line 93, in main
print(f"vertices: {len(mesh.vertices)}")
AttributeError: 'list' object has no attribute 'vertices'
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 4509) of binary: /root/miniconda3/envs/neuralangelo/bin/python
Traceback (most recent call last):
File "/root/miniconda3/envs/neuralangelo/bin/torchrun", line 10, in
sys.exit(main())
File "/root/miniconda3/envs/neuralangelo/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper
return f(*args, **kwargs)
File "/root/miniconda3/envs/neuralangelo/lib/python3.8/site-packages/torch/distributed/run.py", line 794, in main
run(args)
File "/root/miniconda3/envs/neuralangelo/lib/python3.8/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/root/miniconda3/envs/neuralangelo/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/root/miniconda3/envs/neuralangelo/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
Root Cause (first observed failure):
[0]:
time : 2023-10-28_12:49:21
host :
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 4509)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
From the line: AttributeError: 'list' object has no attribute 'vertices' I'm guessing that there just isn't any vertices to be turned into a mesh in the output.
I'm guessing my training just failed and I need to rerun it, but just to be sure I'm posting this here and incase anyone else stumbles across this issue.
joevento
commented
10 months ago
I'm not sure if it's something I did or what, but for some reason my output has 0 vertices after 50k iterations. I set my save_iter to 25k. Running the first checkpoint (25k) I get "vertices: 3146952 faces: 6195326". Running the second checkpoint (50k) I get "vertices: 468872 faces: 924836". (which is already a huge jump down). Running any of the next outputs I get: (Setting affinity with NVML failed, skipping...) Running mesh extraction with 1 GPUs. Setup trainer. Using random seed 0 model parameter count: 99,166,504 Initialize model weights using type: none, gain: None Using random seed 0 Allow TensorFloat32 operations on supported devices Loading checkpoint (local): logs/example_group/example_name/75000.pt
projects/neuralangelo/scripts/extract_mesh.py FAILED
Failures: