search

NVIDIAGameWorks / kaolin-wisp

NVIDIA Kaolin Wisp is a PyTorch library powered by NVIDIA Kaolin Core to work with neural fields (including NeRFs, NGLOD, instant-ngp and VQAD).
Other
1.45k stars 132 forks source link

put wandb setup first in _setup_dashboards and arrange cast data pass… #184

Open barikata1984 opened 8 months ago

barikata1984 commented 8 months ago

…ed to wandb.Table into a double-nested list

barikata1984 commented 8 months ago

confirmed

barikata1984 commented 8 months ago

Why:

  1. training curves, such as total_loss and rgb_loss, are not plotted properly on a synced wandb dashboard: only a single point was plotted on it
  2. log_table() for _WandB did not work with the following error: 
    File "/home/atsushi/workspace/kaolin-wisp/app/nerf/main_nerf.py", line 128, in <module>
trainer.train()  # Run in headless mode
File "/home/atsushi/workspace/kaolin-wisp/wisp/trainers/base_trainer.py", line 371, in train
self.iterate()
File "/home/atsushi/workspace/kaolin-wisp/wisp/trainers/base_trainer.py", line 330, in iterate
self.end_epoch()    # determines if optimization keeps running
File "/home/atsushi/workspace/kaolin-wisp/wisp/trainers/base_trainer.py", line 308, in end_epoch
self.validate()
File "/home/atsushi/workspace/kaolin-wisp/wisp/trainers/multiview_trainer.py", line 294, in validate
self.tracker.log_table("args", record_dict, self.epoch)
File "/home/atsushi/workspace/kaolin-wisp/wisp/trainers/tracker/tracker.py", line 227, in log_table
dashboard.log_table(caption, data, step)
File "/home/atsushi/workspace/kaolin-wisp/wisp/trainers/tracker/tracker.py", line 453, in log_table
table = wandb.Table(columns=data.keys(), data=data.values())
File "/home/atsushi/miniconda3/envs/wisp/lib/python3.9/site-packages/wandb/data_types.py", line 290, in __init__
self._init_from_list(data, columns, optional, dtype)
File "/home/atsushi/miniconda3/envs/wisp/lib/python3.9/site-packages/wandb/data_types.py", line 309, in _init_from_list
assert isinstance(data, list), "data argument expects a `list` object"
AssertionError: data argument expects a `list` object

What:

  1. Configure a wandb dashboard first if it is enabled because wandb.init() should be called earlier than instantiating SummaryWrite according to the official description of wandb (link)
  2. Cast columns argument into a list and data argument into a 2D list because the columns argument should be iterable and, both the data argument and its element should be iterable as well based on the source (link)

P.S. PR #183 was my fault. Sorry about that. Please ignore it