search

NVIDIAGameWorks / kaolin-wisp

NVIDIA Kaolin Wisp is a PyTorch library powered by NVIDIA Kaolin Core to work with neural fields (including NeRFs, NGLOD, instant-ngp and VQAD).
Other
1.45k stars 132 forks source link

nglod does not save the weights+model #160

Closed ttsesm closed 1 year ago

ttsesm commented 1 year ago

I am testing the app/nglod example in an interactive mode. While in the config file it is set to save the model every 5 epochs:

trainer:
  # Base Trainer config
  exp_name: "nglod-octree"  # Name of the experiment: a unique id to use for logging, model names, etc.
  mode: 'train'             # Choices: 'train', 'validate'
  max_epochs: 10            # Number of epochs to run the training.
  save_every: 5             # Saves the optimized model every N epochs
  save_as_new: False        # If True, will save the model as a new file every time the model is saved
  model_format: 'full'      # Format to save the model: 'full' (weights+model) or 'state_dict'
  render_every: 5           # Renders an image of the neural field every N epochs
  valid_every: 10           # Runs validation every N epochs
  enable_amp: True          # If enabled, the step() training function will use mixed precision.
  profile_nvtx: True        # If enabled, nvtx markers will be emitted by torch for profiling.
  grid_lr_weight: 1.0    # Learning rate weighting applied only for the grid parameters (contain "grid" in their name)

when I check on the log folder there is nothing saved. Any idea why?

ttsesm commented 1 year ago

ok, had to comment the following line https://github.com/NVIDIAGameWorks/kaolin-wisp/blob/25249a1db98a603d10b02cde38e822f41b4f26ac/app/nglod/main_nglod.py#L68 for making it work on interactive mode