search

tryolabs / luminoth

Deep Learning toolkit for Computer Vision.
https://tryolabs.com
BSD 3-Clause "New" or "Revised" License
2.4k stars 399 forks source link

Generating multiple checkpoint files #273

Open reorder-cv opened 5 years ago

reorder-cv commented 5 years ago

is it possible to generate multiple checkpoint files at specified intervals? say for example, if the training iteration is 2000, can we generate separate checkpoint files for every 500 iterations or at 1000th and 2000th? if its possible, please help me with the ways to do.

Thank you in advance.

StephenRUK commented 5 years ago

When a (TensorFlow) checkpoint is stored, the current weights are saved. If they aren't saved in a checkpoint, there's no way to get them back at a later time. You can only set a time interval for checkpoints in the Luminoth config file.

See base_config.yml for possible configurations (importantly the two options below):

  # The frequency, in seconds, that a checkpoint is saved.
  save_checkpoint_secs: 600
  # The maximum number of checkpoints to keep
  checkpoints_max_keep: 1