pytorch-ignite / examples

Examples, tutorials, and how-to guides
BSD 3-Clause "New" or "Revised" License
11 stars 11 forks source link

Elapsed training time #77

Closed RMeli closed 2 years ago

RMeli commented 2 years ago

Added a short section in 03 Time Profiling, under Custom profiling using Timer to show how to use Timer to track and display the elapsed training time.

elapsed_time = Timer()

elapsed_time.attach(
    trainer,
    start=Events.STARTED,         # Start timer at the beginning of training
    resume=Events.EPOCH_STARTED,  # Resume timer at the beginning of each epoch
    pause=Events.EPOCH_COMPLETED, # Pause timer at the end of each epoch
    step=Events.EPOCH_COMPLETED,  # Step (update) timer at the end of each epoch
)

@trainer.on(Events.EPOCH_COMPLETED)
def log_elapsed_time(trainer):
    print(f"   Elapsed time: {elapsed_time.value()}")

trainer.run(train_loader, max_epochs=3)

I also added the files and folders that are automatically generated while running the notebook to .gitignore.

vfdev-5 commented 2 years ago

@RMeli thanks for the PR ! Seeing the output, trainer continue the run from epoch 2 to epoch 3. Maybe, we can either set max_epochs=5 to make more epochs: 3, 4, 5 or we can reset trainer and run from 0 by:

trainer.state.max_epochs = None
trainer.run(train_loader, max_epochs=3)
RMeli commented 2 years ago

Good point! How comes the previous sections do not suffer of the same issue?

vfdev-5 commented 2 years ago

Because they do max_epochs=2 for all the runs and thus restart from 0. In your case, you set it to 3 and trainer stopped at 2, so it can continue to 3. I'd suggest to make 2 epochs as well, as we'll change how to reset the trainer in future.