I am monitoring energy consumption of a pytorch model. I am sampling several times during training loop with EnergyContext and record (code snippet below). I am noticing that there are more than half samples showing zero consumption. See attached partial log. joules_sample.log
Any ideas?
for i, (images, labels) in enumerate(train_loader):
# get the inputs; data is a list of [inputs, labels]
images = images.to(device)
labels = labels.to(device)
# zero the parameter gradients
optimizer.zero_grad()
# Monitor joules sparingly
if (i % monitor_interval) == (monitor_interval-1):
if monitor_joules:
# pyjoules
with EnergyContext(handler=pd_handler, start_tag='forward') as ctx:
# forward + backward + optimize
outputs = model(images)
ctx.record(tag='loss')
loss = criterion(outputs, labels)
ctx.record(tag='backward')
loss.backward()
ctx.record(tag='step')
optimizer.step()
ctx.record(tag='overhead')
I am monitoring energy consumption of a pytorch model. I am sampling several times during training loop with EnergyContext and record (code snippet below). I am noticing that there are more than half samples showing zero consumption. See attached partial log.
joules_sample.log
Any ideas?