fig, ax = plt.subplots(1, 2, figsize=(15, 5))
for j, lr in enumerate(LEARNING_RATES):
ax[0].plot(grad_norms[j], label=f'Learning rate: {lr}')
ax[0].legend()
ax[0].set_xlabel('Iteration')
ax[0].set_ylabel('grad norm')
ax[0].set_title('Gradient norm for each learning rate')
for j, lr in enumerate(LEARNING_RATES):
ax[1].plot(np.log10(losses[j]), label=f'Learning rate: {lr}')
ax[1].legend()
ax[1].set_xlabel('Iteration')
ax[1].set_ylabel('log(loss)')
ax[1].set_title('Loss for each learning rate')
plt.show()
And the cell's current output:
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-18-91a4bf5517c3> in <module>
9
10 for j, lr in enumerate(LEARNING_RATES):
---> 11 ax[1].plot(np.log10(losses[j]), label=f'Learning rate: {lr}')
12 ax[1].legend()
13 ax[1].set_xlabel('Iteration')
/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/torch/tensor.py in __array__(self, dtype)
619 return handle_torch_function(Tensor.__array__, (self,), self, dtype=dtype)
620 if dtype is None:
--> 621 return self.numpy()
622 else:
623 return self.numpy().astype(dtype, copy=False)
RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.
We need to detach a tensor in the "Signs of Healthy Training" section on the Gradient Descent page.
Here's the cell:
And the cell's current output: