fate-ubw
commented
11 months ago This blog is the best I have ever read. But here are some mistake in trainning softmax class fix method.
idx = random.sample(range(len(X)), len(X))
for i in range(0, len(X), self._batch_size):
loss = self._sgd_step(
X[i:i+self._batch_size],
y[i:i+self._batch_size],
lr=self._base_lr / math.sqrt(1 + epoch)
)the batch data X[i:i+self._batch_size] and y[i:i+self._batch_size] is wrong. Becaues each batch should be random sampleing form source data. Here is my version of fit method:
def fit(self, X, y):
''' trains the model on the given data '''
history = []
batch_count = 0
for epoch in range(self._epochs):
# shuffle indices
# pdb.set_trace()
idx = random.sample(range(len(X)), len(X))
for i in range(0, len(X), self._batch_size):
random_idx = idx[i:i+self._batch_size]
loss = self._sgd_step(
np.array([X[i] for i in random_idx]),
np.array([y[i] for i in random_idx]),
lr=self._base_lr / math.sqrt(1 + epoch)
)
history.append(loss)
if epoch % 5 == 0:
print(f'epoch: {epoch}\tloss: {loss:.3f}')
return history
e-dorigatti
Automatic differentiation from scratch | Emilio’s Blog
Automatic differentiation (AD) is one of the most important features present in modern frameworks such as Tensorflow, Pytorch, Theano, etc. AD has made paramter optimization through gradient descent an order of magnitude faster and easier, and drastically lowered the barrier of entry for people without a solid mathematical background. In spite of its utility, AD is surprisingly simple to implement, which is what we are going to do here.
https://e-dorigatti.github.io/math/deep%20learning/2020/04/07/autodiff.html