Option to save best weights during training

[agrawalshubham01]

https://github.com/RoboticsClubIITJ/ML-DL-implementation/blob/e30d8d4a3da242f59c9fe491d870fdfc029df07b/MLlib/models.py#L8-L29

In above code only the last weight is being saved, instead one can give user option to take which weight

 class LinearRegression(): 
     def fit(self, X, Y, optimizer=GradientDescent, epochs=25, zeros=False): 
         _weight ={weight:None, loss = None} 
         self.weights = generate_weights(X.shape[1], 1, zeros=zeros)  
         print("Starting training with loss:", 
               optimizer.loss_func.loss(X, Y, self.weights)) 
         for epoch in range(1, epochs+1): 
             print("======================================") 
             self.weights = optimizer.iterate(X, Y, self.weights) 
             print("epoch:", epoch) 
             print("Loss in this step: ", 
                   optimizer.loss_func.loss(X, Y, self.weights)) 
         print("======================================\n") 
         print("Finished training with final loss:", 
               optimizer.loss_func.loss(X, Y, self.weights))
         print("=====================================================\n")
         if _weight['weight'] is None:
             _weight['weight']=self.weights
             _weight['loss'] = optimizer.loss_func.loss(X, Y, self.weights))
         elif _weight[weight] is not None and optimizer.loss_func.loss(X, Y, self.weights))<_weight['loss']:
             _weight['loss'] = optimizer.loss_func.loss(X, Y, self.weights))
             _weight['weight']=self.weights
     def predict(self, X, weights= 'best'): 
         if weights == 'best':
              return  np.dot(X, _weight['weight'])
        elif weights =='last':
             return np.dot(X, self.weights) 

In this way we can save best/last weight.

[JHunterHobbs]

Hello, may I be assigned this issue? Thanks

[agrawalshubham01]

@JHunterHobbs Sure, Go ahead.

[rohansingh9001]

@JHunterHobbs please add a parameter to the .fit() function so as to give the user an option to save the last weights or the best weights. You can decide the API and comment it first here. If we find that it integrates smoothly with the rest of the library and how we envision it moving forward, you can work on it and send a PR.

[JHunterHobbs]

@rohansingh9001 - Can we just let .fit() always save the best and last weights, and then allow the user to specify which weights to use for the prediction? Like the example from @agrawalshubham01 ?

[TarunTomar122]

@JHunterHobbs as @rohansingh9001 mentioned in the current standard libaries like tensorflow and pytorch fit function provides an arguement to save best or last. We want our modules to be as close to standard ones as possible. So It would be better if .fit() asks a parameter to save the last weights or the best weights.

[JHunterHobbs]

Ok I'll make some changes to my PR. Thanks for the clarification

[JHunterHobbs]

I'm a little confused because the issue documentation is asking for both the best and last weights to be saved but from this discussion it sounds like you are looking for either the best or last weights/model to be saved during training.

I also looked through both Tensorflow and PyTorch documentation to try see they were accomplishing this during fitting the model and I was not able to find anything. tf.keras.Sequential.fit does not provide a flag to save best or last weights, however you can provide a callback to accomplish early stopping during training, but I don't think that is what you're looking for. @TarunTomar122 - Could you link some documentation to what you are referring to. That'd be really helpful. Thanks.

[rohansingh9001]

@agrawalshubham01 please change the title of the issue to something better like "Option to save best weights during training" as our contributors are getting confused because of it. Also @JHunterHobbs I have made a comment on your PR to clarify this issue. Hope that helps!