search

vishakha-lall / MapBot

A chatbot developed in Python
MIT License
145 stars 108 forks source link

Add functionality to persist and retrain ML model #59

Closed janakrajchadha closed 4 years ago

janakrajchadha commented 4 years ago

Requirement Currently, the ML model is trained from scratch whenever the application is setup. We need a way to persist the trained state of the model and retrain it based on certain conditions.

Pre-requisite Intermediate knowledge of Python Elementary understanding of Docker

Dependencies

57, since the logic to pick the model to persist will come from the previous issue

Description As the database state is persisted (issue #14), we need to persist the state of the trained classifier as well. The model needs to be saved in a file (pkl, h5 etc.) from which it can be read again whenever the application is restarted instead of complete retraining (not a major issue as of now but can become one once the dataset size increases significantly). Also, the model needs to be retrained from time to time, depending on either the number of times the bot wasn't able to respond correctly since the last run (a counter would be required for this) or a timed job (every week or two weeks). I'd suggest that we include both ways in the system and then pick one based on which method seems more appropriate to use for a particular scenario.

abhishekgupta368 commented 4 years ago

can you please assign me?

vishakha-lall commented 4 years ago

@janakrajchadha can you discuss @abhishekgupta368's approach and then assign this issue to him?

abhishekgupta368 commented 4 years ago

ok

abhishekgupta368 commented 4 years ago

we can train data and dump it into pickle format and store for later training.

abhishekgupta368 commented 4 years ago

later, we will load it and use it, as per our usage

janakrajchadha commented 4 years ago

@abhishekgupta368 Your approach makes sense. However, that is only part of the solution. You'll have to modify the Docker config and set up a retraining job as well. Would you be able to pick up those things as well?

abhishekgupta368 commented 4 years ago

i don't know about docker? where i can study about it?

vishakha-lall commented 4 years ago

@abhishekgupta368 please refrain from sending individual statements in issue discussions. This is not a chat. Try to collate your thoughts and put them together, before sending your conversation. @janakrajchadha does it make sense to split this issue into two parts?

vishakha-lall commented 4 years ago

@abhishekgupta368 you can refer to the official documentation here. Also try setting up the project the Docker way.

janakrajchadha commented 4 years ago

@vishakha-lall We can work on the retraining part in another issue. However, persisting the ML model and the configuration in Docker should be handled in one issue.

preeti13456 commented 4 years ago

Interested

vishakha-lall commented 4 years ago

@preeti13456 can you discuss your approach here before we assign this to you

preeti13456 commented 4 years ago

We can use tensorflow to retain and store the train model. MlOps Model management, deployment, and monitoring for configuring and retraining the model. Can use a function or CNN fr verifying the accuracy.

janakrajchadha commented 4 years ago

@preeti13456 Can you please elaborate? I want to understand why tensorflow is specifically needed to store the trained model. MLOps is a good option for overall monitoring and retraining. As far as verifying performance metrics is concerned, I'm not quite sure how a CNN would help to verify the accuracy of a tree-based classifier. Can you elaborate on this as well?

preeti13456 commented 4 years ago

With tensorflow we can use a trained model without having to retrain it, or pick-up training where we left off—in case the training process was interrupted. This way checkpoints created and automatically saved reduces out time. And CNN can help us in determining accuracy by creating a function epoch and including the specific error like sigmoid according to dataset.

Get Outlook for Androidhttps://aka.ms/ghei36

From: Janak Raj Chadha notifications@github.com Sent: Monday, March 16, 2020 12:47:40 PM To: vishakha-lall/MapBot MapBot@noreply.github.com Cc: preeti13456 preetijecrc99@outlook.com; Mention mention@noreply.github.com Subject: Re: [vishakha-lall/MapBot] Add functionality to persist and retrain ML model (#59)

@preeti13456https://github.com/preeti13456 Can you please elaborate? I want to understand why tensorflow is specifically needed to store the trained model. MLOps is a good option for overall monitoring and retraining. As far as verifying performance metrics is concerned, I'm not quite sure how a CNN would help to verify the accuracy of a tree-based classifier. Can you elaborate on this as well?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/vishakha-lall/MapBot/issues/59#issuecomment-599380911, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AITUUQ2DEXGIAGD4OJG2ZPDRHXHBJANCNFSM4LCMORVA.

janakrajchadha commented 4 years ago

@preeti13456 I understand that tensorflow offers these capabilities as a library. Our current system is not using tensorflow models, is it possible to store models created using other libraries with tensorflow functions? Since this application is being containerized using docker, does tensorflow have implicit capabilities which help with persisting model states and re-training as necessary? If yes, can you please share the sources which support your suggestion?

I'm not sure if you've gone through the current code, I'm not aware of a method of using CNN to determine the accuracy of models unless the model itself is a CNN which we have neither discussed as a possible model for text classification needs nor as part of this issue.

preeti13456 commented 4 years ago

sure

Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10

From: Vishakha Lall notifications@github.com Sent: Friday, March 13, 2020 8:31:12 AM To: vishakha-lall/MapBot MapBot@noreply.github.com Cc: preeti13456 preetijecrc99@outlook.com; Mention mention@noreply.github.com Subject: Re: [vishakha-lall/MapBot] Add functionality to persist and retrain ML model (#59)

@preeti13456https://github.com/preeti13456 can you discuss your approach here before we assign this to you

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/vishakha-lall/MapBot/issues/59#issuecomment-598524956, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AITUUQ36YXUM7IASYYC6R6TRHGOXRANCNFSM4LCMORVA.

vishakha-lall commented 4 years ago

@preeti13456 what is the progress on this issue? When will you be ready to raise a PR?

preeti13456 commented 4 years ago

Yes

Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10

From: Vishakha Lall notifications@github.com Sent: Friday, March 20, 2020 11:24:13 AM To: vishakha-lall/MapBot MapBot@noreply.github.com Cc: preeti13456 preetijecrc99@outlook.com; Mention mention@noreply.github.com Subject: Re: [vishakha-lall/MapBot] Add functionality to persist and retrain ML model (#59)

@preeti13456https://github.com/preeti13456 what is the progress on this issue? When will you be ready to raise a PR?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/vishakha-lall/MapBot/issues/59#issuecomment-601547978, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AITUUQYLPNYXJMCTWZBSP6TRIMAILANCNFSM4LCMORVA.

vishakha-lall commented 4 years ago

@preeti13456 I'm not receiving updates from your end. I'm unassigning this issue.

preeti13456 commented 4 years ago

NO Wait I have been busy today push the PR tomorrow fr sure.

Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10

From: Vishakha Lallmailto:notifications@github.com Sent: Friday, March 20, 2020 8:01 PM To: vishakha-lall/MapBotmailto:MapBot@noreply.github.com Cc: preeti13456mailto:preetijecrc99@outlook.com; Mentionmailto:mention@noreply.github.com Subject: Re: [vishakha-lall/MapBot] Add functionality to persist and retrain ML model (#59)

@preeti13456https://github.com/preeti13456 I'm not receiving updates from your end. I'm unassigning this issue.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/vishakha-lall/MapBot/issues/59#issuecomment-601730578, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AITUUQ6MARDSYCFUGUEUVODRIN43VANCNFSM4LCMORVA.

chttrjeankr commented 4 years ago

The model needs to be saved in a file (pkl, h5 etc.) from which it can be read again whenever the application is restarted instead of complete retraining (not a major issue as of now but can become one once the dataset size increases significantly).

I found a pretty simple way of how a model can be persisted across various runtimes. Persisting it in docker wouldn't be much of trouble either.

Also, the model needs to be retrained from time to time, depending on either the number of times the bot wasn't able to respond correctly since the last run (a counter would be required for this) or a timed job (every week or two weeks).

Implementing regularly timed retraining is also achievable where the program would check for the last modified time of the pickle file and compare it to the present time during the program runtime. If the time difference exceeds a threshold, we can go for retraining and rewriting(?) the pickle file.

@janakrajchadha I'd be willing to work on this.