alifarid0991
commented
4 years ago Points to be considered before finding a solution for the syncing problem:
- The flow of operation: ML service gets the indicators from the logframe indicators/ endpoint page by page, saves it in an in memory db ( Redis ), and then updates its knowledge-base, when the update is over it replaces the knowlodge-base with the old one.
- ML service is a CPU consuming and memory consuming service because of the ML models it's using.
- Updating KB (knowledge-base) is a time-consuming (30 minutes) and CPU consuming process so it might increase the response time of other requests while happening.
- Getting the indicators logframe due to the number of pages and indicators is also a time consuming process. Options that can be implemented in syncing ML indicators and logframe:
- A process running on ML service that checks if the indicators are changed or not: This is a good and easy approach but ML service's resources already full and this overhead due to the fact that the operation is time and CPU consuming and it would lower the performance.
- ML service provides an endpoint that when it's called it will trigger the update KB function and it will provide an option that whenever an indicator is added update KB would happen: The problem is that updating the KB happens very often and the update takes almost 30 minutes.
- A process on backend call the update KB endpoint on intervals: Updating the indicators should be all about the one adding indicators so that it can be customizable for him. This interval should be something that could easily be tested to have a value that outcomes the best result. If we put together the facts and options, the last option would be almost easy to implement approach also may provide the best result and fewer resources.
What I've talked with Ali Farid was that he wanted for us to call his
update-knowledge-baseendpoint every night that there an update or creation of indicators. Open this discussion because Ali Kolai wanted to understand better our options.