Interaction Diagram for high uses cases

[jekram]

We have done the Interaction diagram for Twitter to Messenger.

Please identify other 10 uses cases where we should do Interaction diagram.

Examples would be:

1. When we receive a Webhook
2. When we get a subscriber
3. When we do various broadcast (one for each uses cases)
4. Various Page Scroll

We need to do this understand the scaling issues. This task is to identify the list and then we can distribute it.

We are struggling with Performance and until we have Interaction diagrams we would not know the real cause.

Also, we would need for issues like https://github.com/Cloudkibo/KiboPush/issues/6148#

Why account server and other servers are running high

[sojharo]

Imran and I have discussed and have come up with following use cases that we should create interaction diagrams for. They are heavy logic features and we might scale them while inspecting the diagrams.

1. subscriber create
2. page connect/disconnect
3. broadcast using batch api
4. broadcast using broadcast api
5. poll using send api
6. poll using broadcast api
7. survey using send api
8. survey using broadcast api
9. pagination
10. subscriber fetch

[jekram]

@sojharo @imran Let's do these in these orders

1. When we receive webhook
2. When we get a subscriber (subscriber create)
3. Broadcast using batch api
4. broadcast using broadcast api (for each use case)
5. pagination
6. subscriber fetch

After these are done we will pick next

[sojharo]

Ok Sir, I will do 1, 2 and 5 today

[jekram]

@sojharo When we are doing these diagrams - let also document any insight on what improvements we can make

[sojharo]

Ok Sir.

[sojharo]

We have created all the interaction diagrams for the said use cases. Please see this document:

https://docs.google.com/document/d/1xqyGEFuQBnj0ToO-kfKECXjmpx4rUfUXb76KHn9xfmE/edit#

[sojharo]

I have added the information on which keys and tables we use when we do I/O calls discussed in our interaction diagrams. I have done for 3 diagrams on which I worked.

https://docs.google.com/document/d/1xqyGEFuQBnj0ToO-kfKECXjmpx4rUfUXb76KHn9xfmE/edit#

I am going to read up on the following article in this issue:

https://docs.mongodb.com/manual/applications/indexes/

[sojharo]

I completed reading the article on indexing and has understood the ways to create indexes. When we create index, we should do it outside of main application code as it would start building indexes at that time and would slow down the database for that particular time. We should do it from script or from database cli and do at night time.

Once the indexing is enabled, then on each new record, it would update the indexes so during that time write will be slower. However, it has then benefits that indexes are also stored in ram and they make the query faster.

Once our discussion on above document is complete and we have identified all the indexes, then we should enable them.

[sojharo]

I completed reading on indexes from mongodb docs. Let Imran complete identifying keys and tables in this document then we can go ahead with indexing work in this.

[jekram]

@ImranBinShoukat What is the update here?

[ImranBinShoukat]

I have added the keys and tables info for each interaction diagrams. Also, read the article on indexes from the mongoDB docs.

[sojharo]

We worked on the document and did the math for the database level I/Os with and without indexing. We discussed how indexing will reduce the number of records scanned for a given query.

I have just read more about doing indexing from MongooseJS documentation. This is the ODM that we use in all of our applications. It says that autoIndex which is way of mongoose to create indexes when app starts should be disabled as it slows down the application. So our app should not be creating indexes and this work should be done outside the application. Read these two links:

https://mongoosejs.com/docs/guide.html#indexes

https://mongoosejs.com/docs/guide.html#autoIndex

Also, let's discuss that indexing already populated tables also have some precaution steps to take as mongodb locks the table being indexed.

https://docs.mongodb.com/manual/core/index-creation/#index-operations

[ImranBinShoukat]

Task Update:

• I have created the index on following keys:
  • Subscribers -- companyId, page_id,
  • Pages -- companyId, pageId
• Updated the I/O calls before and after indexing in the document.

https://docs.google.com/document/d/1xqyGEFuQBnj0ToO-kfKECXjmpx4rUfUXb76KHn9xfmE/edit#