search

Cloudkibo / KiboPush

0 stars 1 forks source link

Customer discussion EZShifa #11058

Open sojharo opened 3 years ago

sojharo commented 3 years ago

This is the task to discuss and track work related to EZShifa and all the meeting minutes and discussion will be put here.

On high level, we will be looking into following:

  1. Having a separate KiboPush instance for them
  2. Look into database level data encryption
sojharo commented 3 years ago

Next steps:

ImranBinShoukat commented 3 years ago
  1. Option# 01: We ask them to create a separate dedicated Amazon instance to store all of their data.
  2. Option# 02: We create an instance on Amazon to store all of our + their data.
  3. Option# 03: We send all the data to them and they store it in their server.
sojharo commented 3 years ago

Following are the services of KiboPush including ones which have mongodb database running alongside them. I am reading up on Amazon instances and will update here more about what options we have for this.

Our DB layer code is written in a way that with few code additions it can start pointing towards cloud database as well. For now, it is pointing towards mongodb which is running on same droplet besides it.

Untitled Diagram

sojharo commented 3 years ago

Base on above diagram, we can think of following options:

1. Moving entire infrastructure to Amazon.

We move our entire infrastructure to Amazon and dblayers will be modified to talk to their cloud database. However, moving to this infrastructure will also require us to do some changes in our codebase. There will be 3 cloud databases running for us. All of our services will run as separate services inside amazon ecosystem. For each type of service they have a service i.e.

  1. Our business logic and code will run on EC2 instance (equivalent to DigitalOcean droplet, see here and here)
  2. Our db layer code (on EC2) will now point to cloud databases i.e. MongoDB on AWS or Amazon DocumentDB (which has encryption at rest options and is compatible to MongoDB, see here)
  3. Our CDN server can either run on EC2 instance or CDN service provided by aws called CloudFront
  4. Our Cronjobs which are currently running on droplets beside kiboengage and kibochat codebase will run in Amazon Cron job service

This path will require some learning of different AWS services and require changes at separate places. One option in this to make it easier is to just run all of our system components on EC2 of Amazon and then just do changes in DB layer part to point to AWS encrypted documentDB.

2. Moving chat data to Amazon Cloud Encrypted database i.e. DocumentDB

Second option is we just keep using existing infrastructure on DigitalOcean and just move the chat related database on Amazon encrypted database called DocumentDB. The benefit on this is less friction and less work involved and less changes to our existing codebase. We do any of two things here:

  1. Move the entire KiboChat dblayer tables to Amazon Cloud and don't manage mongodb for KiboChat Db layer on digitalocean droplet.
  2. Just manage the chat messages table on Amazon cloud for encrypted chat storage and rest of the data i.e. subscriber names and chatbot related data will be on mongodb instance on our digital ocean.

First option is better as all the kibochat related database tables will be in one place and we will not have to access two different database for kibochat.

Untitled Diagram

3. We send all the data to them and they store it on their instance and take care of encryption

This option is most feasible however, but will require us to write special logic for them in live chat. We will send messages received from customer to their system and will also retrieve the chat messages from them on showing on live chat UI.

This will also require them to create API endpoints for us to send chat messages to them and also retrieve from them to show on our UI when their admins use our UI to talk to customers.

Untitled Diagram

We can talk with them and have their opinion on this.

4. We do just the encryption work on our own live chat data and remain on existing infrastructure.

There are options to store data in encrypted state on MongoDb and there are certain frameworks for this doing this. On our server application, nodejs, we can store encryption key in environment variables. So Nodejs can use that key to decrypt the data for doing data search or other access queries.

There are two ways to encrypt data in Mongodb which is provided by mongodb frameworks.

  1. Encrypt the whole mongodb document (complete row is encrypted)
  2. Do the field level encryption (certain columns in table are encrypted)

There is an overhead in performance when we try to fetch and decrypt encrypted data. Therefore, experts have suggested that we only do Field level encryption and only encrypt fields which contain sensitive information such as social security numbers or actual chat messages.

This is called application level encryption and applications need to use frameworks (sdks) provided by Mongodb to encrypt and send data to Mongodb or decrypt it while getting it back. Therefore, we will have to write some encryption / decryption logic on our application. Mostly, the hard work will be done by frameworks provided by mongodb for so many languages.

For this, we may need to change the mongodb driver that we are using on our Nodejs server. All the example code with encryption I have seen from them is using their own mongodb driver for nodejs. We are using one other open source driver called Mongoose.

sojharo commented 3 years ago

Here are the meeting minutes that I captured:

Participants from EZShifa:

  1. Zubair Ahmad
  2. Hamid Hassan
  3. Faisal Saleem

Participants from CloudKibo:

  1. Imran Shoukat
  2. Jawaid Ekram
  3. Sojharo

Pointers

ImranBinShoukat commented 3 years ago

Next steps for EZShifa:

  1. Message flow from Whatsapp Mobile
  2. Message Flow from Whatsapp (Web Page)
  3. Message Flow from Web Page from Messenger (option)
  4. Screenshots and a Demo
  5. Send them the DB Schema for Document DB
  6. Proposal From US
  7. A deck to walk them thru Whatsapp Billing so they have a good understanding.
ImranBinShoukat commented 3 years ago

Task Update:

ImranBinShoukat commented 3 years ago

Task Update:

Proposal: https://drive.google.com/file/d/1v6_HJ28sUjcsvCpLW9t4c3KlLxWNNxPB/view?usp=sharing Pricing Deck: https://docs.google.com/presentation/d/1zLiGbRnZLdrML2RDH2wOIKEzDP42AvM27nPPTqm1qV8/edit#slide=id.p