search

darkua / json_delta

a service that tracks json model changes and lists them
0 stars 0 forks source link

Parallel parking #2

Open donbonifacio opened 7 years ago

donbonifacio commented 7 years ago

Hello,

Somewhere down the line, this service is deployed/delivered via several machines and has a considerable load. It will also need to have its persistence somewhere else. Because of some reasons that you cannot control, you start receiving duplicate requests for the same data. For example: 

// request 1
{
  "_id": 1,
  "transactionId": "tx1",
  "name": "Bruce Willis",
  "address": {
    "street": "Nakatomi Plaza"
  }
}

// request 2
{
  "_id": 1,
  "transactionId": "tx1",
  "name": "Bruce Willis",
  "address": {
    "street": "Nakatomi Plaza"
  }

The systems that send these requests, also started sending a transactionId that allows you to detect they are duplicate. For some more reasons, it's very problematic business wise to store duplicate data. You can only store one, even if you receive may. And all those many requests can be delivered at exactly the same time.

Can you elaborate on an approach to this?

darkua commented 7 years ago

So this is a sync issue between many backends without a shared state/persistence, and i solved it back in the past using a simple cache system, implemented using redis. So use transacationId as redis key, and on every request put to redis, in order to keep redundancy to not have a single point of failure, redis cluster to the rescue, im made a lib in node.js for it https://github.com/darkua/redis-node-cluster :) And of course the downside of this is having an external call to potentially another network and all that, so delaying the all request-response time, but if i remember good for my issue adding this was completely irrelevant to the all request cycle, so it was quite a good fix.

donbonifacio commented 7 years ago

And how would you test this? To replicate the problem and then assert that it was solved?