clustering / failover / live deployment

[jlhughes]

Version 3 - FileLock Scenario

• One Journal with all operations interleaved.
• Journal on NFS

All instances in 'cluster' attempt to acquire NFS File Lock Those without the lock (SLAVE):

• tail the journal keeping MDAOs up-to-date
• NOT process transactions
• DIG/DUG enabled for all but transaction processing (how ensure only one replies)
• Enable Web UI

Instance with lock (MASTER):

• append to journal
• process transactions
• DIG/DUG disabled except for transaction processing
• Disable Web UI - to all but 'admin'

Issues

• DIG/DUG - how to only get one DUG response, may have be handled by MASTER

Additional

• AppConfig - control mode (MASTER, SLAVE)
• Repo journal.0 files must be packaged by build and stored in jar as jar will be signed.

Security

• PKI - all instances have same keys? Does anything in a signed transaction need to be decrypted?

[jlhughes]

Thoughts: Use DUG to create clustering. This handles fail-over, which could be the same case as an upgrade as we upgrade one half at a time.

Discovery

• add 'realm' to AppConfig
• broadcast AppConfig over TCP (not UDP) on internal AWS network

Cluster Master/Slave still needs consideration

Slave

1. listen for AppConfig with matching realm
2. perform select on all DIG services to sync. All data is signed, each instance has the same private key to decrypt.
3. register on DUG for all services ( ? register on DUG before select on DIG ? )

Master

• ? how to decide who is master
  • listen for x seconds ?

Failover

• when master fails

Load Sharing

• Only one instance will be performing transactions
• Other instances will be handling all other non-transaction request, web....

[jlhughes]

Version 2 - Simplified Scenario

• 3 instances x, y, z
• all share nas/raid file system
• on startup
  • if acquire lock, then master
  • process transactions
  • disable non-admin web interface and non-txn dig
  • no lock, then slave
  • ignore transactions
  • enable regular web, dig
• all continually try to get lock.
  • when instance acquires lock it doesn't read all journals,
  • it reads appropriate journal path to determine user balance

[ronin00]

Issue moved to JIRA: https://nanopay.atlassian.net/browse/DOP-54