[ec] es data

[Alice52]

1. go-mysql-transfer

1. this is according to third party middle-ware to sync data, it can handle complex logic by lua
2. reference

2. code[es data sync issue]

1. diagram:

   • link

   • image v1

     

   • image v2

   

[Alice52]

mq

1. set prefech_count to guarantee the theoretical maximum throughput

2. multiple consumers

   // handle connection issue
   ExecutorService es = Executors.newFixedThreadPool(20);
   Connection conn = factory.newConnection(es);

3. multiple thread

4. bulk ACK

启用Prefetch count设置； 先1个消费者，1次只接收1条，处理完毕后再传输下一条，这样可以避免并发冲突和消息顺序问题； 如果消费速度不满足要求，则1次接收多条，按接收顺序处理； 如果消费速度还是不满足要求，则1次接收多条，并行处理； 如果消费速度还是不满足要求，则启动多个消费者，并行处理。 如果消费速度还是不满足要求，改需求，或者换别的中间件。

[Alice52]

image v2

• Update design about split es-data logic from job
• Job just cron trigger, pull data from database then send to mq. Or it sends signal, then es-elasticsearch service will handle data sync.
• Now all press of es-data sync is split from ec-product service
• Job enqueue is in es-job-queue, and immediately trigger enqueue is es-immediately-queue

[Alice52]

notice

1. this sync service is minor service, so it cannot have any side impact on ec-product service
2. schedule job will pull data form db, it will lead to poor performance, so plan to use slave database for data sync

[Alice52]

task framework

1. quartz
2. can update cron-expresion dynamically
3. link

mq

1. using rocket mq.