Replicated data is not synced to disk

[amclain]

When using disk persistence and intentionally causing a partition split, I noticed that data written on one side of the split is not synced to disk on the nodes on the other side of the split after the partition has been repaired and replication has completed. This could cause problems if the nodes on one side of a split are restarted during the split, as they would lose the data they had replicated from the other side of the partition. I have set up an example to reproduce the issue.

Steps To Reproduce

• Save the following docker-compose.yaml to a directory:

version: "3"
services:
  db1:
    image: jemc/jylis
    ports:
      - "6379:6379"
    volumes:
      - ./data1:/data
    command:
      - "--disk-dir=/data"
      - "--addr=db1:9999:db1"

  db2:
    image: jemc/jylis
    ports:
      - "6382:6379"
    volumes:
      - ./data2:/data
    command:
      - "--disk-dir=/data"
      - "--addr=db2:9999:db2"
      - "--seed-addrs=db1:9999:db1"
    links:
      - db1
      - db3

  db3:
    image: jemc/jylis
    ports:
      - "6383:6379"
    command:
      - "--addr=db3:9999:db3"
      - "--seed-addrs=db1:9999:db1"
    links:
      - db1

  cli1:
    image: redis:4.0-alpine
    restart: "no"
    command: redis-cli -h db1
    links:
      - db1

  cli2:
    image: redis:4.0-alpine
    restart: "no"
    command: redis-cli -h db2
    links:
      - db2

  cli3:
    image: redis:4.0-alpine
    restart: "no"
    command: redis-cli -h db3
    links:
      - db3

• Create directories data1 and data2 to hold the disk persistance data. We'll leave db3 as in-memory only.

$ mkdir data1
$ mkdir data2

• Run docker-compose up -d to start the cluster.

• Run docker-compose run cli1 to connect a CLI to db1. Note: It helps to open the CLIs in separate terminals.

• Run the command MVREG SET foo 1.

• Run docker-compose run cli2 to connect a CLI to db2.

• Run the command MVREG GET foo, which should return 1) "1".

• Comment out all of the links and --seed-addrs for db2 in the yaml to separate the node from the cluster.

db2:
  image: jemc/jylis
  ports:
    - "6382:6379"
  volumes:
    - ./data2:/data
  command:
    - "--disk-dir=/data"
    - "--addr=db2:9999:db2"
  #   - "--seed-addrs=db1:9999:db1"
  # links:
  #   - db1
  #   - db3

• Shut down the cluster with docker-compose down.

• Bring the cluster back up with docker-compose up -d

• On cli1 run MVREG GET foo. The result is 1) "1".

• On cli2 run MVREG GET foo. The result is (empty list or set), but should be "1".

• The data written to db1 was not persisted to db2's disk

• On cli2 run MVREG SET foo 2.

• Uncomment the yaml for db2 so that the node will rejoin the cluster.

• Shut down the cluster with docker-compose down.

• Bring the cluster back up with docker-compose up -d

• On cli2 run MVREG GET foo. This will return 1) "2" 2) "1".

• On cli2 run MVREG SET foo 3.

• On cli1 run MVREG GET foo. This will return "3".

• Current state: Value "1" is persisted to disk on db1 and "3" is on disk on db2. They both have a value of "3" in memory.

• Comment out the links and --seed-addrs for db2 again.

• Shut down the cluster with docker-compose down.

• Bring the cluster back up with docker-compose up -d

• On cli1 run MVREG GET foo. This will return 1) "1".

• On cli2 run MVREG GET foo. This will return 1) "3".

• This shows that the values synced in the replication process have not been persisted to disk.

[jemc]

Thanks for the very detailed steps to reproduce.

I'll make sure this is fixed and that we have a regression test for it before the 1.0 version.

[jemc]

Just to check though, considering my comment here: https://github.com/jemc/jylis/pull/11#discussion_r194225412

Have you confirmed that setting up one-directional links in docker-compose actually allows Jylis to establish a bidirectional link between nodes? Note that each node in the cluster needs to be able to connect to the declared addr of each other node.

I haven't read through the entire steps to reproduce in this ticket yet, but having only uni-directional links between nodes could potentially explain some issues with partition tolerance

I can check this myself a bit later if I haven't heard back from you by then.

[amclain]

Have you confirmed that setting up one-directional links in docker-compose actually allows Jylis to establish a bidirectional link between nodes?

Yes, the links are bidirectional. Adding the links in both directions results in an error: ERROR: Circular dependency between db3 and db2

[amclain]

I noticed something else as well: The replicated data is being stored on disk after replication and there is a PDAT command in the file with the MVREG command following it. Maybe that's being skipped over when the file is read?