Jepsen bank-multitable test failed with anomalies

zyguan commented 2 years ago

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Run jepsen bank-multitable with the following test options (note that the follower read is enabled):

{:auto-retry-limit :default,
 :ssh-private-key "jepsen.pem",
 :isolation :repeatable-read,
 :concurrency 15,
 :predicate-read false,
 :tarball-url
 "http://minio.pingcap.net:9000/tp-team/tests/jepsen/debug/plan-exec-634871-2511458490/tidb-master.tar.gz",
 :txn-mode "optimistic",
 :force-reinstall true,
 :update-in-place false,
 :ssh
 {:username "root",
  :password "root",
  :strict-host-key-checking false,
  :private-key-path "jepsen.pem"},
 :nemesis-long-recovery false,
 :follower-read true,
 :nemesis
 {:interval 10,
  :schedules true,
  :kill true,
  :stop true,
  :partition-one true},
 :nodes
 ["node-0.node-peer.debug-tps-605471-1-460"
  "node-1.node-peer.debug-tps-605471-1-460"
  "node-2.node-peer.debug-tps-605471-1-460"
  "node-3.node-peer.debug-tps-605471-1-460"
  "node-4.node-peer.debug-tps-605471-1-460"],
 :test-count 1,
 :read-lock nil,
 :use-index false,
 :os #object[tidb.core.Image 0x19fec3d6 "tidb.core.Image@19fec3d6"],
 :time-limit 600,
 :recovery-time 10,
 :version "master",
 :auto-retry :default,
 :workload :bank-multitable,
 :init-txn-sql nil,
 :init-sql
 ("set @@tidb_enable_mutation_checker=1, @@tidb_txn_assertion_level=strict")}

2. What did you expect to see? (Required)

No error reported by the test.

3. What did you see instead (Required)

INFO [2022-03-01 18:05:35,528] jepsen test runner - jepsen.core {:perf
 {:latency-graph {:valid? true},
  :rate-graph {:valid? true},
  :valid? true},
 :workload
 {:SI
  {:valid? false,
   :read-count 21453,
   :error-count 1,
   :first-error
   {:type :wrong-total,
    :total 96,
    :op
    {:type :ok,
     :f :read,
     :process 1654,
     :time 585631850697,
     :value {0 2, 1 16, 2 10, 3 18, 4 18, 5 4, 6 16, 7 12},
     :txn-info {:start_ts 431530414194032702},
     :index 84165}},
   :errors
   {:wrong-total
    {:count 1,
     :first
     {:type :wrong-total,
      :total 96,
      :op
      {:type :ok,
       :f :read,
       :process 1654,
       :time 585631850697,
       :value {0 2, 1 16, 2 10, 3 18, 4 18, 5 4, 6 16, 7 12},
       :txn-info {:start_ts 431530414194032702},
       :index 84165}},
     :worst
     {:type :wrong-total,
      :total 96,
      :op
      {:type :ok,
       :f :read,
       :process 1654,
       :time 585631850697,
       :value {0 2, 1 16, 2 10, 3 18, 4 18, 5 4, 6 16, 7 12},
       :txn-info {:start_ts 431530414194032702},
       :index 84165}},
     :last
     {:type :wrong-total,
      :total 96,
      :op
      {:type :ok,
       :f :read,
       :process 1654,
       :time 585631850697,
       :value {0 2, 1 16, 2 10, 3 18, 4 18, 5 4, 6 16, 7 12},
       :txn-info {:start_ts 431530414194032702},
       :index 84165}},
     :lowest
     {:type :wrong-total,
      :total 96,
      :op
      {:type :ok,
       :f :read,
       :process 1654,
       :time 585631850697,
       :value {0 2, 1 16, 2 10, 3 18, 4 18, 5 4, 6 16, 7 12},
       :txn-info {:start_ts 431530414194032702},
       :index 84165}},
     :highest
     {:type :wrong-total,
      :total 96,
      :op
      {:type :ok,
       :f :read,
       :process 1654,
       :time 585631850697,
       :value {0 2, 1 16, 2 10, 3 18, 4 18, 5 4, 6 16, 7 12},
       :txn-info {:start_ts 431530414194032702},
       :index 84165}}}}},
  :plot {:valid? true},
  :valid? false},
 :valid? false}

Analysis invalid! (ﾉಥ益ಥ）ﾉ ┻━┻