the jepsen case append-optimistic-predicate-follower-neterr failed

[ChenPeng2013]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

12:50:15.389 [main] INFO  jepsen.cli - Test options:
 {:auto-retry-limit :default,
 :ssh-private-key "jepsen.pem",
 :isolation :repeatable-read,
 :concurrency 15,
 :predicate-read true,
 :tarball-url
 "http://minio.pingcap.net:9000/tp-team/tests/jepsen/tidb-v6.5.1-pre-amd64.tar.gz",
 :txn-mode "optimistic",
 :force-reinstall true,
 :update-in-place false,
 :ssh
 {:username "root",
  :password "root",
  :strict-host-key-checking false,
  :private-key-path "jepsen.pem"},
 :nemesis-long-recovery false,
 :follower-read true,
 :nemesis
 {:interval 10,
  :schedules true,
  :partition-pd-leader true,
  :partition-half true,
  :partition-ring true},
 :nodes
 ["node-0.node-peer.jepsen-tps-1654165-1-532"
  "node-1.node-peer.jepsen-tps-1654165-1-532"
  "node-2.node-peer.jepsen-tps-1654165-1-532"
  "node-3.node-peer.jepsen-tps-1654165-1-532"
  "node-4.node-peer.jepsen-tps-1654165-1-532"],
 :test-count 1,
 :read-lock nil,
 :use-index false,
 :table-cache false,
 :os #object[tidb.core.Image 0x604d28c6 "tidb.core.Image@604d28c6"],
 :time-limit 300,
 :recovery-time 10,
 :version "master",
 :auto-retry :default,
 :workload :append,
 :init-txn-sql nil,
 :init-sql
 ("set @@tidb_enable_mutation_checker=1, @@tidb_txn_assertion_level=strict, @@tidb_constraint_check_in_place_pessimistic=off")}

2. What did you expect to see? (Required)

test passed

3. What did you see instead (Required)

 {:latency-graph {:valid? true},
  :rate-graph {:valid? true},
  :valid? true},
 :workload
 {:valid? false,
  :anomaly-types (:G-single),
  :anomalies
  {:G-single
   ("Let:\n  T1 = {:type :ok, :f :txn, :value [[:r 3297 [1]]], :process 30, :time 160951742494, :txn-info {:txn_scope \"global\", :start_ts 439860452924588115, :commit_ts 439860452937695261, :txn_commit_mode \"1pc\", :async_commit_fallback false, :one_pc_fallback false}, :index 87381}\n  T2 = {:type :ok, :f :txn, :value [[:r 3295 [1 2 3]] [:append 3293 14] [:append 3297 2]], :process 42, :time 160745971376, :txn-info {:txn_scope \"global\", :start_ts 439860452924588134, :commit_ts 439860452937695234, :txn_commit_mode \"1pc\", :async_commit_fallback false, :one_pc_fallback false}, :index 87359}\n\nThen:\n  - T1 < T2, because T1 did not observe T2's append of 2 to 3297.\n  - However, T2 < T1, because T2 completed at index 87359, 0.004 seconds before the invocation of T1, at index 87364: a contradiction!")}},
 :valid? false}

4. What is your TiDB version? (Required)

[2023/03/04 12:51:03.395 +00:00] [INFO] [printer.go:34] ["Welcome to TiDB."] ["Release Version"=v6.5.1] [Edition=Community] ["Git Commit Hash"=8fe13795127032ac30b670852a02e17b5b88e3ee] ["Git Branch"=heads/refs/tags/v6.5.1] ["UTC Build Time"="2023-03-03 03:07:29"] [GoVersion=go1.19.5] ["Race Enabled"=false] ["Check Table Before Drop"=false] ["TiKV Min Version"=6.2.0-alpha]

[cfzjywxk]

Note this invalid result is detected using real-time graph.

[cfzjywxk]

As it could not be produced by now using the debug patch version, change the severity to major.