roachperf: regression around 2022-07-07

[renatolabs]

kv95:

Similar drops are seen in kv95/nodes=3 and some ycsb benchmarks.

% GOOD_SHA=41228d10ac5e326328507b98245fc076d67d2ecd
% BAD_SHA=7405c568e482fbfff4ad2bcba101ef42678d0b1a
% git log $BAD_SHA --not $GOOD_SHA^ --topo-order --merges --pretty="format:%h %s (%ch)" | sed 's/^/- [ ] /g'

• [ ] ab9eabfac4 Merge #84006 #84036 #84051 (Fri Jul 8 16:53)
• [ ] 4d8d0b57ce Merge #83529 #83678 #83795 #84007 #84046 (Fri Jul 8 15:05)
• [ ] 18f6450dfb Merge #84062 (Fri Jul 8 13:53)
• [ ] 672f201c1c Merge #83729 (Fri Jul 8 06:11)
• [ ] cd1abbb383 Merge #84005 (Fri Jul 8 04:58)
• [ ] 88789feea1 Merge #83484 (Fri Jul 8 03:49)
• [ ] 8edc7ba730 Merge #83941 (Fri Jul 8 01:41)
• [ ] 365b9c4a4b Merge #83686 (Fri Jul 8 00:20)
• [ ] 2dd8e762bb Merge #83014 #83707 #84003 #84039 (Thu Jul 7 22:04)
• [ ] 6b7082af58 Merge #83875 #83945 #83992 #84015 (Thu Jul 7 20:57)
• [ ] e67e47f722 Merge #83720 #83732 #83938 #83953 (Thu Jul 7 18:12)
• [x] bce22672f7 Merge #83123 #83590 #83943 #83961 #83975 #83979 (Thu Jul 7 17:02) 🔴🔴 (mean: 63677.4, stdev: 523.7)
• [ ] 4dfbe65c8a Merge #83547 #83896 #83903 #83926 (Thu Jul 7 15:44)
• [ ] 3b22cddc26 Merge #83908 (Thu Jul 7 13:03)
• [ ] 5fb4478b94 Merge #83331 #83685 (Thu Jul 7 07:25)
• [ ] f636444094 Merge #83095 #83918 (Thu Jul 7 05:41)
• [ ] f93ca3a91f Merge #83804 (Thu Jul 7 04:33)
• [ ] 03d5260a46 Merge #83870 #83876 #83924 #83947 #83952 (Thu Jul 7 02:58)
• [x] 1abab4ca0b Merge #83812 #83923 #83927 #83944 (Thu Jul 7 01:52) 🔴🔴 (mean: 62634.5, stdev: 1940.1)
• [x] a0b12222e8 Merge #83675 #83800 #83838 (Thu Jul 7 00:15) 🟡🟡 (mean: 64224.3, stdev: 704.4)
• [x] ce1b42bd34 Merge #83616 #83762 #83873 #83902 (Wed Jul 6 22:45) 🔴🔴 (mean: 64129.5, stdev: 1555.2)
• [x] 0848d90c4e Merge #83134 #83615 (Wed Jul 6 21:37) 🔴🔴 (mean: 62893.2, stdev: 2932.7) -- second run: 🟢🟢 (mean: 67211.4, stdev: 1107.4).
  • Reverting #83615: 🟢🟢 (mean: 68235.1, stdev: 269.6)
  • Reverting #83134: 🟢🟢 (mean: 68038.3, stdev: 286.4)
  • ❓❓
• [x] 41228d10ac Merge #83684 #83709 #83834 (Wed Jul 6 20:27) 🟢🟢 (mean: 67750.42, stdev: 742.2)
• [x] 264cd04d43 Merge #78519 #83421 (Wed Jul 6 19:20) 🟢 🟢 (mean: 68207.9, stdev: 413.9)
• [ ] 7cfa5d4af8 Merge #83866 (Wed Jul 6 14:04)
• [ ] 8b6e3c3fd3 Merge #83703 (Wed Jul 6 12:57)
• [ ] d239a2fbf9 Merge #83366 #83629 (Wed Jul 6 06:37)
• [ ] 90b15cf2df Merge #82865 #83358 #83683 (Wed Jul 6 05:27)
• [ ] 33d7099871 Merge #82151 (Wed Jul 6 02:25)
• [x] 9c5472e9dc Merge #83711 #83733 #83809 (Tue Jul 5 19:22) 🟢🟢 (mean: 68017, stdev: 1618.3)

Jira issue: CRDB-17923

[smg260]

This regression seems to coincide with https://github.com/cockroachdb/cockroach/issues/88122 which singles out sql, sqlstats: create temporary stats container for all txns as the probable cause. Merge commit: ce1b42b

[srosenberg]

Root cause appears to be sqlstats, closing.