[infra] 600 台立ち上げの素振りをする on 8/19

ShotaKitazawa commented 3 years ago

本番ポータルを使ってベンチ600台立ち上げ試験を実施する

broker: 本番ポータルの用意 by @takonomura さん

ShotaKitazawa commented 3 years ago

contestant VM も1セット展開し、ポータルからベンチが実行できるかも確認する

ShotaKitazawa commented 3 years ago

起動済みベンチ一覧の取得 (TODO: ソート)

aws ec2 describe-instances \
  --filter "Name=tag:Kind,Values=bench" \
  --filter "Name=instance-state-name,Values=running" \
  | jq -r '.Reservations[].Instances[].Tags[] | select(.Key == "Name") | .Value'

追記: このコマンド1つ目の filter 効いてない

ShotaKitazawa commented 3 years ago

[x] ベンチ 600 台展開出来た
- 多重で作成リクエストが送られて duplicated address みたいなのが出て失敗した → terraform import で管理下に入れて解決
[x] 初期状態でベンチを回せた
- なんか isucondition.go が上がってこないケースもあった (env.sh が無いらしい) ので、要確認 → #1385
[ ] Ansible で bench の置き換えが出来た
- 1台に対してなら問題ないのに、600 台に対してだと何故かコケた
- Playbook の実行は、踏み台サーバでやるほうが時間がかからんので、そっちでやる
[x] ブラウザから競技者 VM → JIA Auth へアクセスしてログインできた
- jia-auth.xi.isucon.dev が 503 だった😇 → security group がバグってたので修正した

ShotaKitazawa commented 3 years ago

1台に対してなら問題ないのに、600 台に対してだと何故かコケた

https://portal.isucon.net/admin/benchmark_jobs/17

ベンチ側の /var/log/isuxportal-supervisor/job-17.err.log には普通のベンチ実行結果が入ってた

portal への gRPC 再接続の負荷が原因っぽい？

03:41:06.920617 ISUCON11 benchmarker 329b55d2
03:41:06.920874 start: load initial data
03:41:06.923617 finish: load initial data
03:41:12.395389 SCORE: 00.StartBenchmark       : 0
03:41:12.395413 LOAD INFO
Language: go
Campaign: None
03:41:12.395419 SCORE: 01.GraphExcellent       : 0
03:41:12.395448 SCORE: 02.GraphGood            : 0
03:41:12.395456 SCORE: 03.GraphNormal          : 0
03:41:12.395463 SCORE: 04.GraphBad             : 0
03:41:12.395469 SCORE: 05.GraphWorst           : 0
03:41:12.395516 SCORE: 06.TodayGraphExcellent  : 0
03:41:12.395527 SCORE: 07.TodayGraphGood       : 0
03:41:12.395534 SCORE: 08.TodayGraphNormal     : 0
03:41:12.395540 SCORE: 09.TodayGraphBad        : 0
03:41:12.395547 SCORE: 10.TodayGraphWorst      : 0
03:41:12.395553 SCORE: 11.ReadInfoCondition    : 0
03:41:12.395561 SCORE: 12.ReadWarningCondition : 0
03:41:12.395567 SCORE: 13.ReadCriticalCondition: 0
03:41:12.395575 SCORE: _1.IsuInitialize        : 0
03:41:12.395582 SCORE: _2.NormalUserInitialize : 0
03:41:12.395588 SCORE: _3.ViewerInitialize     : 0
03:41:12.395595 SCORE: _4.ViewerDropout        : 0
03:41:12.395602 SCORE: _5.RepairIsu            : 0
03:41:12.395609 SCORE: _6.PostInfoCondition    : 0
03:41:12.395616 SCORE: _7.PostWarningCondition : 0
03:41:12.395622 SCORE: _8.PostCriticalCondition: 0
03:41:15.395386 SCORE: 00.StartBenchmark       : 1
03:41:15.395421 SCORE: 01.GraphExcellent       : 0
03:41:15.395428 SCORE: 02.GraphGood            : 0
03:41:15.395432 SCORE: 03.GraphNormal          : 0
03:41:15.395436 SCORE: 04.GraphBad             : 0
...
03:42:09.402212 SCORE: 00.StartBenchmark       : 1
03:42:09.402251 SCORE: 01.GraphExcellent       : 0
03:42:09.402259 SCORE: 02.GraphGood            : 0
03:42:09.402264 SCORE: 03.GraphNormal          : 0
03:42:09.402268 SCORE: 04.GraphBad             : 0
03:42:09.402272 SCORE: 05.GraphWorst           : 53
03:42:09.402276 SCORE: 06.TodayGraphExcellent  : 0
03:42:09.402280 SCORE: 07.TodayGraphGood       : 0
03:42:09.402285 SCORE: 08.TodayGraphNormal     : 0
03:42:09.402289 SCORE: 09.TodayGraphBad        : 0
03:42:09.402293 SCORE: 10.TodayGraphWorst      : 28
03:42:09.402297 SCORE: 11.ReadInfoCondition    : 17
03:42:09.402301 SCORE: 12.ReadWarningCondition : 50
03:42:09.402305 SCORE: 13.ReadCriticalCondition: 19
03:42:09.402309 SCORE: _1.IsuInitialize        : 48
03:42:09.402313 SCORE: _2.NormalUserInitialize : 7
03:42:09.402317 SCORE: _3.ViewerInitialize     : 21
03:42:09.402322 SCORE: _4.ViewerDropout        : 21
03:42:09.402327 SCORE: _5.RepairIsu            : 18
03:42:09.402334 SCORE: _6.PostInfoCondition    : 867
03:42:09.402341 SCORE: _7.PostWarningCondition : 3490
03:42:09.402348 SCORE: _8.PostCriticalCondition: 8731
03:42:12.396009 LOAD WAIT
03:42:12.423149 <=== LOAD END
03:42:12.423201 SCORE: 00.StartBenchmark       : 1
03:42:12.423209 SCORE: 01.GraphExcellent       : 0
03:42:12.423213 SCORE: 02.GraphGood            : 0
03:42:12.423218 SCORE: 03.GraphNormal          : 0
03:42:12.423222 SCORE: 04.GraphBad             : 0
03:42:12.423226 SCORE: 05.GraphWorst           : 53
03:42:12.423230 SCORE: 06.TodayGraphExcellent  : 0
03:42:12.423234 SCORE: 07.TodayGraphGood       : 0
03:42:12.423239 SCORE: 08.TodayGraphNormal     : 0
03:42:12.423243 SCORE: 09.TodayGraphBad        : 0
03:42:12.423247 SCORE: 10.TodayGraphWorst      : 28
03:42:12.423252 SCORE: 11.ReadInfoCondition    : 20
03:42:12.423255 SCORE: 12.ReadWarningCondition : 53
03:42:12.423259 SCORE: 13.ReadCriticalCondition: 20
03:42:12.423263 SCORE: _1.IsuInitialize        : 48
03:42:12.423267 SCORE: _2.NormalUserInitialize : 7
03:42:12.423271 SCORE: _3.ViewerInitialize     : 21
03:42:12.423276 SCORE: _4.ViewerDropout        : 21
03:42:12.423280 SCORE: _5.RepairIsu            : 18
03:42:12.423284 SCORE: _6.PostInfoCondition    : 892
03:42:12.423288 SCORE: _7.PostWarningCondition : 3515
03:42:12.423293 SCORE: _8.PostCriticalCondition: 9785
03:42:12.426973 <=== sendResult finish

isucon / isucon11-qualify

[infra] 600 台立ち上げの素振りをする on 8/19 #1316