search

aptos-labs / aptos-ait3

6 stars 9 forks source link

Many failed proposals #67

Open Kant1742 opened 2 years ago

Kant1742 commented 2 years ago

During the previous stress test the node couldn't propose at all and currently having many failed proposals. I have always updated docker images, and because it was recommended to keep the ip, I decided not to move to another server, which is the reason I can't even get 80% rewards performance. Address 0xf2e802753bbaa2b3e0861180dfd2dd3fe8dc9bb0d49e0094376f6d565bab9884

SSD, 350GB (50gb free), here IOPS:

root@v1352565:~# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=fiotest --filename=testfio --bs=4k --iodepth=64 --size=8G --readwrite=randrw --rwmixread=75
fiotest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.16
Starting 1 process
fiotest: Laying out IO file (1 file / 8192MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=77.8MiB/s,w=26.3MiB/s][r=19.9k,w=6721 IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=151527: Sun Sep 11 14:38:31 2022
  read: IOPS=18.4k, BW=71.9MiB/s (75.4MB/s)(6141MiB/85361msec)
   bw (  KiB/s): min=45008, max=108288, per=99.79%, avg=73516.88, stdev=10840.61, samples=170
   iops        : min=11252, max=27072, avg=18379.23, stdev=2710.16, samples=170
  write: IOPS=6150, BW=24.0MiB/s (25.2MB/s)(2051MiB/85361msec); 0 zone resets
   bw (  KiB/s): min=14864, max=36968, per=99.79%, avg=24549.61, stdev=3654.85, samples=170
   iops        : min= 3716, max= 9242, avg=6137.38, stdev=913.71, samples=170
  cpu          : usr=7.33%, sys=41.20%, ctx=597253, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=1572145,525007,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=71.9MiB/s (75.4MB/s), 71.9MiB/s-71.9MiB/s (75.4MB/s-75.4MB/s), io=6141MiB (6440MB), run=85361-85361msec
  WRITE: bw=24.0MiB/s (25.2MB/s), 24.0MiB/s-24.0MiB/s (25.2MB/s-25.2MB/s), io=2051MiB (2150MB), run=85361-85361msec

Disk stats (read/write):
  vda: ios=1589291/532192, merge=10/5853, ticks=4337161/1044676, in_queue=1434740, util=99.56%

  read: IOPS=18.4k, BW=71.9MiB/s (75.4MB/s)(6141MiB/85361msec)

Speedtest: speedtest

nload. Incoming incoming

nload. Outgoing outgoing

htop htop

Grafana http://95.216.2.219:3000/d/tWti5eZ4k/aptos-validator-overview-by-l0vd?orgId=1&refresh=30s&var-old_node_name=l0vd2-devnet&var-old_HostAddr=All&var-node_name=KANT1742&var-HostAddr=All&var-shortHostAddr=All&var-IOwaitDevice=&from=now-6h&to=now

Logs logs.log

Edit: Playing with concurrency_level and DownloadLatestStates didn't help.