search

quiver-team / quiver-feature

High performance RDMA-based distributed feature collection component for training GNN model on EXTREMELY large graph
Apache License 2.0
48 stars 5 forks source link

特征聚合性能测试数据 #1

Closed eedalong closed 2 years ago

eedalong commented 2 years ago

NONE

eedalong commented 2 years ago

基于RDMA的聚合，带宽利用效率是一个问题

eedalong commented 2 years ago

Our Design + with TLB Optimization

  # Params Setting， Test 1000 times
  POST_LIST_SIZE = 128
  CQ_MOD = 1
  QP_NUM = 8
  TX_DEPTH = 2048

PGAS 2机器2卡

Komodo02:

Result Check Successed! Throughput = 12791.699892329863 MB/s

Komodo01:

Result Check Successed! Throughput = 13126.010728462832 MB/s

PGAS 2机器4卡

Komodo02:

Result Check Successed! Throughput = 11182.3179829859 MB/s Result Check Successed! Throughput = 11284.241280822509 MB/s

Komodo01:

Result Check Successed! Throughput = 11560.66684214394 MB/s Result Check Successed! Throughput = 11302.003786406727 MB/s

PGAS 2机器6卡

Komodo02:

Result Check Successed! Throughput = 8272.912436920255 MB/s Result Check Successed! Throughput = 8148.610107064886 MB/s Result Check Successed! Throughput = 8283.092280248591 MB/s

Komodo01

Result Check Successed! Throughput = 8189.57876741657 MB/s Result Check Successed! Throughput = 8066.924804759149 MB/s Result Check Successed! Throughput = 8216.104604876296 MB/s

eedalong commented 2 years ago

Baseline: TensorPipe + IB + CPU

RPC 2机器2卡

Komodo02: Bandwidth in Rank 0 = 2.481324825642474GB/s

Komodo01: Bandwidth in Rank 1 = 2.4741790550186495GB/s

RPC 2机器4卡

Komodo02:

Bandwidth in Rank 1 = 2.0325350597053786GB/s Bandwidth in Rank 0 = 2.0048697618224747GB/s

Komodo01:

Bandwidth in Rank 2 = 2.027988951516192GB/s Bandwidth in Rank 3 = 2.0567716979987267GB/s

RPC 2机器6卡 Komodo02:

Bandwidth in Rank 0 = 1.6407886943095806GB/s Bandwidth in Rank 1 = 1.6125125382928056GB/s Bandwidth in Rank 2 = 1.6290961790689487GB/s

Komodo01:

Bandwidth in Rank 3 = 1.6392393241055634GB/s Bandwidth in Rank 4 = 1.6672241215764283GB/s Bandwidth in Rank 5 = 1.6509420853368661GB/s

eedalong commented 2 years ago

TensorPipe + IB + Quiver.ShardTensor(100% Data On GPU)

RPC 2机器2卡

Komodo02: Bandwidth in Rank 0 = 6.647192608055932GB/s

Komodo01:

Bandwidth in Rank 1 = 6.631595107518604GB/s

RPC 2机器4卡

Komodo02:

Bandwidth in Rank 0 = 5.6239369001778385GB/s Bandwidth in Rank 1 = 5.6514885267247585GB/s

Komodo01:

Bandwidth in Rank 2 = 5.60434187045844GB/s Bandwidth in Rank 3 = 5.640148911211554GB/s

RPC 2机器6卡 Komodo02:

Bandwidth in Rank 1 = 3.938194500600181GB/s Bandwidth in Rank 0 = 3.913047391308261GB/s Bandwidth in Rank 2 = 3.9266709899314267GB/s

Komodo01:

Bandwidth in Rank 3 = 3.8997557387993003GB/s Bandwidth in Rank 4 = 3.995974189203517GB/s Bandwidth in Rank 5 = 3.925330787625473GB/s

eedalong commented 2 years ago

Our Design + Without TLB Optimization

  # Params Setting， Test 1000 times
  POST_LIST_SIZE = 128
  CQ_MOD = 1
  QP_NUM = 8
  TX_DEPTH = 2048

PGAS 2机器2卡

Komodo02:

Result Check Successed! Throughput = 13045.923042032027 MB/s

Komodo01:

Result Check Successed! Throughput = 13071.25667929728 MB/s

PGAS 2机器4卡

Komodo02:

Result Check Successed! Throughput = 8601.997632501761 MB/s Result Check Successed! Throughput = 8505.97377417787 MB/s

Komodo01:

Result Check Successed! Throughput = 8925.55079539093 MB/s Result Check Successed! Throughput = 8866.858644932026 MB/s

PGAS 2机器6卡

Komodo02:

Result Check Successed! Throughput = 6374.34019976485 MB/s Result Check Successed! Throughput = 6348.525887581576 MB/s Result Check Successed! Throughput = 6290.3157640227355 MB/s

Komodo01

Result Check Successed! Throughput = 6377.566696559296 MB/s Result Check Successed! Throughput = 6374.742420564069 MB/s Result Check Successed! Throughput = 6342.615555745302 MB/s