search

dfuse-io / dfuse-eosio

dfuse for EOSIO
https://dfuse.io
Apache License 2.0
202 stars 45 forks source link

dfuse statedb leaks memory #111

Closed matthewdarwin closed 4 years ago

matthewdarwin commented 4 years ago

Running on jungle, the process configured as below was taking nearly 500GB of RSS after 12 hours

start:
  args:
    - statedb
  flags:
    config-file: ""
    log-format: text
    log-to-file: false
    common-chain-id: 2a02a0053e5a8cf73a56ba0fda11e4d92e0238a4a2aa74fccf46d5a910746840
    common-blockstream-addr: jungle-drly13.xxxxx.local:9000
    common-blocks-store-url: s3://ceph-gw13.xxxxx.local:7480/jungle-dfuse-merged-blocks?region=none&insecure=true
    statedb-http-listen-addr: :8080
    statedb-store-dsn: tikv://jungle-pd1.xxxxx.local,jungle-pd2.xxxxx.local,jungle-pd3.xxxxx.local:2379/state # kvdb connection string to State database
    statedb-enable-server-mode: false # Enables flux server mode, launch a server
    statedb-enable-inject-mode: true # Enables flux inject mode, writes into its database
    statedb-enable-pipeline: true # Enables statedb without a blocks pipeline, useful for running a development server (**do not** use this in prod)
    # statedb-reproc-shard-store-url: s3://ceph-gw13.xxxxx.local:7480/jungle-dfuse-v1-reproc-shards?region=none&insecure=true # [BATCH] Storage url where all reproc shard write requests should be written to
    # Default configs from here:
    statedb-enable-reproc-sharder-mode: false # [BATCH] Enables flux reproc shard mode, exclusive option, cannot be set if either server, injector or reproc-injector mode is set
    statedb-enable-reproc-injector-mode: false # [BATCH] Enables flux reproc injector mode, exclusive option, cannot be set if either server, injector or reproc-shard mode is set
    statedb-reproc-shard-count: 0 # [BATCH] Number of shards to split in (in 'reproc-sharder' mode), or join (in 'reproc-injector' mode)
    statedb-reproc-shard-start-block-num: 0 # [BATCH] Start processing block logs at this height, must be on a 100-blocks boundary
    statedb-reproc-shard-stop-block-num: 0 # [BATCH] Stop processing block logs at this height, must be on a 100-blocks boundary
    statedb-reproc-injector-shard-index: 0 # [BATCH] Index of the shard to perform injection for, should be lower than shard-count
sduchesneau commented 4 years ago

fixed, the issue was actually in the initialization of start block in the statedb: the forkable was eating the first block, and there was a gate AFTER the forkable which was also expecting that first block before it started. I removed that bstream.Gate