How to start the DPS indexer

[kiselev-sv]

Hello! I have read https://github.com/optakt/flow-dps/blob/master/README.md and built all components flow-dps-client flow-dps-indexer flow-dps-live flow-dps-server

What are the next steps, how can I run dps-live or indexer to start from scratch?

I tried to start it, but there are errors

./flow-dps-indexer  -l debug -d /var/flow/data/protocol -t /var/flow/data/execution -c /var/flow/bootstrap/root.checkpoint -i /var/flow/data/index
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/prometheus/tsdb/wal.newSegmentBufReader(...)
        /root/gopath/pkg/mod/github.com/prometheus/tsdb@v0.7.1/wal/wal.go:716
github.com/prometheus/tsdb/wal.NewSegmentsRangeReader(0xc000591808, 0x1, 0x1, 0xc00000e420, 0x1, 0x0, 0x0)
        /root/gopath/pkg/mod/github.com/prometheus/tsdb@v0.7.1/wal/wal.go:699 +0x770
github.com/prometheus/tsdb/wal.NewSegmentsReader(...)
        /root/gopath/pkg/mod/github.com/prometheus/tsdb@v0.7.1/wal/wal.go:671
main.run(0x0)
        /root/flow-dps/cmd/flow-dps-indexer/main.go:135 +0xe53
main.main()

Where can I find root.checkpoint? Any information needed Thank You

[Ullaakut]

Hi @kiselev-sv !

Thanks for opening this issue, sorry I did not see it later.

Indeed, trying to run the DPS with an empty hand-made root checkpoint will not work, this is expected.

However, it is your lucky day, since last Friday the Flow team made public all of the information required to run your own DPS instance!

See the sporks.json file in the Flow repository for the information for each spork. Besides that, the currently available documentation of the Flow DPS should be enough for you to figure out easily what goes in which flag.

Keep in mind that only the current network can be indexed using the flow-dps-live binary, and only historical networks can be indexed with flow-dps-indexer.

Once you have an index folder built out and complete, you can use the Flow DPS Rosetta API or the Flow DPS Access API to expose the data through a known API standard, or you can build your own binary that reads data from the index database.

Cheers!

[Ullaakut]

Also, in case you did not already find it, the documentation that explains how to index data is available here, in our Getting Started guide.

[kiselev-sv]

I tried to index mainnet-14 spork so downloaded root.checkpoint to /var/flow/bootstrap node-infos.pub.json root-protocol-state-snapshot.json to /var/flow/bootstrap/public-root-information and flow_public_mainnet14_execution_state to /var/flow/data/execution

and tried to start indexer: ~/flow-dps-bin/1.4.8 # ./dps-indexer -l debug -d /var/flow/data/protocol -t /var/flow/data/execution -c /var/flow/bootstrap/root.checkpoint -i /var/flow/data/index panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]: github.com/prometheus/tsdb/wal.newSegmentBufReader(...) /home/runner/go/pkg/mod/github.com/prometheus/tsdb@v0.7.1/wal/wal.go:716 github.com/prometheus/tsdb/wal.NewSegmentsRangeReader({0xc0003e3980, 0x1, 0x1094960}) /home/runner/go/pkg/mod/github.com/prometheus/tsdb@v0.7.1/wal/wal.go:699 +0x22d github.com/prometheus/tsdb/wal.NewSegmentsReader(...) /home/runner/go/pkg/mod/github.com/prometheus/tsdb@v0.7.1/wal/wal.go:671 main.run() /home/runner/work/flow-dps/flow-dps/cmd/flow-dps-indexer/main.go:135 +0xd4f main.main() /home/runner/work/flow-dps/flow-dps/cmd/flow-dps-indexer/main.go:46 +0x19

what I did wrong?

[Ullaakut]

Hi @kiselev-sv ! Thanks for coming back so quickly.

If you get this error, it's that the checkpoint you are passing is not a valid checkpoint. Are you sure you downloaded the mainnet-14 checkpoint? It should be approximately 29GB of binary data, if my memory serves me well.

Could you show me what commands you ran exactly to download and extract/move the files to the folders you mention above? Thanks.

Also in your message I see another discrepancy: You used /var/flow/data/protocol for the protocol data folder but did not mention putting anything there.

[kiselev-sv]

1. wget https://storage.googleapis.com/flow-genesis-bootstrap/mainnet-14-execution/public-root-information/root.checkpoint direct to /var/flow/bootstrap/ (about 29G)
2. wget https://storage.googleapis.com/flow-genesis-bootstrap/mainnet-14-execution/public-root-information/root-protocol-state-snapshot.json direct to /var/flow/bootstrap/public-root-information
3. wget https://storage.googleapis.com/flow-genesis-bootstrap/mainnet-14-execution/public-root-information/node-infos.pub.json to /var/flow/bootstrap/public-root-information
4. wget https://storage.googleapis.com/flow_public_mainnet14_execution_state to /var/flow/data/execution/

[bluesign]

@kiselev-sv on 4h you need to download all files from there individually.

[kiselev-sv]

Yes, thank you! Downloaded with gsutil

So now I have ./dps-live -u flow-block-data -i /var/flow/index -d /var/flow/data -c /var/flow/bootstrap/public-root-information/root.checkpoint -b /var/flow/bootstrap --seed-address access-001.mainnet14.nodes.onflow.org:9000 --seed-key 28a0d9edd0de3f15866dfe4aea1560c4504fe313fc6ca3f63a63e4f98d0e295144692a58ebe7f7894349198613f65b2d960abf99ec2625e247b1c78ba5bf2eae {"level":"info","time":"2022-02-17T16:16:31Z","message":"Flow DPS Live Server starting"} {"level":"info","node_role":"consensus_follower","node_id":"not set","time":"2022-02-17T16:16:31Z","message":"flow consensus_follower node starting up"} {"level":"info","start":"2022-02-17T16:16:31Z","time":"2022-02-17T16:16:31Z","message":"Flow DPS Live Indexer starting"} {"level":"info","component":"mapper_transitions","commit":"568f4ec740fe3b5de88034cb7b1fbddb41548b068f31aebc8ae9189e429c5749","time":"2022-02-17T16:16:31Z","message":"added empty tree to forest"} {"level":"info","node_role":"consensus_follower","node_id":"not set","block_id":"23ac4033dee62797378d7897ccccd63aacac9124eb375c5c6d82008b07732995","height":19050753,"time":"2022-02-17T16:16:34Z","message":"last finalized block"} {"level":"info","node_role":"consensus_follower","node_id":"not set","component":"protocol-state-id-cache","blockID":"23ac4033dee62797378d7897ccccd63aacac9124eb375c5c6d82008b07732995","time":"2022-02-17T16:16:34Z","message":"updating cached identities"} {"level":"info","node_role":"consensus_follower","node_id":"not set","module":"id providers","time":"2022-02-17T16:16:34Z","message":"module initialization complete"} {"level":"info","node_role":"consensus_follower","node_id":"not set","module":"network middleware","time":"2022-02-17T16:16:34Z","message":"module initialization complete"} {"level":"info","node_role":"consensus_follower","node_id":"not set","module":"mutable follower state","time":"2022-02-17T16:16:34Z","message":"module initialization complete"} {"level":"info","node_role":"consensus_follower","node_id":"not set","module":"sync core","time":"2022-02-17T16:16:34Z","message":"module initialization complete"} {"level":"info","node_role":"consensus_follower","node_id":"not set","module":"committee","time":"2022-02-17T16:16:35Z","message":"module initialization complete"} {"level":"info","node_role":"consensus_follower","node_id":"not set","module":"latest header","time":"2022-02-17T16:16:35Z","message":"module initialization complete"} {"level":"info","node_role":"consensus_follower","node_id":"not set","time":"2022-02-17T16:16:35Z","message":"network will run on address: 0.0.0.0:0"} {"level":"info","node_role":"consensus_follower","node_id":"not set","component":"unstaked network","time":"2022-02-17T16:16:35Z","message":"component initialization complete"} {"level":"info","node_role":"consensus_follower","node_id":"not set","time":"2022-02-17T16:16:35Z","message":"Updating protocol state node addresses"} {"level":"info","node_role":"consensus_follower","node_id":"not set","component":"unstaked network","time":"2022-02-17T16:16:35Z","message":"component startup complete"} {"level":"info","node_role":"consensus_follower","node_id":"not set","component":"upstream connector","time":"2022-02-17T16:16:35Z","message":"component initialization complete"} {"level":"error","node_role":"consensus_follower","node_id":"not set","bootstap_node":"access-0000000000000000000000000000000000000000000000000000000000000000@access-001.mainnet14.nodes.onflow.org:9000=0","error":"failed to dial QmUyQoWbG4Nb4FZwHybKPfdDw9DzJ7JC41s1cEjTf9rSXo:\n [/ip4/23.236.58.237/tcp/9000] failed to negotiate security protocol: message did not have trailing newline","time":"2022-02-17T16:16:35Z","message":"failed to connect"} {"level":"fatal","node_role":"consensus_follower","node_id":"not set","error":"1 error occurred:\n\t failed to dial QmUyQoWbG4Nb4FZwHybKPfdDw9DzJ7JC41s1cEjTf9rSXo:\n * [/ip4/23.236.58.237/tcp/9000] failed to negotiate security protocol: message did not have trailing newline\n\n","time":"2022-02-17T16:16:35Z","message":"Failed to connect to a bootstrap node. Please ensure the network address and public key of the bootstrap access node are correct and that the node is running and reachable."}

Where can I find any seeds? Is it a main problem?

[kiselev-sv]

All info at https://github.com/onflow/flow/blob/master/sporks.json

with latest root-protocol-state-snapshot.json for mainnet16 and dps 1.4.8

{"level":"error","error":"could not decode protocol snapshot: json: cannot unmarshal string into Go struct field Seal.LatestSeal.FinalState of type flow.StateCommitment","time":"2022-03-03T08:17:47Z","message":"could not initialize protocol state"} Can anybody help?

[kiselev-sv]

Downloaded and extracted protocol, tried to start indexer on mainnet-15

root@ph243 /nodes-data/flow/flow-dps-bin/1.4.8 # ./dps-indexer -l debug -t /nodes-data/flow/data/execution -d /nodes-data/flow/data/protocol -c /nodes-data/flow/bootstrap/public-root-information/root.checkpoint {"level":"info","start":"2022-03-06T03:53:42Z","time":"2022-03-06T03:53:42Z","message":"Flow DPS Indexer starting"} {"level":"info","component":"mapper_transitions","commit":"568f4ec740fe3b5de88034cb7b1fbddb41548b068f31aebc8ae9189e429c5749","time":"2022-03-06T03:53:42Z","message":"added empty tree to forest"} Killed

[1674969.073514] Out of memory: Kill process 10137 (dps-indexer) score 925 or sacrifice child [1674969.073900] Killed process 10137 (dps-indexer) total-vm:196580572kB, anon-rss:182859944kB, file-rss:0kB, shmem-rss:0kB

I have MemTotal: 197945536 kB MemFree: 182456272 kB MemAvailable: 184011180 kB Buffers: 1337292 kB Cached: 1376516 kB SwapCached: 0 kB Active: 11389932 kB Inactive: 1199872 kB Active(anon): 10000848 kB Inactive(anon): 1176 kB Active(file): 1389084 kB Inactive(file): 1198696 kB Unevictable: 269308 kB Mlocked: 269308 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 47000 kB Writeback: 0 kB AnonPages: 10144964 kB Mapped: 402216 kB Shmem: 1736 kB Slab: 1743600 kB SReclaimable: 739352 kB SUnreclaim: 1004248 kB KernelStack: 24064 kB PageTables: 255520 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 98972768 kB Committed_AS: 15436376 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB Percpu: 112128 kB :

[1674973.803671] oom_reaper: reaped process 10137 (dps-indexer), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

How much memory it needs?

[chaen-foundry]

All info at https://github.com/onflow/flow/blob/master/sporks.json

with latest root-protocol-state-snapshot.json for mainnet16 and dps 1.4.8

{"level":"error","error":"could not decode protocol snapshot: json: cannot unmarshal string into Go struct field Seal.LatestSeal.FinalState of type flow.StateCommitment","time":"2022-03-03T08:17:47Z","message":"could not initialize protocol state"} Can anybody help?

@kiselev-sv Have you gotten dps-live working with mainnet-16? this is as far as i've got https://github.com/dapperlabs/dps-rosetta-docker/pull/8

[kiselev-sv]

no, I am not

[admincheg]

@chaen-foundry as I see - new protocol state has new fields and formats, which not compatible with old version of flow-go. You can compare formats with mainnet-14, but after fixing live didn't start properly. Fresh run with empty database ends with segmentation fault (strange defer behavior, it closes resources before accessing to they) and next runs just silently shutdown. If you remove data and index directories - segmentation fault returns. If you comment defers in code of live module - first run doesn't produce SIGSEGV, but it's still not working.

[chaen-foundry]

@admincheg I ended up getting it to work (partly), here's the dockerfile. As it turns out the access node port is 3570 not 3569 like it says here. Now it is able to index mainnet 16 from block 23830813 to 23916853 but then it gets stuck with the error:

{"level":"warn","node_role":"consensus_follower","node_id":"237b0eedfe14a2bca930014a1475a7f3ac6d122ea8bed0463d860c00efbcf809","engine":"follower","error_type":"outdated_input","error":"could not process block proposal: could not process pending children: 1 error occurred:

no error is listed after "1 error occured:" This error comes from flow-go here, not sure how to fix this.

[pgebheim]

@admincheg I ended up getting it to work (partly), here's the dockerfile. As it turns out the access node port is 3570 not 3569 like it says here. Now it is able to index mainnet 16 from block 23830813 to 23916853 but then it gets stuck with the error:

{"level":"warn","node_role":"consensus_follower","node_id":"237b0eedfe14a2bca930014a1475a7f3ac6d122ea8bed0463d860c00efbcf809","engine":"follower","error_type":"outdated_input","error":"could not process block proposal: could not process pending children: 1 error occurred:

no error is listed after "1 error occured:" This error comes from flow-go here, not sure how to fix this.

Thanks @chaen-foundry -- @m4ksio from the Flow team is working on testing out DPS for mainnet-16. Should be able to see if he can reproduce this issue / can see why it would be happening.

[m4ksio]

What version are you trying to run? We have tested https://github.com/dapperlabs/flow-dps/tree/v0.24 and it was running fine, but haven't really check in a few weeks

[chaen-foundry]

@m4ksio I was using v0.24 https://github.com/chaen-foundry/flow-dps-docker/blob/mainnet-16.1/Dockerfile Then once it gets up to block 23916853 the GetLast gRPC endpoint will just get stuck there. Do you have a Dockerfile for mainnet-16 that you were using that I could test?

[chaen-foundry]

@m4ksio any luck with that? I haven't been able to get it running on mainnet-17 yet with this error cannot bootstrap invalid root snapshot: invalid root sealing segment: invalid segment: failed to lookup execution result: (59f5122101be0430179d96cbc50c175abf904f7cfd50c260bf337af677688bb2) result (id=59f5122101be0430179d96cbc50c175abf904f7cfd50c260bf337af677688bb2) not found in segment