[Access] Test pebble execution data db on testnet

[peterargue]

Using an observer on testnet, test running execution sync using a pebble db datastore.

Monitor logs and metrics for any errors or halts while syncing, or any other signs of performance issues.

You can use the configs from localnet to setup a local version of grafana to view metrics if that's useful

[peterargue]

@UlyanaAndrukhiv can you add details about how to set this up

[UlyanaAndrukhiv]

The Pebble execution data DB setup requires the following flags:

--execution-data-db=pebble
--execution-data-dir=<path_to_pebble_db>
--execution-data-sync-enabled=true

[Guitarheroua]

@peterargue

To check the pebble on mainnet, the Observer Node was up and running with next arguments:

docker run --rm \
-v $PWD/flow_observer/bootstrap:/bootstrap:ro \
-v $PWD/flow_observer/data:/data:rw \
--name flow_observer \
-p 80:80 \
-p 8080:8080 \
-p 3569:3569 \
-p 9000:9000 \
-p 9001:9001 \
gcr.io/flow-container-registry/observer:v0.37.10 \
--bootstrapdir=/bootstrap \
--datadir=/data/protocol \
--bind 0.0.0.0:3569 \
--rest-addr=:80 \
--loglevel=error \
--secretsdir=/data/secrets  \
--upstream-node-addresses=access-008.mainnet25.nodes.onflow.org:9001 \
--upstream-node-public-keys=11742552d21ac93da37ccda09661792977e2ca548a3b26d05f22a51ae1d99b9b75c8a9b3b40b38206b38951e98e4d145f0010f8942fd82ddf0fb1d670202264a \
--bootstrap-node-addresses=access-008.mainnet25.nodes.onflow.org:3570 \
--bootstrap-node-public-keys=11742552d21ac93da37ccda09661792977e2ca548a3b26d05f22a51ae1d99b9b75c8a9b3b40b38206b38951e98e4d145f0010f8942fd82ddf0fb1d670202264a \
--observer-networking-key-path=/bootstrap/network.key \
--execution-data-db=pebble \
--execution-data-dir=/data/execution-data \
--execution-data-sync-enabled=true \
--execution-state-dir=/data/execution-state \
--execution-state-checkpoint=/bootstrap/execution-state

After syncing up data, I got the next content in my /data/execution-state/blobstore:

total 3.1G
-rw-r--r-- 1 root root 5.7M Sep 13 15:03 002377.sst
-rw-r--r-- 1 root root  16M Sep 13 15:30 002817.sst
-rw-r--r-- 1 root root 8.1M Sep 13 15:30 002879.sst

...

-rw-r--r-- 1 root root 5.5M Sep 13 16:56 005085.sst
-rw-r--r-- 1 root root 6.2M Sep 13 16:56 005086.sst
-rw-r--r-- 1 root root 1.5M Sep 13 16:56 005087.sst
-rw-r--r-- 1 root root   16 Sep 13 12:13 CURRENT
-rw-r--r-- 1 root root    0 Sep 13 12:13 LOCK
-rw-r--r-- 1 root root 866K Sep 13 16:56 MANIFEST-000001
-rw-r--r-- 1 root root 2.5K Sep 13 12:13 OPTIONS-000003
-rw-r--r-- 1 root root    0 Sep 13 12:13 marker.format-version.000015.016
-rw-r--r-- 1 root root    0 Sep 13 12:13 marker.manifest.000001.MANIFEST-000001

So the data is stored in pebble db and *.sst files store the data.

During the sync process, however, I got errors, that may be not related to pebble, and they do not crash the node, but I mention them here:

{
  "level": "error",
  "node_role": "observer",
  "node_id": "bfe8cfd3018871cc104a61b8c5971ba9b7518fcc8e4ec77792896de573ed121e",
  "engine": "follower_core",
  "origin_id": "1f46b4d23c7557e7b3c21f85aebddcb7bcda7cc4d589732ce53087cb1b5faf8e",
  "chain_id": "flow-mainnet",
  "first_block_height": 86932614,
  "first_block_view": 957352,
  "last_block_height": 86932614,
  "last_block_view": 957352,
  "last_block_id": "83ba52d0f9467266a4a698c9e0a955b72476b9f940de7a30873ee01cd24ad2c9",
  "range_length": 1,
  "error": "error verifying leader signature for block 83ba52d0f9467266a4a698c9e0a955b72476b9f940de7a30873ee01cd24ad2c9: error retrieving voter Identity at view 957352: by-view query for unknown epoch",
  "time": "2024-09-13T15:44:41.779121803Z",
  "message": "unable to validate proposal with view from unknown epoch"
}

This happens for blocks from range 86932614 to 86932724.