search

ceramicnetwork / js-ceramic

Typescript implementation of the Ceramic protocol
http://ceramic.network
Other
414 stars 127 forks source link

feat: wait for metrics model to be available locally #3265

Closed stbrody closed 1 month ago

stbrody commented 1 month ago

Builds on https://github.com/ceramicnetwork/js-ceramic/pull/3260/

When starting up a clean js-ceramic node for the first time with node metrics publishing enabled, the node needs to wait for the NodeMetrics model to be synced via recon and made available locally before it tries to index it. Example out from a clean node as it waits for this to happen below:

[2024-07-19T18:56:30.467Z] IMPORTANT: Connecting to IPFS node with version "ceramic-one/0.28.1" available as /ip4/127.0.0.1/udp/4101/quic-v1/p2p/12D3KooWL9zNBFnNEgEPRCtrzFXX7iyN8vnGWJGX7z6eaFkmdy5e, /ip4/10.0.0.188/udp/4101/quic-v1/p2p/12D3KooWL9zNBFnNEgEPRCtrzFXX7iyN8vnGWJGX7z6eaFkmdy5e, /ip4/127.0.0.1/tcp/4101/p2p/12D3KooWL9zNBFnNEgEPRCtrzFXX7iyN8vnGWJGX7z6eaFkmdy5e, /ip4/10.0.0.188/tcp/4101/p2p/12D3KooWL9zNBFnNEgEPRCtrzFXX7iyN8vnGWJGX7z6eaFkmdy5e
[2024-07-19T18:56:30.469Z] IMPORTANT: Initializing SQLite connection
[2024-07-19T18:56:30.509Z] IMPORTANT: Node DID set to 'did:key:z6MkhXxQCx33gofNSNSRuipu5xr8cNnW2b4aoXebyfG6FXwe'. This DID will be used to authenticate to the anchor service
[2024-07-19T18:56:30.509Z] IMPORTANT: Connecting to ceramic network 'testnet-clay' using ceramic-one with Recon for data synchronization.
[2024-07-19T18:56:30.519Z] IMPORTANT: Creating ComposeDB config table: ceramic_models
[2024-07-19T18:56:30.519Z] IMPORTANT: Creating ComposeDB config table: ceramic_config
[2024-07-19T18:56:30.519Z] IMPORTANT: Creating ComposeDB config table: ceramic_model_implements
[2024-07-19T18:56:30.536Z] IMPORTANT: Connecting to bootstrap peers for network testnet-clay
[2024-07-19T18:56:30.543Z] WARNING: Can not connect to /dns4/bootstrap-tnet-rust-ceramic-1.3box.io/tcp/4101/p2p/12D3KooWMqCFj5bnwuNi6D6KLhYiK4C8Eh9xSUKv2E6Jozs4nWEE
[2024-07-19T18:56:30.543Z] WARNING: HTTPError: An internal error occurred
[2024-07-19T18:56:30.545Z] WARNING: Can not connect to /dns4/bootstrap-tnet-rust-ceramic-2.3box.io/tcp/4101/p2p/12D3KooWPFGbRHWfDaWt5MFFeqAHBBq3v5BqeJ4X7pmn2V1t6uNs
[2024-07-19T18:56:30.545Z] WARNING: HTTPError: An internal error occurred
[2024-07-19T18:56:30.990Z] IMPORTANT: Connected to anchor service 'https://cas-clay.3boxlabs.com' with supported anchor chains ['eip155:100']
[2024-07-19T18:56:30.992Z] WARNING: No pinned streams detected. This is expected if this is the first time this node has been run, but may indicate a problem with the node's persistence setup if it should have pinned streams
[2024-07-19T18:56:30.997Z] IMPORTANT: Waiting for Model kjzl6hvfrbw6cb9pd0bl7zmm28h3qszh56ccpn50vsmrl7clroy4fvln00z7q6q used to publish NodeMetrics to be available locally
[2024-07-19T18:56:34.134Z] ERROR: Error loading Model kjzl6hvfrbw6cb9pd0bl7zmm28h3qszh56ccpn50vsmrl7clroy4fvln00z7q6q used to publish NodeMetrics: HTTPError: block was not found locally (offline)
[2024-07-19T18:56:37.344Z] ERROR: Error loading Model kjzl6hvfrbw6cb9pd0bl7zmm28h3qszh56ccpn50vsmrl7clroy4fvln00z7q6q used to publish NodeMetrics: HTTPError: block was not found locally (offline)
[2024-07-19T18:56:43.757Z] ERROR: Error loading Model kjzl6hvfrbw6cb9pd0bl7zmm28h3qszh56ccpn50vsmrl7clroy4fvln00z7q6q used to publish NodeMetrics: HTTPError: block was not found locally (offline)
[2024-07-19T18:56:56.560Z] ERROR: Error loading Model kjzl6hvfrbw6cb9pd0bl7zmm28h3qszh56ccpn50vsmrl7clroy4fvln00z7q6q used to publish NodeMetrics: HTTPError: block was not found locally (offline)
[2024-07-19T18:57:09.362Z] IMPORTANT: Model kjzl6hvfrbw6cb9pd0bl7zmm28h3qszh56ccpn50vsmrl7clroy4fvln00z7q6q used to publish NodeMetrics loaded successfully
[2024-07-19T18:57:09.362Z] IMPORTANT: Starting indexing for Model kjzl6hvfrbw6cb9pd0bl7zmm28h3qszh56ccpn50vsmrl7clroy4fvln00z7q6q
[2024-07-19T18:57:09.408Z] IMPORTANT: Creating ComposeDB Indexing table for model: kjzl6hvfrbw6cb9pd0bl7zmm28h3qszh56ccpn50vsmrl7clroy4fvln00z7q6q
[2024-07-19T18:57:09.419Z] IMPORTANT: Publishing Node Metrics publicly to the Ceramic Network.  To learn more, including how to disable publishing, please see the NODE_METRICS.md file for your branch, e.g. https://github.com/ceramicnetwork/js-ceramic/blob/develop/docs-dev/NODE_METRICS.md
[2024-07-19T18:57:09.422Z] IMPORTANT: Ceramic API running on 0.0.0.0:7007'
linear[bot] commented 1 month ago

AES-232 Subscribe to NodeMetrics model when nodeMetrics enabled

stbrody commented 1 month ago

Okay, new version up that waits for up to 10 seconds for the model to become available and then proceeds with startup anyway if the model hasn't been loaded in time. New output if the model is unable to be synced:

[2024-07-23T21:31:30.142Z] IMPORTANT: Connected to anchor service 'https://cas-clay.3boxlabs.com' with supported anchor chains ['eip155:100']
[2024-07-23T21:31:30.143Z] WARNING: No pinned streams detected. This is expected if this is the first time this node has been run, but may indicate a problem with the node's persistence setup if it should have pinned streams
[2024-07-23T21:31:30.151Z] IMPORTANT: Waiting for Model kjzl6hvfrbw6cb9pd0bl7zmm28h3qszh56ccpn50vsmrl7clroy4fvln00z7q6q used to publish Node Metrics to be available locally
[2024-07-23T21:31:30.688Z] ERROR: Error loading Model kjzl6hvfrbw6cb9pd0bl7zmm28h3qszh56ccpn50vsmrl7clroy4fvln00z7q6q used to publish Node Metrics: HTTPError: block was not found locally (offline)
[2024-07-23T21:31:31.229Z] ERROR: Error loading Model kjzl6hvfrbw6cb9pd0bl7zmm28h3qszh56ccpn50vsmrl7clroy4fvln00z7q6q used to publish Node Metrics: HTTPError: block was not found locally (offline)
[2024-07-23T21:31:31.764Z] ERROR: Error loading Model kjzl6hvfrbw6cb9pd0bl7zmm28h3qszh56ccpn50vsmrl7clroy4fvln00z7q6q used to publish Node Metrics: HTTPError: block was not found locally (offline)
[2024-07-23T21:31:32.291Z] ERROR: Error loading Model kjzl6hvfrbw6cb9pd0bl7zmm28h3qszh56ccpn50vsmrl7clroy4fvln00z7q6q used to publish Node Metrics: HTTPError: block was not found locally (offline)
[2024-07-23T21:31:32.821Z] ERROR: Error loading Model kjzl6hvfrbw6cb9pd0bl7zmm28h3qszh56ccpn50vsmrl7clroy4fvln00z7q6q used to publish Node Metrics: HTTPError: block was not found locally (offline)
[2024-07-23T21:31:33.352Z] ERROR: Error loading Model kjzl6hvfrbw6cb9pd0bl7zmm28h3qszh56ccpn50vsmrl7clroy4fvln00z7q6q used to publish Node Metrics: HTTPError: block was not found locally (offline)
[2024-07-23T21:31:33.884Z] ERROR: Error loading Model kjzl6hvfrbw6cb9pd0bl7zmm28h3qszh56ccpn50vsmrl7clroy4fvln00z7q6q used to publish Node Metrics: HTTPError: block was not found locally (offline)
[2024-07-23T21:31:34.413Z] ERROR: Error loading Model kjzl6hvfrbw6cb9pd0bl7zmm28h3qszh56ccpn50vsmrl7clroy4fvln00z7q6q used to publish Node Metrics: HTTPError: block was not found locally (offline)
[2024-07-23T21:31:34.946Z] ERROR: Error loading Model kjzl6hvfrbw6cb9pd0bl7zmm28h3qszh56ccpn50vsmrl7clroy4fvln00z7q6q used to publish Node Metrics: HTTPError: block was not found locally (offline)
[2024-07-23T21:31:35.471Z] WARNING: Could not load Model kjzl6hvfrbw6cb9pd0bl7zmm28h3qszh56ccpn50vsmrl7clroy4fvln00z7q6q used to publish Node Metrics, disabling metrics publishing
[2024-07-23T21:31:35.480Z] IMPORTANT: Ceramic API running on 0.0.0.0:7007'
stbrody commented 1 month ago

Increased the timeout to 15 seconds because in my local testing the node was taking more than 10 seconds to sync the model fairly regularly.

Interestingly, how long it takes the node to sync the model on an brand new clean node seems to vary quite wildly. I saw it sync within milliseconds, the data already being present by the time js-ceramic tried to load it the first time, and I also saw it take as long as almost 20 seconds at least once.

FYI @dav1do @nathanielc

stbrody commented 1 month ago

Closing in favor of https://github.com/ceramicnetwork/js-ceramic/pull/3268