Logs output the status of the main stellar-core node, but not the other stellar-cores that are run by horizon and rpc. This leads to some false feedback where maybe the main node catches up by horizon is lagging behind and there's no visibility of that. Also this leads to no feedback when running only rpc or horizon since no status of their stellar-cores are outputted.
Example
These logs show when a stellar-core node and a soroban-rpc (with stellar-core as well) are running together. You can imagine if only the rpc was running, then the stellar-core(rpc) logs would be the only ones showing. This is comparable with today where you get no stellar-core logs when only the rpc is running.
soroban-rpc: started
stellar-core(rpc): Joining SCP; Catching up to ledger 833534: Downloading state file history/00/0c/b7/history-000cb7ff.json for ledger 833535
stellar-core(node): Catching up; Waiting for trigger ledger: 833584/833601, ETA: 85s
stellar-core(rpc): Joining SCP; Catching up to ledger 833534: downloading and verifying buckets: 0/20 (0%)
stellar-core(node): Catching up; Catching up to ledger 833535: downloading and verifying buckets: 0/20 (0%)
stellar-core(rpc): Joining SCP; Catching up to ledger 833534: downloading and verifying buckets: 11/20 (55%)
stellar-core(node): Catching up; Catching up to ledger 833535: downloading and verifying buckets: 13/20 (65%)
stellar-core(rpc): Joining SCP; Catching up to ledger 833534: downloading and verifying buckets: 16/20 (80%)
stellar-core(node): Catching up; Catching up to ledger 833535: downloading and verifying buckets: 16/20 (80%)
stellar-core(rpc): Joining SCP; Catching up to ledger 833534: downloading and verifying buckets: 17/20 (85%)
stellar-core(node): Catching up; Catching up to ledger 833535: downloading and verifying buckets: 17/20 (85%)
stellar-core(rpc): Joining SCP; Catching up to ledger 833534: downloading and verifying buckets: 19/20 (95%)
stellar-core(node): Catching up; Catching up to ledger 833535: downloading and verifying buckets: 19/20 (95%)
stellar-core(rpc): Joining SCP; Catching up to ledger 833534: Waiting: apply-buckets : 0/1 children completed
stellar-core(node): Catching up; Catching up to ledger 833535: Applying buckets 8%. Currently on level 9
stellar-core(rpc): Joining SCP; Catching up to ledger 833534: Download & apply checkpoints: num checkpoints left to apply:1 (0% done)
stellar-core(node): Catching up; Catching up to ledger 833535: Applying buckets 19%. Currently on level 9
stellar-core(rpc): Connected
stellar-core(node): Catching up; Catching up to ledger 833535: Applying buckets 31%. Currently on level 9
stellar-core(node): Catching up; Catching up to ledger 833535: Applying buckets 43%. Currently on level 9
stellar-core(node): Catching up; Catching up to ledger 833535: Applying buckets 54%. Currently on level 8
stellar-core(node): Catching up; Catching up to ledger 833535: Applying buckets 66%. Currently on level 8
stellar-core(node): Catching up; Catching up to ledger 833535: Applying buckets 74%. Currently on level 8
stellar-core(node): Catching up; Catching up to ledger 833535: Applying buckets 84%. Currently on level 7
stellar-core(node): Catching up; Catching up to ledger 833535: Applying buckets 92%. Currently on level 7
stellar-core(node): Catching up; Catching up to ledger 833535: Download & apply checkpoints: num checkpoints left to apply:2 (0% done)
stellar-core(node): Catching up; Catching up to ledger 833535: Download & apply checkpoints: num checkpoints left to apply:1 (50% done)
stellar-core(node): Catching up; Catching up to ledger 833535: Applying buffered ledgers: apply ledger 833536
stellar-core(node): Catching up; Waiting for trigger ledger: 833589/833601, ETA: 60s
stellar-core(node): Catching up; Waiting for trigger ledger: 833590/833601, ETA: 55s
stellar-core(node): Catching up; Waiting for trigger ledger: 833591/833601, ETA: 50s
stellar-core(node): Catching up; Waiting for trigger ledger: 833592/833601, ETA: 45s
stellar-core(node): Catching up; Waiting for trigger ledger: 833593/833601, ETA: 40s
stellar-core(rpc): Catching up; Waiting for trigger ledger: 833536/833537, ETA: 5s
stellar-core(node): Catching up; Waiting for trigger ledger: 833594/833601, ETA: 35s
soroban-rpc: waiting for ready state, 1 minutes...
stellar-core(node): Catching up; Waiting for trigger ledger: 833595/833601, ETA: 30s
stellar-core(node): Catching up; Waiting for trigger ledger: 833596/833601, ETA: 25s
stellar-core(node): Catching up; Waiting for trigger ledger: 833597/833601, ETA: 20s
stellar-core(node): Catching up; Waiting for trigger ledger: 833598/833601, ETA: 15s
stellar-core(node): Catching up; Waiting for trigger ledger: 833599/833601, ETA: 10s
stellar-core(node): Catching up; Waiting for trigger ledger: 833600/833601, ETA: 5s
stellar-core(rpc): Catching up; Catching up to ledger 833535: Downloading state file history/00/0c/b7/history-000cb7ff.json for ledger 833535
stellar-core(rpc): Catching up; Catching up to ledger 833535: downloading ledger files 1/1 (100%)
stellar-core(rpc): Catching up; Catching up to ledger 833535: Applying buffered ledgers: apply ledger 833536
stellar-core(rpc): Catching up; Catching up to ledger 833599: Downloading state file history/00/0c/b8/history-000cb83f.json for ledger 833599
stellar-core(node): Catching up; Catching up to ledger 833599: Downloading state file history/00/0c/b8/history-000cb83f.json for ledger 833599
stellar-core(node): Catching up; Catching up to ledger 833599: downloading ledger files 1/1 (100%)
stellar-core(rpc): Catching up; Catching up to ledger 833599: downloading ledger files 1/1 (100%)
stellar-core(rpc): Catching up; Catching up to ledger 833599: Download & apply checkpoints: num checkpoints left to apply:1 (0% done)
stellar-core(node): Catching up; Catching up to ledger 833599: Applying buffered ledgers: apply ledger 833601
stellar-core(rpc): Catching up; Catching up to ledger 833599: Applying buffered ledgers: apply ledger 833601
stellar-core(node): Synced!
stellar-core(rpc): Synced!
soroban-rpc: up and ready
What
Display logs for all stellar-cores running.
Why
Logs output the status of the main stellar-core node, but not the other stellar-cores that are run by horizon and rpc. This leads to some false feedback where maybe the main node catches up by horizon is lagging behind and there's no visibility of that. Also this leads to no feedback when running only rpc or horizon since no status of their stellar-cores are outputted.
Example
These logs show when a stellar-core node and a soroban-rpc (with stellar-core as well) are running together. You can imagine if only the rpc was running, then the
stellar-core(rpc)logs would be the only ones showing. This is comparable with today where you get no stellar-core logs when only the rpc is running.Close #584