search

stellar / go

Stellar's public monorepo of go code
https://stellar.org/developers
Apache License 2.0
1.28k stars 497 forks source link

ingest: captive-core: output offline-info logs #5372

Open 2opremio opened 5 days ago

2opremio commented 5 days ago

The captive core backend invokes stellar-core offline-info:

https://github.com/stellar/go/blob/b589529f102f0b92c106e1015a80531f52623036/ingest/ledgerbackend/run_from.go#L42-L58

However:

  1. It doesn't print it's logs (stderr) as it executes. This is confusing when offline-info requires indexing (which can take more than 4 minutes). The user doesn't know what's going on during that time.
  2. It doesn't print its output in case of error.
2opremio commented 5 days ago

Here's the output (stderr) of a slow execution of stellar-core offfline-info:

2024-07-03T21:44:26.890 [default INFO] Config from /Users/fons/soroban-rpc/pubnet/stellar-core/captive-core/stellar-core.conf
2024-07-03T21:44:26.891 [default INFO] Generated QUORUM_SET: {
   "t" : 5,
   "v" : [
      {
         "t" : 2,
         "v" : [ "blockdaemon_1", "blockdaemon_2", "Blockdaemon Validator 3" ]
      },
      {
         "t" : 2,
         "v" : [ "sdf_live_3", "sdf_live_1", "sdf_live_2" ]
      },
      {
         "t" : 2,
         "v" : [ "coinqvest_fi", "coinqvest_hk", "coinqvest_de" ]
      },
      {
         "t" : 2,
         "v" : [ "satoshipay_us", "satoshipay_sg", "satoshipay_de" ]
      },
      {
         "t" : 2,
         "v" : [ "FT_SCV_1", "FT_SCV_3", "FT_SCV_2" ]
      },
      {
         "t" : 3,
         "v" : [
            "lobstr_5_au",
            "lobstr_4_asia",
            "lobstr_2_eu",
            "lobstr_1_eu",
            "lobstr_3_north_america"
         ]
      },
      {
         "t" : 2,
         "v" : [ "hercules", "lyra", "bootes" ]
      }
   ]
}
2024-07-03T21:44:26.891 [default INFO] Assigning calculated value of 2 to FAILURE_SAFETY
2024-07-03T21:44:26.891 GBOYC [Database INFO] Connecting to: sqlite3:///Users/fons/soroban-rpc/pubnet/stellar-captive-core.db
2024-07-03T21:44:26.893 GBOYC [SCP INFO] LocalNode::LocalNode@GBOYC qSet: 5c464e
2024-07-03T21:44:26.893 GBOYC [Database INFO] DB schema is in current version
2024-07-03T21:44:26.893 GBOYC [default INFO] Dropping accounts
2024-07-03T21:44:26.893 GBOYC [default INFO] Dropping trustlines
2024-07-03T21:44:26.893 GBOYC [default INFO] Dropping accountdata
2024-07-03T21:44:26.893 GBOYC [default INFO] Dropping claimablebalances
2024-07-03T21:44:26.893 GBOYC [default INFO] Dropping liquiditypools
2024-07-03T21:44:26.893 GBOYC [default INFO] Dropping contractdata
2024-07-03T21:44:26.893 GBOYC [default INFO] Dropping contractcode
2024-07-03T21:44:26.893 GBOYC [default INFO] Dropping configsettings
2024-07-03T21:44:26.893 GBOYC [default INFO] Dropping ttl
2024-07-03T21:44:26.894 GBOYC [Bucket INFO] BucketListDB enabled: pageSizeExponent: 12 indexCutOff: 20MB, persist indexes: true
2024-07-03T21:44:26.894 GBOYC [default INFO] Connection effective settings:
2024-07-03T21:44:26.894 GBOYC [default INFO] TARGET_PEER_CONNECTIONS: 8
2024-07-03T21:44:26.894 GBOYC [default INFO] MAX_ADDITIONAL_PEER_CONNECTIONS: 64
2024-07-03T21:44:26.894 GBOYC [default INFO] MAX_PENDING_CONNECTIONS: 500
2024-07-03T21:44:26.894 GBOYC [default INFO] MAX_OUTBOUND_PENDING_CONNECTIONS: 56
2024-07-03T21:44:26.894 GBOYC [default INFO] MAX_INBOUND_PENDING_CONNECTIONS: 444
2024-07-03T21:44:26.894 GBOYC [Ledger INFO] Last closed ledger (LCL) hash is a3bc9714396ab701a3d77ceba1148dc0afb32acd773ba96dc33ad6e211ba0515
2024-07-03T21:44:26.894 GBOYC [Ledger INFO] Loaded LCL header from database: [seq=52405532, hash=a3bc97]
2024-07-03T21:44:26.895 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing individual keys in bucket buckets/bucket-6f9ed4c47378613f22274731c0ec9aa5b841a7bf9543c3d0549cd1ab661e1cd6.xdr
2024-07-03T21:44:26.895 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing individual keys in bucket buckets/bucket-15345f69959746b01b69ae7579216154bb31a4545fd298d466ee3731f3243ddf.xdr
2024-07-03T21:44:26.895 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing individual keys in bucket buckets/bucket-5a9f52e70912edaa627c6bd6eb57a3c0ac8ca3c83e06c756f45ea02b314c7156.xdr
2024-07-03T21:44:26.895 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing individual keys in bucket buckets/bucket-990f8c6310eaafbc106853fc8d705a19106d25d205cf07269e4005528dd360cd.xdr
2024-07-03T21:44:26.895 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing individual keys in bucket buckets/bucket-532f0fd12c80cbb3fd2b1817c5e3b8b8ac8f5863aaa4e818df54e2558c2d3e13.xdr
2024-07-03T21:44:26.895 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing individual keys in bucket buckets/bucket-ea9cfec7a712e51840cdc231df681f4ffe3e9cbfc5d60709368770aed75ef897.xdr
2024-07-03T21:44:26.895 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing individual keys in bucket buckets/bucket-6db69edd11e1909aa621e6ef9623c5f6feb0d66454228687fb3014170fa71785.xdr
2024-07-03T21:44:26.895 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing individual keys in bucket buckets/bucket-df63858bd1d189ee74074c4da760aeed01d6e2a7c22b0ac39b5bccb9ae9ddd05.xdr
2024-07-03T21:44:26.895 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing individual keys in bucket buckets/bucket-59935832039af857e7ace8d213084f2080bf46831fe498ec83cc179d6c47bc03.xdr
2024-07-03T21:44:26.895 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing individual keys in bucket buckets/bucket-3c85e4da61503a736c346d6d0f9250c50fd852a98c5902e76af785f5bdc6de90.xdr
2024-07-03T21:44:26.895 GBOYC [Bucket WARNING] Outdated index file: buckets/bucket-060fdae66a38182b01e7f15fd228c520064742ca9700a41e5b6f42f082d38a29.index
2024-07-03T21:44:26.895 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing individual keys in bucket buckets/bucket-060fdae66a38182b01e7f15fd228c520064742ca9700a41e5b6f42f082d38a29.xdr
2024-07-03T21:44:26.932 GBOYC [Bucket WARNING] Outdated index file: buckets/bucket-184a74744cb72c5ba886be7a05ff25ef00c8db04c506dd9fd24b5fbb3cc09c6e.index
2024-07-03T21:44:26.933 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing key range with page size 4096 in bucket buckets/bucket-184a74744cb72c5ba886be7a05ff25ef00c8db04c506dd9fd24b5fbb3cc09c6e.xdr
2024-07-03T21:44:26.947 GBOYC [Bucket WARNING] Outdated index file: buckets/bucket-3d9236663619946aa241b39e96c3c1f3d104fefd88a11e0f19e9537d6d308273.index
2024-07-03T21:44:26.947 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing key range with page size 4096 in bucket buckets/bucket-3d9236663619946aa241b39e96c3c1f3d104fefd88a11e0f19e9537d6d308273.xdr
2024-07-03T21:44:27.042 GBOYC [Bucket WARNING] Outdated index file: buckets/bucket-a0b6961f0d21e28a5e4cc7f7e490e4bcc2c10fcff22b0a8034dbb977a948f1ab.index
2024-07-03T21:44:27.044 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing key range with page size 4096 in bucket buckets/bucket-a0b6961f0d21e28a5e4cc7f7e490e4bcc2c10fcff22b0a8034dbb977a948f1ab.xdr
2024-07-03T21:44:27.093 GBOYC [Bucket WARNING] Outdated index file: buckets/bucket-7dc332c00a6cc7712f4da3a506b154134a1c36cd76ba52066032dde7a4b933f8.index
2024-07-03T21:44:27.094 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing key range with page size 4096 in bucket buckets/bucket-7dc332c00a6cc7712f4da3a506b154134a1c36cd76ba52066032dde7a4b933f8.xdr
2024-07-03T21:44:27.429 GBOYC [Bucket WARNING] Outdated index file: buckets/bucket-c51d5f7e169f6f153146cb0d8c0de71763d821c5ffcbc0f4cebc27589b6c7566.index
2024-07-03T21:44:27.429 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing key range with page size 4096 in bucket buckets/bucket-c51d5f7e169f6f153146cb0d8c0de71763d821c5ffcbc0f4cebc27589b6c7566.xdr
2024-07-03T21:44:27.435 GBOYC [Bucket WARNING] Outdated index file: buckets/bucket-765225f2605cb000efb89e977c562cc7ece199d9cd44fb61bb0493abbdecab6c.index
2024-07-03T21:44:27.436 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing key range with page size 4096 in bucket buckets/bucket-765225f2605cb000efb89e977c562cc7ece199d9cd44fb61bb0493abbdecab6c.xdr
2024-07-03T21:44:27.452 GBOYC [Bucket WARNING] Outdated index file: buckets/bucket-a6730367fae3b9a76026bb7d351e99e4a98928404eaecb0a8aee5cbb3923bd10.index
2024-07-03T21:44:27.453 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing key range with page size 4096 in bucket buckets/bucket-a6730367fae3b9a76026bb7d351e99e4a98928404eaecb0a8aee5cbb3923bd10.xdr
2024-07-03T21:44:27.750 GBOYC [Bucket WARNING] Outdated index file: buckets/bucket-dcf3aff443c35aefa9433ac37332cdac1a72bbdb72e24e0e7167cb22215ebfc8.index
2024-07-03T21:44:27.751 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing key range with page size 4096 in bucket buckets/bucket-dcf3aff443c35aefa9433ac37332cdac1a72bbdb72e24e0e7167cb22215ebfc8.xdr
2024-07-03T21:44:27.881 GBOYC [Bucket WARNING] Outdated index file: buckets/bucket-b893a6356c3f1ee1561719411b165cce327b2faafc9c51316157b33c2fbe522f.index
2024-07-03T21:44:27.882 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing key range with page size 4096 in bucket buckets/bucket-b893a6356c3f1ee1561719411b165cce327b2faafc9c51316157b33c2fbe522f.xdr
2024-07-03T21:44:29.085 GBOYC [Bucket WARNING] Outdated index file: buckets/bucket-cd402152dccfbcc6cffa58c85c40b6fabe01411df813e611d2e44cf67502cb9d.index
2024-07-03T21:44:29.085 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing key range with page size 4096 in bucket buckets/bucket-cd402152dccfbcc6cffa58c85c40b6fabe01411df813e611d2e44cf67502cb9d.xdr
2024-07-03T21:44:29.878 GBOYC [Bucket WARNING] Outdated index file: buckets/bucket-aa16961ed23f53397b3764112949c0c2c2f6fcb9b08ccf64c8ee9d3010745073.index
2024-07-03T21:44:29.878 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing key range with page size 4096 in bucket buckets/bucket-aa16961ed23f53397b3764112949c0c2c2f6fcb9b08ccf64c8ee9d3010745073.xdr
2024-07-03T21:44:30.508 GBOYC [Bucket WARNING] Outdated index file: buckets/bucket-1c6b47a65e8014d935c8e21131aee3616ad477d2261790f31d85c607205a090a.index
2024-07-03T21:44:30.509 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing key range with page size 4096 in bucket buckets/bucket-1c6b47a65e8014d935c8e21131aee3616ad477d2261790f31d85c607205a090a.xdr
2024-07-03T21:44:31.652 GBOYC [Bucket WARNING] Outdated index file: buckets/bucket-7d678a9925dda72aa91855377a27171472a9e42f22d6bb54b41e45cf3e9ffe88.index
2024-07-03T21:44:31.652 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing key range with page size 4096 in bucket buckets/bucket-7d678a9925dda72aa91855377a27171472a9e42f22d6bb54b41e45cf3e9ffe88.xdr
2024-07-03T21:44:32.489 GBOYC [Bucket WARNING] Outdated index file: buckets/bucket-0a8ce35d923d903da80c5263d644ec426c4e588e0767231899df265c6dc0027e.index
2024-07-03T21:44:32.490 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing key range with page size 4096 in bucket buckets/bucket-0a8ce35d923d903da80c5263d644ec426c4e588e0767231899df265c6dc0027e.xdr
2024-07-03T21:44:40.095 GBOYC [Perf WARNING] 'Indexing bucket' took 13.050000 s
2024-07-03T21:44:40.110 GBOYC [Bucket WARNING] Outdated index file: buckets/bucket-e5ec0f26e472150cb455576c1cf63a66ac76ff7dbe356a0c9d2de7c9a7221621.index
2024-07-03T21:44:40.113 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing key range with page size 4096 in bucket buckets/bucket-e5ec0f26e472150cb455576c1cf63a66ac76ff7dbe356a0c9d2de7c9a7221621.xdr
2024-07-03T21:44:44.065 GBOYC [Perf WARNING] 'Indexing bucket' took 16.636000 s [skipped: 1]
2024-07-03T21:44:44.086 GBOYC [Bucket WARNING] Outdated index file: buckets/bucket-402dba735f087533bc8f574498356ea8b82c56f6d0802b08bbb84e18f452cabc.index
2024-07-03T21:44:44.091 GBOYC [Bucket INFO] BucketIndex::createIndex() indexing key range with page size 4096 in bucket buckets/bucket-402dba735f087533bc8f574498356ea8b82c56f6d0802b08bbb84e18f452cabc.xdr
2024-07-03T21:44:54.684 GBOYC [Perf WARNING] 'Indexing bucket' took 27.231001 s [skipped: 1]
2024-07-03T21:45:16.386 GBOYC [Perf WARNING] 'Indexing bucket' took 48.633999 s
2024-07-03T21:45:27.355 GBOYC [Perf WARNING] 'Indexing bucket' took 59.917999 s
2024-07-03T21:45:51.620 GBOYC [Perf WARNING] 'Indexing bucket' took 82.533997 s
2024-07-03T21:46:16.603 GBOYC [Perf WARNING] 'Indexing bucket' took 108.720001 s
2024-07-03T21:46:21.029 GBOYC [Perf WARNING] 'Indexing bucket' took 111.151001 s
2024-07-03T21:46:26.066 GBOYC [Perf WARNING] 'Indexing bucket' took 115.556999 s
2024-07-03T21:46:40.354 GBOYC [Perf WARNING] 'Indexing bucket' took 127.863998 s
2024-07-03T21:46:42.023 GBOYC [Perf WARNING] 'Indexing bucket' took 130.369995 s
2024-07-03T21:49:31.581 GBOYC [Perf WARNING] 'Indexing bucket' took 291.467987 s
2024-07-03T21:49:36.138 GBOYC [Perf WARNING] 'Indexing bucket' took 292.046997 s
2024-07-03T21:49:36.745 GBOYC [Ledger INFO] Assumed bucket-state for LCL: [seq=52405532, hash=a3bc97]
2024-07-03T21:49:36.745 GBOYC [default INFO] Reporting application info
2024-07-03T21:49:36.745 GBOYC [default INFO] Application destructing
2024-07-03T21:49:36.746 GBOYC [default INFO] Application destroyed
2opremio commented 5 days ago

CC @tamirms