stellar-core restarts automatically

[shenhao217]

Hello, stellar-core automatically restarts more frequently after updating to the latest version. I'm not quite sure if this is normal. It will cause the node to be in a state of catching up all the time. Version Information： "horizon_version": "2.15.1-e29c7803d487c3f5b44a74773e6277fee16482cf", "core_version": "stellar-core 18.3.0 (2f9ce11b2e7eba7d7d38b123ee6da9e0144249f8)",

error log:

ime="2022-03-10T15:21:19.695+08:00" level=error msg="failed to load the stellar-core info" err="http request errored: Get \"http://localhost:11626/info\": read tcp 127.0.0.1:36120->127.0.0.1:11626: read: connection reset by peer" pid=24897 stack="[main.go:43 client.go:67 app.go:223 app.go:435 asm_amd64.s:1581]" time="2022-03-10T15:21:19.696+08:00" level=warning msg="could not load stellar-core info: http request errored: Get \"http://localhost:11626/info\": read tcp 127.0.0.1:36118->127.0.0.1:11626: read: connection reset by peer" pid=24897 time="2022-03-10T15:21:19.759+08:00" level=error msg="failed to load the stellar-core info" err="http request errored: Get \"http://localhost:11626/info\": dial tcp [::1]:11626: connect: connection refused" pid=24897 stack="[main.go:43 client.go:67 app.go:223 app.go:435 asm_amd64.s:1581]" time="2022-03-10T15:21:19.764+08:00" level=warning msg="could not load stellar-core info: http request errored: Get \"http://localhost:11626/info\": dial tcp [::1]:11626: connect: connection refused" pid=24897 time="2022-03-10T15:21:19.857+08:00" level=error msg="Error in ingestion state machine" current_state="resume(latestSuccessfullyProcessedLedger=39963971)" error="error getting ledger blocking: error reading frame length: unmarshalling XDR frame header: xdr:DecodeUint: EOF while decoding 4 bytes - read: '[]'" next_state=start pid=24897 service=ingest time="2022-03-10T15:21:20.378+08:00" level=error msg="failed to load the stellar-core info" err="http request errored: Get \"http://localhost:11626/info\": dial tcp [::1]:11626: connect: connection refused" pid=24897 stack="[main.go:43 client.go:67 app.go:223 app.go:435 asm_amd64.s:1581]" time="2022-03-10T15:21:20.378+08:00" level=warning msg="could not load stellar-core info: http request errored: Get \"http://localhost:11626/info\": dial tcp [::1]:11626: connect: connection refused" pid=24897 time="2022-03-10T15:21:21.378+08:00" level=error msg="failed to load the stellar-core info" err="http request errored: Get \"http://localhost:11626/info\": dial tcp [::1]:11626: connect: connection refused" pid=24897 stack="[main.go:43 client.go:67 app.go:223 app.go:435 asm_amd64.s:1581]" time="2022-03-10T15:21:21.378+08:00" level=warning msg="could not load stellar-core info: http request errored: Get \"http://localhost:11626/info\": dial tcp [::1]:11626: connect: connection refused" pid=24897 time="2022-03-10T15:21:22.378+08:00" level=error msg="failed to load the stellar-core info" err="http request errored: Get \"http://localhost:11626/info\": dial tcp [::1]:11626: connect: connection refused" pid=24897 stack="[main.go:43 client.go:67 app.go:223 app.go:435 asm_amd64.s:1581]" time="2022-03-10T15:21:22.378+08:00" level=warning msg="could not load stellar-core info: http request errored: Get \"http://localhost:11626/info\": dial tcp [::1]:11626: connect: connection refused" pid=24897 time="2022-03-10T15:21:23.239+08:00" level=info msg="Submitter result" duration=0.000621642 err="tx failed: AAAAAAAAASz////0AAAAAA==" pid=24897 req=stellar/XBU6gesj6f-6517991 service=txsub.submitter time="2022-03-10T15:21:26.003+08:00" level=error msg="History: Could not download file: archive Blockdaemon_Validator_3 maybe missing file history/02/61/cd/history-0261cd7f.json" pid=24897 service=ingest subservice=stellar-core time="2022-03-10T15:21:26.003+08:00" level=error msg="History: Missing HAS for ledger 39964031: maybe stale archive Blockdaemon_Validator_3" pid=24897 service=ingest subservice=stellar-core time="2022-03-10T15:21:28.106+08:00" level=error msg="History: Could not download file: archive keybase_io maybe missing file history/02/61/cd/history-0261cd7f.json" pid=24897 service=ingest subservice=stellar-core time="2022-03-10T15:21:28.106+08:00" level=error msg="History: Missing HAS for ledger 39964031: maybe stale archive keybase_io" pid=24897 service=ingest subservice=stellar-core time="2022-03-10T15:21:52.749+08:00" level=error msg="History: Could not download file: archive satoshipay_singapore maybe missing file .well-known/stellar-history.json" pid=24897 service=ingest subservice=stellar-core time="2022-03-10T15:21:52.749+08:00" level=error msg="History: Could not download .well-known/stellar-history.json file: corrupt archive satoshipay_singapore" pid=24897 service=ingest subservice=stellar-core time="2022-03-10T15:21:52.749+08:00" level=error msg="History: Failed to obtain archive information: satoshipay_singapore" pid=24897 service=ingest subservice=stellar-core time="2022-03-10T15:23:39.941+08:00" level=error msg="History: Could not download file: archive sdf_1 maybe missing file history/02/61/cd/history-0261cd7f.json" pid=24897 service=ingest subservice=stellar-core time="2022-03-10T15:23:39.941+08:00" level=error msg="History: Missing HAS for ledger 39964031: maybe stale archive sdf_1" pid=24897 service=ingest subservice=stellar-core time="2022-03-10T15:23:44.678+08:00" level=error msg="History: Could not download file: archive satoshipay_frankfurt maybe missing file history/02/61/cd/history-0261cd7f.json" pid=24897 service=ingest subservice=stellar-core time="2022-03-10T15:23:44.678+08:00" level=error msg="History: Missing HAS for ledger 39964031: maybe stale archive satoshipay_frankfurt" pid=24897 service=ingest subservice=stellar-core time="2022-03-10T15:23:53.392+08:00" level=error msg="History: Could not download file: archive Blockdaemon_Validator_1 maybe missing file history/02/61/cd/history-0261cd7f.json" pid=24897 service=ingest subservice=stellar-core time="2022-03-10T15:23:53.392+08:00" level=error msg="History: Missing HAS for ledger 39964031: maybe stale archive Blockdaemon_Validator_1" pid=24897 service=ingest subservice=stellar-core time="2022-03-10T15:24:06.284+08:00" level=error msg="History: Could not download file: archive wirexUS maybe missing file history/02/61/cd/history-0261cd7f.json" pid=24897 service=ingest subservice=stellar-core time="2022-03-10T15:24:06.284+08:00" level=error msg="History: Missing HAS for ledger 39964031: maybe stale archive wirexUS" pid=24897 service=ingest subservice=stellar-core time="2022-03-10T15:26:42.213+08:00" level=error msg="History: Could not download file: archive sdf_3 maybe missing file history/02/61/cd/history-0261cd7f.json" pid=24897 service=ingest subservice=stellar-core time="2022-03-10T15:26:42.213+08:00" level=error msg="History: Missing HAS for ledger 39964031: maybe stale archive sdf_3" pid=24897 service=ingest subservice=stellar-core time="2022-03-10T15:29:50.629+08:00" level=error msg="History: Could not download file: archive sdf_3 maybe missing file history/02/61/cd/history-0261cd7f.json" pid=24897 service=ingest subservice=stellar-core time="2022-03-10T15:29:50.629+08:00" level=error msg="History: Missing HAS for ledger 39964031: maybe stale archive sdf_3" pid=24897 service=ingest subservice=stellar-core time="2022-03-10T15:33:25.669+08:00" level=error msg="History: Could not download file: archive sdf_3 maybe missing file history/02/61/cd/history-0261cd3f.json" pid=24897 service=ingest subservice=stellar-core time="2022-03-10T15:33:25.669+08:00" level=error msg="History: Missing HAS for ledger 39963967: maybe stale archive sdf_3" pid=24897 service=ingest subservice=stella

[MonsieurNicolas]

Thanks for the report

Can you provide:

• information as per the bug template on things like environment information (containerized, OS, CPU, memory and disk information, etc) https://github.com/stellar/stellar-core/blob/master/.github/ISSUE_TEMPLATE/bug-report.md#issue-description
• logs right before and after the restart (1 minute before and after would be perfect)
• any system logs that could be relevant here (syslog/postgresql)

[shenhao217]

Thanks for the report

Can you provide:

• information as per the bug template on things like environment information (containerized, OS, CPU, memory and disk information, etc) https://github.com/stellar/stellar-core/blob/master/.github/ISSUE_TEMPLATE/bug-report.md#issue-description
• logs right before and after the restart (1 minute before and after would be perfect)
• any system logs that could be relevant here (syslog/postgresql)

Hello, My machine version: Ubuntu 18.04 server 64bit Machine configuration: 8vCPUs | 16 GiB Here is the horizon configuration information: export STELLAR_CORE_BINARY_PATH="/usr/bin/stellar-core" export CAPTIVE_CORE_STORAGE_PATH="./" export CAPTIVE_CORE_CONFIG_APPEND_PATH="/usr/local/product/horizon/horizon-v2.15.1/captive-core-pubnet.cfg" export CAPTIVE_CORE_PEER_PORT=11625 export ENABLE_CAPTIVE_CORE_INGESTION=true export DATABASE_URL="postgresql://root:@horizon" export PORT=8000 export STELLAR_CORE_URL="http://localhost:11626" export LOG_LEVEL="info" export INGEST=true export PER_HOUR_RATE_LIMIT="180000" export HORIZON_DB_MAX_OPEN_CONNECTIONS="200" export HORIZON_DB_MAX_IDLE_CONNECTIONS="50" export MAX_DB_CONNECTIONS="300" export NETWORK_PASSPHRASE="Public Global Stellar Network ; September 2015" export HISTORY_ARCHIVE_URLS="https://stellar-full-history3.bdnodes.net/" export ADMIN_PORT=6060

Here is the one minute log error log where the error occurred: time="2022-03-13T14:43:03.016+08:00" level=info msg="Overlay: Dropping peer 34.243.114.22:11625, reason error reading message header: End of file" pid=30906 service=ingest subservice=stellar-core time="2022-03-13T14:43:50.407+08:00" level=warning msg="could not load stellar-core info: http request errored: Get \"http://localhost:11626/info\": read tcp 127.0.0.1:36550->127.0.0.1:11626: read: connection reset by peer" pid=30906 time="2022-03-13T14:43:50.408+08:00" level=error msg="failed to load the stellar-core info" err="http request errored: Get \"http://localhost:11626/info\": read tcp 127.0.0.1:36548->127.0.0.1:11626: read: connection reset by peer" pid=30906 stack="[main.go:43 client.go:67 app.go:223 app.go:435 asm_amd64.s:1581]" time="2022-03-13T14:43:50.411+08:00" level=warning msg="could not load stellar-core info: http request errored: Get \"http://localhost:11626/info\": dial tcp [::1]:11626: connect: connection refused" pid=30906 time="2022-03-13T14:43:50.411+08:00" level=error msg="failed to load the stellar-core info" err="http request errored: Get \"http://localhost:11626/info\": dial tcp [::1]:11626: connect: connection refused" pid=30906 stack="[main.go:43 client.go:67 app.go:223 app.go:435 asm_amd64.s:1581]" time="2022-03-13T14:43:50.458+08:00" level=error msg="Error in ingestion state machine" current_state="resume(latestSuccessfullyProcessedLedger=40006675)" error="error getting ledger blocking: error reading frame length: unmarshalling XDR frame header: xdr:DecodeUint: EOF while decoding 4 bytes - read: '[]'" next_state=start pid=30906 service=ingest time="2022-03-13T14:43:51.291+08:00" level=error msg="failed to load the stellar-core info" err="http request errored: Get \"http://localhost:11626/info\": dial tcp [::1]:11626: connect: connection refused" pid=30906 stack="[main.go:43 client.go:67 app.go:223 app.go:435 asm_amd64.s:1581]" time="2022-03-13T14:43:51.291+08:00" level=warning msg="could not load stellar-core info: http request errored: Get \"http://localhost:11626/info\": dial tcp [::1]:11626: connect: connection refused" pid=30906 time="2022-03-13T14:43:52.292+08:00" level=error msg="failed to load the stellar-core info" err="http request errored: Get \"http://localhost:11626/info\": dial tcp [::1]:11626: connect: connection refused" pid=30906 stack="[main.go:43 client.go:67 app.go:223 app.go:435 asm_amd64.s:1581]" time="2022-03-13T14:43:52.292+08:00" level=warning msg="could not load stellar-core info: http request errored: Get \"http://localhost:11626/info\": dial tcp [::1]:11626: connect: connection refused" pid=30906 time="2022-03-13T14:43:54.676+08:00" level=error msg="History: Could not download file: archive Blockdaemon_Validator_3 maybe missing file history/02/62/74/history-0262743f.json" pid=30906 service=ingest subservice=stellar-core time="2022-03-13T14:43:54.676+08:00" level=error msg="History: Missing HAS for ledger 40006719: maybe stale archive Blockdaemon_Validator_3" pid=30906 service=ingest subservice=stellar-core time="2022-03-13T14:43:56.473+08:00" level=error msg="History: Could not download file: archive keybase_io maybe missing file history/02/62/74/history-0262743f.json" pid=30906 service=ingest subservice=stellar-core time="2022-03-13T14:43:56.473+08:00" level=error msg="History: Missing HAS for ledger 40006719: maybe stale archive keybase_io" pid=30906 service=ingest subservice=stellar-core thanks for your help