Closed halibobo1205 closed 9 months ago
I use Toolkit for generating or pruning Lite Fullnode data regularly.
Expected behaviour
It can be executed normally.
Actual behaviour
The command sometimes quits abnormally.
Steps to reproduce the behaviour
Backtrace
Start create snapshot. Begin to split the dbs. Begin to merge checkpoint to dataset. Begin to fill 65536 block, genesis block and trans to snapshot. fillBlockAndTrans 99% │███████████████████████████████████████████▉│ 65522/65536 (0:01:34 / 0:00:00) data not found in store, dbName: block-index, key: [0, 0, 0, 0, 3, 64, -96, 32] Usage: <main class> db lite [-h] -ds=<datasetPath> -fn=<fnDataPath> [-o=<operate>] [-t=<type>] Split lite data for java-tron. -ds, --dataset-path=<datasetPath> when operation is `split`,`dataset-path` is the path that store the `snapshot` or `history`,when operation is `split`,`dataset-path` is the `history` data path. -fn, --fn-data-path=<fnDataPath> the database path to be split or merged. -h, --help -o, --operate=<operate> operate: [ split, merge ]. Default: split -t, --type=<type> only used with operate=split: [ snapshot, history ]. Default: snapshot Exit Codes: 0 Successful 1 Internal error: exception occurred,please check toolkit.log
Have you tried to use the official data to generate Lite Fullnode data? Have you tried to stop the node abnormally?
@lurais Because I don't have enough disk space, I regularly use Toolkit for generating or pruning Lite Fullnode data. I stop the node by kill -9
, in most cases there is no problem.
@lurais Because I don't have enough disk space, I regularly use Toolkit for generating or pruning Lite Fullnode data. I stop the node by
kill -9
, in most cases there is no problem.
From the error log, some data may be exist in checkpoint but not found in the other stores(just like block-index), can you provide the config file you used to start the node?
It's also possible that kill -9
is causing the problem?
@forfreeday The error doesn't happen every time.
net {
type = mainnet
# type = testnet
}
# prometheus metrics start
node.metrics = {
prometheus{
enable=true
port="9527"
}
}
# prometheus metrics end
storage {
# Directory for storing persistent data
db.version = 2,
db.engine = "LEVELDB",
db.sync = false,
db.directory = "database",
index.directory = "index",
transHistory.switch = "on",
# You can custom these 14 databases' configs:
# account, account-index, asset-issue, block, block-index,
# block_KDB, peers, properties, recent-block, trans,
# utxo, votes, witness, witness_schedule.
# Otherwise, db configs will remain default and data will be stored in
# the path of "output-directory" or which is set by "-d" ("--output-directory").
// setting can impove leveldb performance .... start
// node: if this will increase process fds,you may be check your ulimit if 'too many open files' error occurs
// see https://github.com/tronprotocol/tips/blob/master/tip-343.md for detail
// if you find block sync has lower performance,you can try this settings
default = {
maxOpenFiles = 100
}
defaultM = {
maxOpenFiles = 500
}
defaultL = {
maxOpenFiles = 1000
}
// setting can impove leveldb performance .... end
# Attention: name is a required field that must be set !!!
properties = []
needToUpdateAsset = true
//dbsettings is needed when using rocksdb as the storage implement (db.version=2 and db.engine="ROCKSDB").
//we'd strongly recommend that do not modify it unless you know every item's meaning clearly.
dbSettings = {
levelNumber = 7
//compactThreads = 32
blocksize = 64 // n * KB
maxBytesForLevelBase = 256 // n * MB
maxBytesForLevelMultiplier = 10
level0FileNumCompactionTrigger = 4
targetFileSizeBase = 256 // n * MB
targetFileSizeMultiplier = 1
}
//backup settings when using rocks db as the storage implement (db.version=2 and db.engine="ROCKSDB").
//if you want to use the backup plugin, please confirm set the db.version=2 and db.engine="ROCKSDB" above.
backup = {
enable = false // indicate whether enable the backup plugin
propPath = "prop.properties" // record which bak directory is valid
bak1path = "bak1/database" // you must set two backup directories to prevent application halt unexpected(e.g. kill -9).
bak2path = "bak2/database"
frequency = 10000 // indicate backup db once every 10000 blocks processed.
}
checkpoint.version = 2
checkpoint.sync = true
# if true, transaction cache initialization will be faster. default false
txCache.initOptimization = true
}
node.discovery = {
enable = true
persist = true
bind.ip = ""
external.ip = null
}
node.backup {
port = 10001
# my priority, each member should use different priority
priority = 8
# peer's ip list, can't contain mine
members = [
# "ip",
# "ip"
]
}
node {
# trust node for solidity node
# trustNode = "ip:port"
trustNode = "127.0.0.1:50051"
# expose extension api to public or not
walletExtensionApi = true
listen.port = 18888
connection.timeout = 2
tcpNettyWorkThreadNum = 0
udpNettyWorkThreadNum = 1
# Number of validate sign thread, default availableProcessors / 2
# validateSignThreadNum = 16
connectFactor = 0.3
activeConnectFactor = 0.1
maxActiveNodes = 30
maxActiveNodesWithSameIp = 2
maxHttpConnectNumber = 50
minParticipationRate = 15
# check the peer data transfer ,disconnect factor
disconnectNumberFactor = 0.4
maxConnectNumberFactor = 0.8
receiveTcpMinDataLength = 2048
isOpenFullTcpDisconnect = true
p2p {
version = 11111 # 11111: mainnet; 20180622: testnet
}
active = [
# Active establish connection in any case
# Sample entries:
# "ip:port",
# "ip:port"
]
passive = [
# Passive accept connection in any case
# Sample entries:
# "ip:port",
# "ip:port"
]
fastForward = [
"100.26.245.209:18888",
"15.188.6.125:18888"
]
http {
fullNodePort = 8090
solidityPort = 8091
PBFTPort = 8092
}
jsonrpc {
# Note: If you turn on jsonrpc and run it for a while and then turn it off, you will not
# be able to get the data from eth_getLogs for that period of time.
httpFullNodeEnable = true
httpFullNodePort = 8545
httpSolidityEnable = true
httpSolidityPort = 8555
httpPBFTEnable = true
httpPBFTPort = 8565
}
rpc {
port = 50051
solidityPort = 50061
PBFTPort = 50071
maxConnectionIdleInMillis = 60000
minEffectiveConnection = 1
}
}
## rate limiter config
rate.limiter = {
# Every api could be set a specific rate limit strategy. Three strategy are supported:GlobalPreemptibleAdapter、IPQPSRateLimiterAdapte、QpsRateLimiterAdapter
# GlobalPreemptibleAdapter: permit is the number of preemptible resource, every client must apply one resourse
# before do the request and release the resource after got the reponse automaticlly. permit should be a Integer.
# QpsRateLimiterAdapter: qps is the average request count in one second supported by the server, it could be a Double or a Integer.
# IPQPSRateLimiterAdapter: similar to the QpsRateLimiterAdapter, qps could be a Double or a Integer.
# If do not set, the "default strategy" is set.The "default startegy" is based on QpsRateLimiterAdapter, the qps is set as 10000.
#
# Sample entries:
#
http = [
{
component = "JsonRpcServlet",
strategy = "QpsRateLimiterAdapter",
paramString = "qps=1"
},
# {
# component = "GetAccountServlet",
# strategy = "IPQPSRateLimiterAdapter",
# paramString = "qps=1"
# },
# {
# component = "ListWitnessesServlet",
# strategy = "QpsRateLimiterAdapter",
# paramString = "qps=1"
# }
],
rpc = [
# {
# component = "protocol.Wallet/GetBlockByLatestNum2",
# strategy = "GlobalPreemptibleAdapter",
# paramString = "permit=1"
# },
# {
# component = "protocol.Wallet/GetAccount",
# strategy = "IPQPSRateLimiterAdapter",
# paramString = "qps=1"
# },
# {
# component = "protocol.Wallet/ListWitnesses",
# strategy = "QpsRateLimiterAdapter",
# paramString = "qps=1"
# },
]
}
seed.node = {
# List of the seed nodes
# Seed nodes are stable full nodes
# example:
# ip.list = [
# "ip:port",
# "ip:port"
# ]
ip.list = [
"3.225.171.164:18888",
"52.53.189.99:18888",
"18.196.99.16:18888",
"34.253.187.192:18888",
"18.133.82.227:18888",
"35.180.51.163:18888",
"54.252.224.209:18888",
"18.231.27.82:18888",
"52.15.93.92:18888",
"34.220.77.106:18888",
"15.207.144.3:18888",
"13.124.62.58:18888",
"13.229.128.108:18888",
"35.182.37.246:18888",
"34.200.228.125:18888",
"18.220.232.201:18888",
"13.57.30.186:18888",
"35.165.103.105:18888",
"18.184.238.21:18888",
"34.250.140.143:18888",
"35.176.192.130:18888",
"52.47.197.188:18888",
"52.62.210.100:18888",
"13.231.4.243:18888",
"18.231.76.29:18888",
"35.154.90.144:18888",
"13.125.210.234:18888",
"13.250.40.82:18888",
"35.183.101.48:18888"
]
}
genesis.block = {
# Reserve balance
assets = [
{
accountName = "Zion"
accountType = "AssetIssue"
address = "TLLM21wteSPs4hKjbxgmH1L6poyMjeTbHm"
balance = "99000000000000000"
},
{
accountName = "Sun"
accountType = "AssetIssue"
address = "TXmVpin5vq5gdZsciyyjdZgKRUju4st1wM"
balance = "0"
},
{
accountName = "Blackhole"
accountType = "AssetIssue"
address = "TLsV52sRDL79HXGGm9yzwKibb6BeruhUzy"
balance = "-9223372036854775808"
}
]
witnesses = [
{
address: THKJYuUmMKKARNf7s2VT51g5uPY6KEqnat,
url = "http://GR1.com",
voteCount = 100000026
},
{
address: TVDmPWGYxgi5DNeW8hXrzrhY8Y6zgxPNg4,
url = "http://GR2.com",
voteCount = 100000025
},
{
address: TWKZN1JJPFydd5rMgMCV5aZTSiwmoksSZv,
url = "http://GR3.com",
voteCount = 100000024
},
{
address: TDarXEG2rAD57oa7JTK785Yb2Et32UzY32,
url = "http://GR4.com",
voteCount = 100000023
},
{
address: TAmFfS4Tmm8yKeoqZN8x51ASwdQBdnVizt,
url = "http://GR5.com",
voteCount = 100000022
},
{
address: TK6V5Pw2UWQWpySnZyCDZaAvu1y48oRgXN,
url = "http://GR6.com",
voteCount = 100000021
},
{
address: TGqFJPFiEqdZx52ZR4QcKHz4Zr3QXA24VL,
url = "http://GR7.com",
voteCount = 100000020
},
{
address: TC1ZCj9Ne3j5v3TLx5ZCDLD55MU9g3XqQW,
url = "http://GR8.com",
voteCount = 100000019
},
{
address: TWm3id3mrQ42guf7c4oVpYExyTYnEGy3JL,
url = "http://GR9.com",
voteCount = 100000018
},
{
address: TCvwc3FV3ssq2rD82rMmjhT4PVXYTsFcKV,
url = "http://GR10.com",
voteCount = 100000017
},
{
address: TFuC2Qge4GxA2U9abKxk1pw3YZvGM5XRir,
url = "http://GR11.com",
voteCount = 100000016
},
{
address: TNGoca1VHC6Y5Jd2B1VFpFEhizVk92Rz85,
url = "http://GR12.com",
voteCount = 100000015
},
{
address: TLCjmH6SqGK8twZ9XrBDWpBbfyvEXihhNS,
url = "http://GR13.com",
voteCount = 100000014
},
{
address: TEEzguTtCihbRPfjf1CvW8Euxz1kKuvtR9,
url = "http://GR14.com",
voteCount = 100000013
},
{
address: TZHvwiw9cehbMxrtTbmAexm9oPo4eFFvLS,
url = "http://GR15.com",
voteCount = 100000012
},
{
address: TGK6iAKgBmHeQyp5hn3imB71EDnFPkXiPR,
url = "http://GR16.com",
voteCount = 100000011
},
{
address: TLaqfGrxZ3dykAFps7M2B4gETTX1yixPgN,
url = "http://GR17.com",
voteCount = 100000010
},
{
address: TX3ZceVew6yLC5hWTXnjrUFtiFfUDGKGty,
url = "http://GR18.com",
voteCount = 100000009
},
{
address: TYednHaV9zXpnPchSywVpnseQxY9Pxw4do,
url = "http://GR19.com",
voteCount = 100000008
},
{
address: TCf5cqLffPccEY7hcsabiFnMfdipfyryvr,
url = "http://GR20.com",
voteCount = 100000007
},
{
address: TAa14iLEKPAetX49mzaxZmH6saRxcX7dT5,
url = "http://GR21.com",
voteCount = 100000006
},
{
address: TBYsHxDmFaRmfCF3jZNmgeJE8sDnTNKHbz,
url = "http://GR22.com",
voteCount = 100000005
},
{
address: TEVAq8dmSQyTYK7uP1ZnZpa6MBVR83GsV6,
url = "http://GR23.com",
voteCount = 100000004
},
{
address: TRKJzrZxN34YyB8aBqqPDt7g4fv6sieemz,
url = "http://GR24.com",
voteCount = 100000003
},
{
address: TRMP6SKeFUt5NtMLzJv8kdpYuHRnEGjGfe,
url = "http://GR25.com",
voteCount = 100000002
},
{
address: TDbNE1VajxjpgM5p7FyGNDASt3UVoFbiD3,
url = "http://GR26.com",
voteCount = 100000001
},
{
address: TLTDZBcPoJ8tZ6TTEeEqEvwYFk2wgotSfD,
url = "http://GR27.com",
voteCount = 100000000
}
]
timestamp = "0" #2017-8-26 12:00:00
parentHash = "0xe58f33f9baf9305dc6f82b9f1934ea8f0ade2defb951258d50167028c780351f"
}
// Optional.The default is empty.
// It is used when the witness account has set the witnessPermission.
// When it is not empty, the localWitnessAccountAddress represents the address of the witness account,
// and the localwitness is configured with the private key of the witnessPermissionAddress in the witness account.
// When it is empty,the localwitness is configured with the private key of the witness account.
//localWitnessAccountAddress =
localwitness = [
]
#localwitnesskeystore = [
# "localwitnesskeystore.json"
#]
block = {
needSyncCheck = true
maintenanceTimeInterval = 21600000
proposalExpireTime = 259200000 // 3 day: 259200000(ms)
}
# Transaction reference block, default is "head", configure to "solid" can avoid TaPos error
trx.reference.block = "solid" // head;solid;
# This property sets the number of milliseconds after the creation of the transaction that is expired, default value is 60000.
# trx.expiration.timeInMilliseconds = 60000
vm = {
supportConstant = true
minTimeRatio = 0.0
maxTimeRatio = 5.0
saveInternalTx = false
# In rare cases, transactions that will be within the specified maximum execution time (default 10(ms)) are re-executed and packaged
# longRunningTime = 10
}
committee = {
allowCreationOfContracts = 0 //mainnet:0 (reset by committee),test:1
allowAdaptiveEnergy = 0 //mainnet:0 (reset by committee),test:1
# allowOldRewardOpt = 1
}
event.subscribe = {
native = {
useNativeQueue = true // if true, use native message queue, else use event plugin.
bindport = 5555 // bind port
sendqueuelength = 1000 //max length of send queue
}
path = "" // absolute path of plugin
server = "" // target server address to receive event triggers
dbconfig="" // dbname|username|password
contractParse = true,
topics = [
{
triggerName = "block" // block trigger, the value can't be modified
enable = false
topic = "block" // plugin topic, the value could be modified
},
{
triggerName = "transaction"
enable = false
topic = "transaction"
},
{
triggerName = "contractevent"
enable = false
topic = "contractevent"
},
{
triggerName = "contractlog"
enable = false
topic = "contractlog"
},
{
triggerName = "solidity" // solidity block event trigger, the value can't be modified
enable = true // the default value is true
topic = "solidity"
},
{
triggerName = "solidityevent"
enable = false
topic = "solidityevent"
},
{
triggerName = "soliditylog"
enable = false
topic = "soliditylog"
}
]
filter = {
fromblock = "" // the value could be "", "earliest" or a specified block number as the beginning of the queried range
toblock = "" // the value could be "", "latest" or a specified block number as end of the queried range
contractAddress = [
"" // contract address you want to subscribe, if it's set to "", you will receive contract logs/events with any contract address.
]
contractTopic = [
"" // contract topic you want to subscribe, if it's set to "", you will receive contract logs/events with any contract topic.
]
}
}
Based on the configuration file, it appears that the issue may be caused by using checkpoint v2 without querying from it in the code. However, I have noticed that the pull request (https://github.com/o85372940/java-tron/pull/8) may fix the issue. You could merge it and check again to see if it resolves the problem.
How to fix this bug? Is there any good solution?
How to fix this bug? Is there any good solution?
The pull request available at https://github.com/o85372940/java-tron/pull/8 can be merged to see if it fixes the issue.
close with #5658.
I use Toolkit pruning Lite Fullnode data regularly.
Expected behaviour
It can be executed normally.
Actual behaviour
The command sometimes quits abnormally.
Steps to reproduce the behaviour
Backtrace