search

vegaprotocol / vega

A Go implementation of the Vega Protocol, a protocol for creating and trading derivatives on a fully decentralised network.
https://vega.xyz
GNU Affero General Public License v3.0
37 stars 21 forks source link

[Bug]: panic: re submit special order would cross #11583

Closed cdummett closed 1 week ago

cdummett commented 1 month ago

Problem encountered

vega-market-sim short fuzz run against core PR #11572 panics.

link to artifacts for failing test - https://jenkins.vega.rocks/job/common/job/vega-market-sim/7858/artifact/network_home/vega-sim-2f_koeo5/

2024-08-14T14:02:03.417Z    INFO    core    node/node.go:99 starting vega   {"version": "v0.78.0-dev", "commit-hash": "088c704619b4e227df4e84dd8684d419e2bfdfdb"}
panic: re submit special order would cross

goroutine 56245 [running]:
go.uber.org/zap/zapcore.CheckWriteAction.OnWrite(0x2?, 0x2?, {0x0?, 0x0?, 0xc00e0fe4a0?})
    /jenkins/GOPATH/pkg/mod/go.uber.org/zap@v1.25.0/zapcore/entry.go:196 +0x54
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc00321a410, {0xc00bb28080, 0x2, 0x2})
    /jenkins/GOPATH/pkg/mod/go.uber.org/zap@v1.25.0/zapcore/entry.go:262 +0x3ec
go.uber.org/zap.(*Logger).Panic(0x9929a705c2760000?, {0x4096659?, 0x0?}, {0xc00bb28080, 0x2, 0x2})
    /jenkins/GOPATH/pkg/mod/go.uber.org/zap@v1.25.0/logger.go:258 +0x51
code.vegaprotocol.io/vega/core/matching.(*OrderBook).ReSubmitSpecialOrders(0xc005091860, 0xc00343bad0)
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/matching/orderbook.go:945 +0x9b6
code.vegaprotocol.io/vega/core/execution/future.(*Market).reSubmitPeggedOrders(0xc005108380, {0x4853b38, 0xc00862c630}, {0xc003269800, 0x4e, 0x0?})
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/execution/future/special_orders.go:112 +0xf3
code.vegaprotocol.io/vega/core/execution/future.(*Market).repriceAllSpecialOrders(0xc005108380, {0x4853b38, 0xc00862c630}, 0x0?, {0xc00a3d1000?, 0x3d5e380?, 0x4853b38?})
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/execution/future/special_orders.go:151 +0x117
code.vegaprotocol.io/vega/core/execution/future.(*Market).checkForReferenceMoves(0xc005108380, {0x4853b38, 0xc00862c630}, {0xc00a3d1000, 0xdb, 0x100}, 0x1)
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/execution/future/reference_price_moves.go:54 +0x2b1
code.vegaprotocol.io/vega/core/execution/future.(*Market).leaveAuction(0xc005108380, {0x4853b38?, 0xc00862c630}, {0x0?, 0xedbec17a9?, 0x0?})
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/execution/future/market.go:1817 +0x1314
code.vegaprotocol.io/vega/core/execution/future.(*Market).checkAuction(0xc005108380, {0x4853b38?, 0xc00862c630}, {0x0?, 0xedbec164b?, 0x0?}, {0x4822980?, 0xc001e838a8})
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/execution/future/auction.go:155 +0x103f
code.vegaprotocol.io/vega/core/execution/future.(*Market).OnTick(0xc005108380, {0x4853b38, 0xc00862c630}, {0x40?, 0x0?, 0x0?})
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/execution/future/market.go:1083 +0x6d8
code.vegaprotocol.io/vega/core/execution.(*Engine).OnTick(0xc000e3d880, {0x4853b38, 0xc00862c630}, {0x1?, 0x48738e0?, 0x0?})
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/execution/engine.go:1223 +0x4f1
code.vegaprotocol.io/vega/core/vegatime.(*Svc).notify(...)
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/vegatime/service.go:111
code.vegaprotocol.io/vega/core/vegatime.(*Svc).SetTimeNow(0xc000d0e900, {0x4853b38, 0xc00862c630}, {0x40?, 0x6dfb500?, 0x0?})
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/vegatime/service.go:80 +0x263
code.vegaprotocol.io/vega/core/processor.(*App).OnBeginBlock(0xc000e3dc00, 0x70a, {0xc00a069740, 0x40}, {0xc00641fb00?, 0x453c69?, 0x0?}, {0x0, 0x0}, {0x6dfb500, ...})
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/processor/abci.go:1347 +0x976
code.vegaprotocol.io/vega/core/blockchain/abci.(*App).FinalizeBlock(0xc0006b0f00, {0xb8?, 0x3ca4000?}, 0xc00863bc80)
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/blockchain/abci/abci.go:168 +0x449
code.vegaprotocol.io/vega/cmd/vega/commands/node.(*appW).FinalizeBlock(0xc0039ec768?, {0x4853b00?, 0x6dfb500?}, 0x3b9aca00?)
    /jenkins/workspace/common/vega-market-sim/extern/vega/cmd/vega/commands/node/app_wrapper.go:91 +0x25
code.vegaprotocol.io/vega/core/blockchain/nullchain.(*NullBlockchain).processBlock(0xc000777200)
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/blockchain/nullchain/nullchain.go:212 +0x556
code.vegaprotocol.io/vega/core/blockchain/nullchain.(*NullBlockchain).ForwardTime(0xc000777200, 0x60db88400)
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/blockchain/nullchain/nullchain.go:289 +0x1ce
code.vegaprotocol.io/vega/core/blockchain/nullchain.(*NullBlockchain).handleForwardTime.func1()
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/blockchain/nullchain/server.go:132 +0x25
created by code.vegaprotocol.io/vega/core/blockchain/nullchain.(*NullBlockchain).handleForwardTime in goroutine 56243
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/blockchain/nullchain/server.go:131 +0x22f

Observed behaviour

panic

Expected behaviour

no panic

Steps to reproduce

replay

Software version

088c704619b4e227df4e84dd8684d419e2bfdfdb

Failing test

vega-market-sim fuzz test

Jenkins run

https://jenkins.vega.rocks/blue/organizations/jenkins/common%2Fvega-market-sim/detail/vega-market-sim/7858/pipeline/

Configuration used

{
  "genesis_time": "2023-05-09T10:00:00.000000000Z",
  "chain_id": "CUSTOM",
  "initial_height": "0",
  "consensus_params": {
    "block": {
      "max_bytes": "22020096",
      "max_gas": "-1",
      "time_iota_ms": "1"
    },
    "evidence": {
      "max_age_num_blocks": "100000",
      "max_age_duration": "172800000000000",
      "max_bytes": "1048576"
    },
    "validator": {
      "pub_key_types": ["ed25519"]
    },
    "version": {}
  },
  "validators": [
    {
      "address": "7DFD3744A773C166B36F170E4F5EF3E998703F4E",
      "pub_key": {
        "type": "tendermint/PubKeyEd25519",
        "value": "2lPO6I0p5uCBZzeoA2+ECJ2+57LtLVMjXpxH5za9TvE="
      },
      "power": "10",
      "name": ""
    }
  ],
  "app_hash": "",
  "app_state": {
    "assets": {
      "0000000000000000000000000000000000000000000000000000000000000000": {
        "name": "VOTE",
        "symbol": "VOTE",
        "decimals": 5,
        "quantum": "1",
        "min_lp_stake": "1",
        "source": {
          "builtin_asset": {
            "max_faucet_amount_mint": "100000000000000000"
          }
        }
      }
    },
    "validators": {
      "6NemqXSfN4zfZvAQeIANflEFqQ98izX9C0KLI2dITSY=": {
        "id": "c2b0ab69fb1bd19862c04df401920ea62d3e0bce41a50148240f42017824d17a",
        "vega_pub_key": "dfcf0fa8d67cfa11b3a511e320f3c47807a5e0735cc06dd9c2bc00f7ccc67117",
        "vega_pub_key_index": 1,
        "ethereum_address": "0x9e5BEEC6E56B28cCbd02864840B0f1e0125e42Ce",
        "tm_pub_key": "6NemqXSfN4zfZvAQeIANflEFqQ98izX9C0KLI2dITSY=",
        "info_url": "",
        "country": "",
        "name": "",
        "avatar_url": ""
      }
    },
    "network": {
      "replay_attack_threshold": 150
    },
    "network_parameters": {
      "blockchains.ethereumConfig": "{\"network_id\": \"3\", \"chain_id\": \"3\", \"collateral_bridge_contract\": { \"address\": \"0xa6F1E140daC13002Dfd9789D6dBA59117c717D7a\" }, \"confirmations\": 50, \"staking_bridge_contract\": { \"address\": \"0xfce2CC92203A266a9C8e67461ae5067c78f67235\", \"deployment_block_height\": 11001702}, \"multisig_control_contract\": {\"address\": \"0xCF6d41235911184fe6F35D47207813bFF3B91601\", \"deployment_block_height\": 12710009 } }",
      "blockchains.evmBridgeConfigs": "{\"configs\":[{\"network_id\": \"3\", \"chain_id\": \"3\", \"collateral_bridge_contract\": { \"address\": \"0xa6F1E140daC13002Dfd9789D6dBA59117c717D7a\" }, \"confirmations\": 50, \"multisig_control_contract\": {\"address\": \"0xCF6d41235911184fe6F35D47207813bFF3B91601\", \"deployment_block_height\": 12710009 } }]}",
      "blockchains.ethereumRpcAndEvmCompatDataSourcesConfig": "{\"configs\": []}",
      "governance.proposal.asset.maxClose": "8760h0m0s",
      "governance.proposal.asset.maxEnact": "8760h0m0s",
      "governance.proposal.asset.minClose": "1s",
      "governance.proposal.asset.minEnact": "1s",
      "governance.proposal.asset.minProposerBalance": "1",
      "governance.proposal.asset.minVoterBalance": "1",
      "governance.proposal.asset.requiredMajority": "0.66",
      "governance.proposal.asset.requiredParticipation": "0.00001",
      "governance.proposal.freeform.maxClose": "8760h0m0s",
      "governance.proposal.freeform.minClose": "1s",
      "governance.proposal.freeform.minProposerBalance": "1",
      "governance.proposal.freeform.minVoterBalance": "1",
      "governance.proposal.freeform.requiredMajority": "0.66",
      "governance.proposal.freeform.requiredParticipation": "0.00001",
      "governance.proposal.market.maxClose": "8760h0m0s",
      "governance.proposal.market.maxEnact": "8760h0m0s",
      "governance.proposal.market.minClose": "1s",
      "governance.proposal.market.minEnact": "2s",
      "governance.proposal.market.minProposerBalance": "1",
      "governance.proposal.market.minVoterBalance": "1",
      "governance.proposal.market.requiredMajority": "0.66",
      "governance.proposal.market.requiredParticipation": "0.00001",
      "governance.proposal.updateAsset.maxClose": "720h",
      "governance.proposal.updateAsset.maxEnact": "720h",
      "governance.proposal.updateAsset.minClose": "1s",
      "governance.proposal.updateAsset.minEnact": "1s",
      "governance.proposal.updateAsset.minProposerBalance": "1",
      "governance.proposal.updateAsset.minVoterBalance": "1",
      "governance.proposal.updateAsset.requiredMajority": "0.66",
      "governance.proposal.updateAsset.requiredParticipation": "0.09",
      "governance.proposal.updateMarket.maxClose": "8760h0m0s",
      "governance.proposal.updateMarket.maxEnact": "8760h0m0s",
      "governance.proposal.updateMarket.minClose": "1s",
      "governance.proposal.updateMarket.minEnact": "1s",
      "governance.proposal.updateMarket.minProposerBalance": "1",
      "governance.proposal.updateMarket.minProposerEquityLikeShare": "0",
      "governance.proposal.updateMarket.minVoterBalance": "1",
      "governance.proposal.updateMarket.requiredMajority": "0.66",
      "governance.proposal.updateMarket.requiredMajorityLP": "0.66",
      "governance.proposal.updateMarket.requiredParticipation": "0.00001",
      "governance.proposal.updateMarket.requiredParticipationLP": "0",
      "governance.proposal.updateNetParam.maxClose": "8760h0m0s",
      "governance.proposal.updateNetParam.maxEnact": "8760h0m0s",
      "governance.proposal.updateNetParam.minClose": "1s",
      "governance.proposal.updateNetParam.minEnact": "1s",
      "governance.proposal.updateNetParam.minProposerBalance": "1",
      "governance.proposal.updateNetParam.minVoterBalance": "1",
      "governance.proposal.updateNetParam.requiredMajority": "0.5",
      "governance.proposal.updateNetParam.requiredParticipation": "0.00001",
      "governance.proposal.referralProgram.maxClose": "8760h0m0s",
      "governance.proposal.referralProgram.maxEnact": "8760h0m0s",
      "governance.proposal.referralProgram.minClose": "1s",
      "governance.proposal.referralProgram.minEnact": "1s",
      "governance.proposal.referralProgram.minProposerBalance": "1",
      "governance.proposal.referralProgram.minVoterBalance": "1",
      "governance.proposal.referralProgram.requiredMajority": "0.5",
      "governance.proposal.referralProgram.requiredParticipation": "0.00001",
      "governance.proposal.VolumeDiscountProgram.maxClose": "8760h0m0s",
      "governance.proposal.VolumeDiscountProgram.maxEnact": "8760h0m0s",
      "governance.proposal.VolumeDiscountProgram.minClose": "1s",
      "governance.proposal.VolumeDiscountProgram.minEnact": "1s",
      "governance.proposal.VolumeDiscountProgram.minProposerBalance": "1",
      "governance.proposal.VolumeDiscountProgram.minVoterBalance": "1",
      "governance.proposal.VolumeDiscountProgram.requiredMajority": "0.5",
      "governance.proposal.VolumeDiscountProgram.requiredParticipation": "0.00001",
      "governance.proposal.VolumeRebateProgram.maxClose": "8760h0m0s",
      "governance.proposal.VolumeRebateProgram.maxEnact": "8760h0m0s",
      "governance.proposal.VolumeRebateProgram.minClose": "1s",
      "governance.proposal.VolumeRebateProgram.minEnact": "1s",
      "governance.proposal.VolumeRebateProgram.minProposerBalance": "1",
      "governance.proposal.VolumeRebateProgram.minVoterBalance": "1",
      "governance.proposal.VolumeRebateProgram.requiredMajority": "0.5",
      "governance.proposal.VolumeRebateProgram.requiredParticipation": "0.00001",
      "governance.proposal.transfer.maxClose": "8760h0m0s",
      "governance.proposal.transfer.maxEnact": "8760h0m0s",
      "governance.proposal.transfer.minClose": "1s",
      "governance.proposal.transfer.minEnact": "1s",
      "governance.proposal.transfer.minProposerBalance": "1",
      "governance.proposal.transfer.minVoterBalance": "1",
      "governance.proposal.transfer.requiredMajority": "0.5",
      "governance.proposal.transfer.requiredParticipation": "0.00001",
      "limits.markets.maxPeggedOrders": "1500",
      "limits.markets.proposePerpetualEnabled": "1",
      "limits.markets.ammPoolEnabled": "1",
      "market.auction.maximumDuration": "168h",
      "market.auction.minimumDuration": "1s",
      "market.fee.factors.infrastructureFee": "0.0005",
      "market.fee.factors.makerFee": "0.0002",
      "market.fee.factors.treasuryFee": "0.0001",
      "market.fee.factors.buybackFee": "0.0001",
      "market.liquidity.bondPenaltyParameter": "0.1",
      "market.liquidity.earlyExitPenalty": "0.1",
      "market.liquidity.maximumLiquidityFeeFactorLevel": "0.03",
      "market.liquidity.minimum.probabilityOfTrading.lpOrders": "1e-6",
      "market.liquidity.probabilityOfTrading.tau.scaling": "10.0",
      "market.liquidity.providersFeeCalculationTimeStep": "1s",
      "market.liquidity.sla.nonPerformanceBondPenaltyMax": "0.5",
      "market.liquidity.sla.nonPerformanceBondPenaltySlope": "2",
      "market.liquidity.stakeToCcyVolume": "1.0",
      "market.liquidity.targetstake.triggering.ratio": "0.25",
      "market.liquidityProvision.minLpStakeQuantumMultiple": "5000",
      "market.liquidityProvision.shapes.maxSize": "100",
      "market.margin.scalingFactors": "{\"search_level\": 1.1, \"initial_margin\": 1.5, \"collateral_release\": 1.7}",
      "market.monitor.price.defaultParameters": "{\"triggers\": [{\"auction_extension\": 300, \"horizon\": 43200, \"probability\": \"0.9999999\"}] }",
      "market.stake.target.scalingFactor": "0.0001",
      "market.stake.target.timeWindow": "1h",
      "market.value.windowLength": "2h0m0s",
      "network.checkpoint.timeElapsedBetweenCheckpoints": "5m",
      "network.floatingPointUpdates.delay": "5m",
      "network.markPriceUpdateMaximumFrequency": "5s",
      "network.transaction.defaultgas": "1",
      "network.transactions.maxgasperblock": "3000000",
      "network.transactions.minBlockCapacity": "32",
      "network.validators.ersatz.multipleOfTendermintValidators": "0",
      "network.validators.ersatz.rewardFactor": "0.85",
      "network.validators.incumbentBonus": "0.05",
      "network.validators.minimumEthereumEventsForNewValidator": "3",
      "network.validators.multisig.numberOfSigners": "13",
      "network.validators.tendermint.number": "13",
      "reward.asset": "0000000000000000000000000000000000000000000000000000000000000000",
      "reward.staking.delegation.competitionLevel": "1.3",
      "reward.staking.delegation.delegatorShare": "0.8",
      "reward.staking.delegation.maxPayoutPerEpoch": "9890000000000000000000",
      "reward.staking.delegation.maxPayoutPerParticipant": "700000000000000000000",
      "reward.staking.delegation.minimumValidatorStake": "3000000000000000000000",
      "reward.staking.delegation.minValidators": "5",
      "reward.staking.delegation.optimalStakeMultiplier": "3.0",
      "reward.staking.delegation.payoutDelay": "0h",
      "reward.staking.delegation.payoutFraction": "1.0",
      "rewards.marketCreationQuantumMultiple": "1",
      "snapshot.interval.length": "300",
      "spam.pow.difficulty": "15",
      "spam.pow.hashFunction": "sha3_24_rounds",
      "spam.pow.increaseDifficulty": "0",
      "spam.pow.numberOfPastBlocks": "100",
      "spam.pow.numberOfTxPerBlock": "1000",
      "spam.protection.delegation.min.tokens": "100000000000000000",
      "spam.protection.max.batchSize": "30",
      "spam.protection.max.delegations": "360",
      "spam.protection.max.proposals": "300",
      "spam.protection.max.votes": "300",
      "spam.protection.maxUserTransfersPerEpoch": "10000",
      "spam.protection.minimumWithdrawalQuantumMultiple": "10",
      "spam.protection.minMultisigUpdates": "100000000000000000000",
      "spam.protection.proposal.min.tokens": "2000000000000000000000",
      "spam.protection.voting.min.tokens": "1000000000000000000",
      "spam.protection.applyReferral.min.funds": "0",
      "transfer.fee.factor": "0.001",
      "transfer.minTransferQuantumMultiple": "0",
      "transfer.fee.maxQuantumAmount": "100",
      "transfer.feeDiscountMinimumTrackedAmount": "0.001",
      "transfer.feeDiscountDecayFraction": "0.5",
      "validator.performance.scaling.factor": "0",
      "validators.delegation.minAmount": "1",
      "validators.epoch.length": "2m",
      "validators.vote.required": "0.67",
      "referralProgram.maxReferralTiers": "10",
      "referralProgram.maxReferralRewardFactor": "1",
      "referralProgram.maxReferralDiscountFactor": "1",
      "referralProgram.maxPartyNotionalVolumeByQuantumPerEpoch": "10000",
      "referralProgram.minStakedVegaTokens": "0",
      "referralProgram.maxReferralRewardProportion": "1",
      "market.aggressiveOrderBlockDelay": "1",
      "market.amm.minCommitmentQuantum": "1",
      "market.liquidity.maxAmmCalculationLevels": "100"
    },
    "network_limits": {
      "propose_market_enabled": true,
      "propose_asset_enabled": true,
      "bootstrap_block_count": 0
    },
    "checkpoint": {
      "load_hash": ""
    }
  }
}

Relevant log output

panic: re submit special order would cross

goroutine 56245 [running]:
go.uber.org/zap/zapcore.CheckWriteAction.OnWrite(0x2?, 0x2?, {0x0?, 0x0?, 0xc00e0fe4a0?})
    /jenkins/GOPATH/pkg/mod/go.uber.org/zap@v1.25.0/zapcore/entry.go:196 +0x54
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc00321a410, {0xc00bb28080, 0x2, 0x2})
    /jenkins/GOPATH/pkg/mod/go.uber.org/zap@v1.25.0/zapcore/entry.go:262 +0x3ec
go.uber.org/zap.(*Logger).Panic(0x9929a705c2760000?, {0x4096659?, 0x0?}, {0xc00bb28080, 0x2, 0x2})
    /jenkins/GOPATH/pkg/mod/go.uber.org/zap@v1.25.0/logger.go:258 +0x51
code.vegaprotocol.io/vega/core/matching.(*OrderBook).ReSubmitSpecialOrders(0xc005091860, 0xc00343bad0)
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/matching/orderbook.go:945 +0x9b6
code.vegaprotocol.io/vega/core/execution/future.(*Market).reSubmitPeggedOrders(0xc005108380, {0x4853b38, 0xc00862c630}, {0xc003269800, 0x4e, 0x0?})
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/execution/future/special_orders.go:112 +0xf3
code.vegaprotocol.io/vega/core/execution/future.(*Market).repriceAllSpecialOrders(0xc005108380, {0x4853b38, 0xc00862c630}, 0x0?, {0xc00a3d1000?, 0x3d5e380?, 0x4853b38?})
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/execution/future/special_orders.go:151 +0x117
code.vegaprotocol.io/vega/core/execution/future.(*Market).checkForReferenceMoves(0xc005108380, {0x4853b38, 0xc00862c630}, {0xc00a3d1000, 0xdb, 0x100}, 0x1)
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/execution/future/reference_price_moves.go:54 +0x2b1
code.vegaprotocol.io/vega/core/execution/future.(*Market).leaveAuction(0xc005108380, {0x4853b38?, 0xc00862c630}, {0x0?, 0xedbec17a9?, 0x0?})
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/execution/future/market.go:1817 +0x1314
code.vegaprotocol.io/vega/core/execution/future.(*Market).checkAuction(0xc005108380, {0x4853b38?, 0xc00862c630}, {0x0?, 0xedbec164b?, 0x0?}, {0x4822980?, 0xc001e838a8})
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/execution/future/auction.go:155 +0x103f
code.vegaprotocol.io/vega/core/execution/future.(*Market).OnTick(0xc005108380, {0x4853b38, 0xc00862c630}, {0x40?, 0x0?, 0x0?})
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/execution/future/market.go:1083 +0x6d8
code.vegaprotocol.io/vega/core/execution.(*Engine).OnTick(0xc000e3d880, {0x4853b38, 0xc00862c630}, {0x1?, 0x48738e0?, 0x0?})
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/execution/engine.go:1223 +0x4f1
code.vegaprotocol.io/vega/core/vegatime.(*Svc).notify(...)
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/vegatime/service.go:111
code.vegaprotocol.io/vega/core/vegatime.(*Svc).SetTimeNow(0xc000d0e900, {0x4853b38, 0xc00862c630}, {0x40?, 0x6dfb500?, 0x0?})
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/vegatime/service.go:80 +0x263
code.vegaprotocol.io/vega/core/processor.(*App).OnBeginBlock(0xc000e3dc00, 0x70a, {0xc00a069740, 0x40}, {0xc00641fb00?, 0x453c69?, 0x0?}, {0x0, 0x0}, {0x6dfb500, ...})
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/processor/abci.go:1347 +0x976
code.vegaprotocol.io/vega/core/blockchain/abci.(*App).FinalizeBlock(0xc0006b0f00, {0xb8?, 0x3ca4000?}, 0xc00863bc80)
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/blockchain/abci/abci.go:168 +0x449
code.vegaprotocol.io/vega/cmd/vega/commands/node.(*appW).FinalizeBlock(0xc0039ec768?, {0x4853b00?, 0x6dfb500?}, 0x3b9aca00?)
    /jenkins/workspace/common/vega-market-sim/extern/vega/cmd/vega/commands/node/app_wrapper.go:91 +0x25
code.vegaprotocol.io/vega/core/blockchain/nullchain.(*NullBlockchain).processBlock(0xc000777200)
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/blockchain/nullchain/nullchain.go:212 +0x556
code.vegaprotocol.io/vega/core/blockchain/nullchain.(*NullBlockchain).ForwardTime(0xc000777200, 0x60db88400)
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/blockchain/nullchain/nullchain.go:289 +0x1ce
code.vegaprotocol.io/vega/core/blockchain/nullchain.(*NullBlockchain).handleForwardTime.func1()
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/blockchain/nullchain/server.go:132 +0x25
created by code.vegaprotocol.io/vega/core/blockchain/nullchain.(*NullBlockchain).handleForwardTime in goroutine 56243
    /jenkins/workspace/common/vega-market-sim/extern/vega/core/blockchain/nullchain/server.go:131 +0x22f
cdummett commented 3 weeks ago

Archived run lost - adding dump from similar panic vega-sim-n8vo26dp.zip

cdummett commented 2 weeks ago

Adding dump from similar panic with core commit and genesis.

vega-sim-725_u08z.zip

2024-08-24T18:41:21.240Z    INFO    core    node/node.go:99 starting vega   {"version": "v0.78.0-dev", "commit-hash": "750a7a3d1d4ababa48132666c8dbb79786c4ba69"}
{
  "genesis_time": "2023-05-09T10:00:00.000000000Z",
  "chain_id": "CUSTOM",
  "initial_height": "0",
  "consensus_params": {
    "block": {
      "max_bytes": "22020096",
      "max_gas": "-1",
      "time_iota_ms": "1"
    },
    "evidence": {
      "max_age_num_blocks": "100000",
      "max_age_duration": "172800000000000",
      "max_bytes": "1048576"
    },
    "validator": {
      "pub_key_types": ["ed25519"]
    },
    "version": {}
  },
  "validators": [
    {
      "address": "7DFD3744A773C166B36F170E4F5EF3E998703F4E",
      "pub_key": {
        "type": "tendermint/PubKeyEd25519",
        "value": "2lPO6I0p5uCBZzeoA2+ECJ2+57LtLVMjXpxH5za9TvE="
      },
      "power": "10",
      "name": ""
    }
  ],
  "app_hash": "",
  "app_state": {
    "assets": {
      "0000000000000000000000000000000000000000000000000000000000000000": {
        "name": "VOTE",
        "symbol": "VOTE",
        "decimals": 5,
        "quantum": "1",
        "min_lp_stake": "1",
        "source": {
          "builtin_asset": {
            "max_faucet_amount_mint": "100000000000000000"
          }
        }
      }
    },
    "validators": {
      "6NemqXSfN4zfZvAQeIANflEFqQ98izX9C0KLI2dITSY=": {
        "id": "c2b0ab69fb1bd19862c04df401920ea62d3e0bce41a50148240f42017824d17a",
        "vega_pub_key": "dfcf0fa8d67cfa11b3a511e320f3c47807a5e0735cc06dd9c2bc00f7ccc67117",
        "vega_pub_key_index": 1,
        "ethereum_address": "0x9e5BEEC6E56B28cCbd02864840B0f1e0125e42Ce",
        "tm_pub_key": "6NemqXSfN4zfZvAQeIANflEFqQ98izX9C0KLI2dITSY=",
        "info_url": "",
        "country": "",
        "name": "",
        "avatar_url": ""
      }
    },
    "network": {
      "replay_attack_threshold": 150
    },
    "network_parameters": {
      "blockchains.ethereumConfig": "{\"network_id\": \"3\", \"chain_id\": \"3\", \"collateral_bridge_contract\": { \"address\": \"0xa6F1E140daC13002Dfd9789D6dBA59117c717D7a\" }, \"confirmations\": 50, \"staking_bridge_contract\": { \"address\": \"0xfce2CC92203A266a9C8e67461ae5067c78f67235\", \"deployment_block_height\": 11001702}, \"multisig_control_contract\": {\"address\": \"0xCF6d41235911184fe6F35D47207813bFF3B91601\", \"deployment_block_height\": 12710009 } }",
      "blockchains.evmBridgeConfigs": "{\"configs\":[{\"network_id\": \"3\", \"chain_id\": \"3\", \"collateral_bridge_contract\": { \"address\": \"0xa6F1E140daC13002Dfd9789D6dBA59117c717D7a\" }, \"confirmations\": 50, \"multisig_control_contract\": {\"address\": \"0xCF6d41235911184fe6F35D47207813bFF3B91601\", \"deployment_block_height\": 12710009 } }]}",
      "blockchains.ethereumRpcAndEvmCompatDataSourcesConfig": "{\"configs\": []}",
      "governance.proposal.asset.maxClose": "8760h0m0s",
      "governance.proposal.asset.maxEnact": "8760h0m0s",
      "governance.proposal.asset.minClose": "1s",
      "governance.proposal.asset.minEnact": "1s",
      "governance.proposal.asset.minProposerBalance": "1",
      "governance.proposal.asset.minVoterBalance": "1",
      "governance.proposal.asset.requiredMajority": "0.66",
      "governance.proposal.asset.requiredParticipation": "0.00001",
      "governance.proposal.freeform.maxClose": "8760h0m0s",
      "governance.proposal.freeform.minClose": "1s",
      "governance.proposal.freeform.minProposerBalance": "1",
      "governance.proposal.freeform.minVoterBalance": "1",
      "governance.proposal.freeform.requiredMajority": "0.66",
      "governance.proposal.freeform.requiredParticipation": "0.00001",
      "governance.proposal.market.maxClose": "8760h0m0s",
      "governance.proposal.market.maxEnact": "8760h0m0s",
      "governance.proposal.market.minClose": "1s",
      "governance.proposal.market.minEnact": "2s",
      "governance.proposal.market.minProposerBalance": "1",
      "governance.proposal.market.minVoterBalance": "1",
      "governance.proposal.market.requiredMajority": "0.66",
      "governance.proposal.market.requiredParticipation": "0.00001",
      "governance.proposal.updateAsset.maxClose": "720h",
      "governance.proposal.updateAsset.maxEnact": "720h",
      "governance.proposal.updateAsset.minClose": "1s",
      "governance.proposal.updateAsset.minEnact": "1s",
      "governance.proposal.updateAsset.minProposerBalance": "1",
      "governance.proposal.updateAsset.minVoterBalance": "1",
      "governance.proposal.updateAsset.requiredMajority": "0.66",
      "governance.proposal.updateAsset.requiredParticipation": "0.09",
      "governance.proposal.updateMarket.maxClose": "8760h0m0s",
      "governance.proposal.updateMarket.maxEnact": "8760h0m0s",
      "governance.proposal.updateMarket.minClose": "1s",
      "governance.proposal.updateMarket.minEnact": "1s",
      "governance.proposal.updateMarket.minProposerBalance": "1",
      "governance.proposal.updateMarket.minProposerEquityLikeShare": "0",
      "governance.proposal.updateMarket.minVoterBalance": "1",
      "governance.proposal.updateMarket.requiredMajority": "0.66",
      "governance.proposal.updateMarket.requiredMajorityLP": "0.66",
      "governance.proposal.updateMarket.requiredParticipation": "0.00001",
      "governance.proposal.updateMarket.requiredParticipationLP": "0",
      "governance.proposal.updateNetParam.maxClose": "8760h0m0s",
      "governance.proposal.updateNetParam.maxEnact": "8760h0m0s",
      "governance.proposal.updateNetParam.minClose": "1s",
      "governance.proposal.updateNetParam.minEnact": "1s",
      "governance.proposal.updateNetParam.minProposerBalance": "1",
      "governance.proposal.updateNetParam.minVoterBalance": "1",
      "governance.proposal.updateNetParam.requiredMajority": "0.5",
      "governance.proposal.updateNetParam.requiredParticipation": "0.00001",
      "governance.proposal.referralProgram.maxClose": "8760h0m0s",
      "governance.proposal.referralProgram.maxEnact": "8760h0m0s",
      "governance.proposal.referralProgram.minClose": "1s",
      "governance.proposal.referralProgram.minEnact": "1s",
      "governance.proposal.referralProgram.minProposerBalance": "1",
      "governance.proposal.referralProgram.minVoterBalance": "1",
      "governance.proposal.referralProgram.requiredMajority": "0.5",
      "governance.proposal.referralProgram.requiredParticipation": "0.00001",
      "governance.proposal.VolumeDiscountProgram.maxClose": "8760h0m0s",
      "governance.proposal.VolumeDiscountProgram.maxEnact": "8760h0m0s",
      "governance.proposal.VolumeDiscountProgram.minClose": "1s",
      "governance.proposal.VolumeDiscountProgram.minEnact": "1s",
      "governance.proposal.VolumeDiscountProgram.minProposerBalance": "1",
      "governance.proposal.VolumeDiscountProgram.minVoterBalance": "1",
      "governance.proposal.VolumeDiscountProgram.requiredMajority": "0.5",
      "governance.proposal.VolumeDiscountProgram.requiredParticipation": "0.00001",
      "governance.proposal.VolumeRebateProgram.maxClose": "8760h0m0s",
      "governance.proposal.VolumeRebateProgram.maxEnact": "8760h0m0s",
      "governance.proposal.VolumeRebateProgram.minClose": "1s",
      "governance.proposal.VolumeRebateProgram.minEnact": "1s",
      "governance.proposal.VolumeRebateProgram.minProposerBalance": "1",
      "governance.proposal.VolumeRebateProgram.minVoterBalance": "1",
      "governance.proposal.VolumeRebateProgram.requiredMajority": "0.5",
      "governance.proposal.VolumeRebateProgram.requiredParticipation": "0.00001",
      "governance.proposal.transfer.maxClose": "8760h0m0s",
      "governance.proposal.transfer.maxEnact": "8760h0m0s",
      "governance.proposal.transfer.minClose": "1s",
      "governance.proposal.transfer.minEnact": "1s",
      "governance.proposal.transfer.minProposerBalance": "1",
      "governance.proposal.transfer.minVoterBalance": "1",
      "governance.proposal.transfer.requiredMajority": "0.5",
      "governance.proposal.transfer.requiredParticipation": "0.00001",
      "limits.markets.maxPeggedOrders": "1500",
      "limits.markets.proposePerpetualEnabled": "1",
      "limits.markets.ammPoolEnabled": "1",
      "market.auction.maximumDuration": "168h",
      "market.auction.minimumDuration": "1s",
      "market.fee.factors.infrastructureFee": "0.0005",
      "market.fee.factors.makerFee": "0.0002",
      "market.fee.factors.treasuryFee": "0.0001",
      "market.fee.factors.buybackFee": "0.0001",
      "market.liquidity.bondPenaltyParameter": "0.1",
      "market.liquidity.earlyExitPenalty": "0.1",
      "market.liquidity.maximumLiquidityFeeFactorLevel": "0.03",
      "market.liquidity.minimum.probabilityOfTrading.lpOrders": "1e-6",
      "market.liquidity.probabilityOfTrading.tau.scaling": "10.0",
      "market.liquidity.providersFeeCalculationTimeStep": "1s",
      "market.liquidity.sla.nonPerformanceBondPenaltyMax": "0.5",
      "market.liquidity.sla.nonPerformanceBondPenaltySlope": "2",
      "market.liquidity.stakeToCcyVolume": "1.0",
      "market.liquidity.targetstake.triggering.ratio": "0.25",
      "market.liquidityProvision.minLpStakeQuantumMultiple": "5000",
      "market.liquidityProvision.shapes.maxSize": "100",
      "market.margin.scalingFactors": "{\"search_level\": 1.1, \"initial_margin\": 1.5, \"collateral_release\": 1.7}",
      "market.monitor.price.defaultParameters": "{\"triggers\": [{\"auction_extension\": 300, \"horizon\": 43200, \"probability\": \"0.9999999\"}] }",
      "market.stake.target.scalingFactor": "0.0001",
      "market.stake.target.timeWindow": "1h",
      "market.value.windowLength": "2h0m0s",
      "network.checkpoint.timeElapsedBetweenCheckpoints": "5m",
      "network.floatingPointUpdates.delay": "5m",
      "network.markPriceUpdateMaximumFrequency": "5s",
      "network.transaction.defaultgas": "1",
      "network.transactions.maxgasperblock": "3000000",
      "network.transactions.minBlockCapacity": "32",
      "network.validators.ersatz.multipleOfTendermintValidators": "0",
      "network.validators.ersatz.rewardFactor": "0.85",
      "network.validators.incumbentBonus": "0.05",
      "network.validators.minimumEthereumEventsForNewValidator": "3",
      "network.validators.multisig.numberOfSigners": "13",
      "network.validators.tendermint.number": "13",
      "reward.asset": "0000000000000000000000000000000000000000000000000000000000000000",
      "reward.staking.delegation.competitionLevel": "1.3",
      "reward.staking.delegation.delegatorShare": "0.8",
      "reward.staking.delegation.maxPayoutPerEpoch": "9890000000000000000000",
      "reward.staking.delegation.maxPayoutPerParticipant": "700000000000000000000",
      "reward.staking.delegation.minimumValidatorStake": "3000000000000000000000",
      "reward.staking.delegation.minValidators": "5",
      "reward.staking.delegation.optimalStakeMultiplier": "3.0",
      "reward.staking.delegation.payoutDelay": "0h",
      "reward.staking.delegation.payoutFraction": "1.0",
      "rewards.marketCreationQuantumMultiple": "1",
      "snapshot.interval.length": "300",
      "spam.pow.difficulty": "15",
      "spam.pow.hashFunction": "sha3_24_rounds",
      "spam.pow.increaseDifficulty": "0",
      "spam.pow.numberOfPastBlocks": "100",
      "spam.pow.numberOfTxPerBlock": "1000",
      "spam.protection.delegation.min.tokens": "100000000000000000",
      "spam.protection.max.batchSize": "30",
      "spam.protection.max.delegations": "360",
      "spam.protection.max.proposals": "300",
      "spam.protection.max.votes": "300",
      "spam.protection.maxUserTransfersPerEpoch": "10000",
      "spam.protection.minimumWithdrawalQuantumMultiple": "10",
      "spam.protection.minMultisigUpdates": "100000000000000000000",
      "spam.protection.proposal.min.tokens": "2000000000000000000000",
      "spam.protection.voting.min.tokens": "1000000000000000000",
      "spam.protection.applyReferral.min.funds": "0",
      "transfer.fee.factor": "0.001",
      "transfer.minTransferQuantumMultiple": "0",
      "transfer.fee.maxQuantumAmount": "100",
      "transfer.feeDiscountMinimumTrackedAmount": "0.001",
      "transfer.feeDiscountDecayFraction": "0.5",
      "validator.performance.scaling.factor": "0",
      "validators.delegation.minAmount": "1",
      "validators.epoch.length": "2m",
      "validators.vote.required": "0.67",
      "referralProgram.maxReferralTiers": "10",
      "referralProgram.maxReferralRewardFactor": "1",
      "referralProgram.maxReferralDiscountFactor": "1",
      "referralProgram.maxPartyNotionalVolumeByQuantumPerEpoch": "10000",
      "referralProgram.minStakedVegaTokens": "0",
      "referralProgram.maxReferralRewardProportion": "1",
      "market.aggressiveOrderBlockDelay": "1",
      "market.amm.minCommitmentQuantum": "1",
      "market.liquidity.maxAmmCalculationLevels": "100"
    },
    "network_limits": {
      "propose_market_enabled": true,
      "propose_asset_enabled": true,
      "bootstrap_block_count": 0
    },
    "checkpoint": {
      "load_hash": ""
    }
  }
}

Can be successfully replayed to..

panic: re submit special order would cross [recovered]
    panic: re submit special order would cross

goroutine 171 [running]:
code.vegaprotocol.io/vega/cmd/vega/commands/node.(*Command).Run.func1.1()
    /Users/caed/Projects/vega-market-sim/extern/vega/cmd/vega/commands/node/node.go:166 +0x58
panic({0x1044f53a0?, 0x1400c634970?})
    /usr/local/go/src/runtime/panic.go:914 +0x218
go.uber.org/zap/zapcore.CheckWriteAction.OnWrite(0x0?, 0x2?, {0x2?, 0x0?, 0x0?})
    /Users/caed/go/pkg/mod/go.uber.org/zap@v1.25.0/zapcore/entry.go:196 +0x78
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0x14002c4a0d0, {0x14007975c00, 0x2, 0x2})
    /Users/caed/go/pkg/mod/go.uber.org/zap@v1.25.0/zapcore/entry.go:262 +0x2c0
go.uber.org/zap.(*Logger).Panic(0x70dd8282b54c000?, {0x103aa575a?, 0x0?}, {0x14007975c00, 0x2, 0x2})
    /Users/caed/go/pkg/mod/go.uber.org/zap@v1.25.0/logger.go:258 +0x54
code.vegaprotocol.io/vega/core/matching.(*OrderBook).ReSubmitSpecialOrders(0x14007424a00, 0x14004079ee0)
    /Users/caed/Projects/vega-market-sim/extern/vega/core/matching/orderbook.go:945 +0x814
code.vegaprotocol.io/vega/core/execution/future.(*Market).reSubmitPeggedOrders(0x14007df9500, {0x104e8ac38, 0x14003c52690}, {0x1400e401800, 0x1a, 0x101bd3f20?})
    /Users/caed/Projects/vega-market-sim/extern/vega/core/execution/future/special_orders.go:112 +0xcc
code.vegaprotocol.io/vega/core/execution/future.(*Market).repriceAllSpecialOrders(0x14007df9500, {0x104e8ac38, 0x14003c52690}, 0x50?, {0x1400e5fbb10?, 0x104e8ac38?, 0x14003c52690?})
    /Users/caed/Projects/vega-market-sim/extern/vega/core/execution/future/special_orders.go:151 +0xf0
code.vegaprotocol.io/vega/core/execution/future.(*Market).checkForReferenceMoves(0x14007df9500, {0x104e8ac38, 0x14003c52690}, {0x1400e5fbb10, 0x2, 0x2}, 0x0)
    /Users/caed/Projects/vega-market-sim/extern/vega/core/execution/future/reference_price_moves.go:54 +0x24c
code.vegaprotocol.io/vega/core/execution/future.(*Market).handleConfirmation(0x14007df9500, {0x104e8ac38, 0x14003c52690}, 0x1400ce3b040, 0x0)
    /Users/caed/Projects/vega-market-sim/extern/vega/core/execution/future/market.go:2914 +0x11dc
code.vegaprotocol.io/vega/core/execution/future.(*Market).submitValidatedOrder(0x14007df9500, {0x104e8ac38?, 0x14003c52690}, 0x14003dda750)
    /Users/caed/Projects/vega-market-sim/extern/vega/core/execution/future/market.go:2639 +0x1ff8
code.vegaprotocol.io/vega/core/execution/future.(*Market).submitOrder(0x14007df9500, {0x104e8ac38, 0x14003c52690}, 0x14003dda750)
    /Users/caed/Projects/vega-market-sim/extern/vega/core/execution/future/market.go:2413 +0x194
code.vegaprotocol.io/vega/core/execution/future.(*Market).SubmitOrderWithIDGeneratorAndOrderID(0x14007df9500, {0x104e8ac38?, 0x14003c52690}, 0x40?, {0x1400d6bd400, 0x40}, {0x104e593e0?, 0x1400426cbb8?}, {0x1400753b000, 0x40}, ...)
    /Users/caed/Projects/vega-market-sim/extern/vega/core/execution/future/market.go:2372 +0x50c
code.vegaprotocol.io/vega/core/execution.(*Engine).SubmitOrder(0x14000e2d880, {0x104e8ac38, 0x14003c52690}, 0x14003f876e0, {0x1400d6bd400, 0x40}, {0x104e593e0, 0x1400426cbb8}, {0x1400753b000, 0x40})
    /Users/caed/Projects/vega-market-sim/extern/vega/core/execution/engine.go:976 +0x380
code.vegaprotocol.io/vega/core/processor.(*BMIProcessor).ProcessBatch(0x14007445370, {0x104e8ac38, 0x14003c52690}, 0x1400c4ca0c0, {0x1400d6bd400, 0x40}, {0x1400753af40, 0x40}, {0x104ee1dc8, 0x140008b35c0})
    /Users/caed/Projects/vega-market-sim/extern/vega/core/processor/batch_market_instructions_processor.go:241 +0x1040
code.vegaprotocol.io/vega/core/processor.(*App).DeliverBatchMarketInstructions(0x14000e2dc00, {0x104e8ac38, 0x14003c52690}, {0x104eb7fd0, 0x14007ea3400}, {0x1400753af40, 0x40})
    /Users/caed/Projects/vega-market-sim/extern/vega/core/processor/abci.go:1875 +0xfc
code.vegaprotocol.io/vega/core/processor.NewApp.addDeterministicID.func50({0x104e8ac38, 0x14003c52690}, {0x104eb7fd0, 0x14007ea3400})
    /Users/caed/Projects/vega-market-sim/extern/vega/core/processor/abci.go:595 +0x64
code.vegaprotocol.io/vega/core/processor.NewApp.(*App).CheckBatchMarketInstructionsW.func51({0x104e8ac38, 0x14003c52690}, {0x104eb7fd0, 0x14007ea3400})
    /Users/caed/Projects/vega-market-sim/extern/vega/core/processor/abci.go:628 +0x78
code.vegaprotocol.io/vega/core/processor.NewApp.(*App).SendTransactionResult.func52({0x104e8ac38, 0x14003c52690}, {0x104eb7fd0?, 0x14007ea3400?})
    /Users/caed/Projects/vega-market-sim/extern/vega/core/processor/abci.go:658 +0x4c
code.vegaprotocol.io/vega/core/blockchain/abci.(*App).FinalizeBlock(0x140006c8600, {0x104c23320?, 0x1?}, 0x140019b6a80)
    /Users/caed/Projects/vega-market-sim/extern/vega/core/blockchain/abci/abci.go:179 +0x490
code.vegaprotocol.io/vega/cmd/vega/commands/node.(*appW).FinalizeBlock(0x1400c3fe930?, {0x104e8ac00?, 0x107607e60?}, 0x1400d60d100?)
    /Users/caed/Projects/vega-market-sim/extern/vega/cmd/vega/commands/node/app_wrapper.go:91 +0x30
code.vegaprotocol.io/vega/core/blockchain/nullchain.(*Replayer).replayChain(0x14000aa31d0, 0x1)
    /Users/caed/Projects/vega-market-sim/extern/vega/core/blockchain/nullchain/replay.go:160 +0x534
code.vegaprotocol.io/vega/core/blockchain/nullchain.(*NullBlockchain).StartChain(0x1400089aea0)
    /Users/caed/Projects/vega-market-sim/extern/vega/core/blockchain/nullchain/nullchain.go:165 +0x500
code.vegaprotocol.io/vega/core/blockchain/nullchain.(*NullBlockchain).StartServer(0x1400089aea0)
    /Users/caed/Projects/vega-market-sim/extern/vega/core/blockchain/nullchain/server.go:72 +0x34c
code.vegaprotocol.io/vega/cmd/vega/commands/node.(*Command).Run.func1()
    /Users/caed/Projects/vega-market-sim/extern/vega/cmd/vega/commands/node/node.go:175 +0xd4
created by code.vegaprotocol.io/vega/cmd/vega/commands/node.(*Command).Run in goroutine 1
    /Users/caed/Projects/vega-market-sim/extern/vega/cmd/vega/commands/node/node.go:160 +0x60c
cdummett commented 2 weeks ago

Reopening as overnight panic against develop after fix merged.

https://jenkins.vega.rocks/job/common/job/vega-market-sim-fuzzing-develop/1/artifact/network_home/vega-sim-cyv4u8rq/

2024-08-29T00:30:47.627Z    INFO    core    node/node.go:99 starting vega   {"version": "v0.78.0-dev", "commit-hash": "001ed76cbd72d564c438bad3e5b81bb9ccd21109"}
panic: re submit special order would cross

goroutine 17444 [running]:
go.uber.org/zap/zapcore.CheckWriteAction.OnWrite(0x2?, 0x2?, {0x0?, 0x0?, 0xc005c01000?})
    /jenkins/GOPATH/pkg/mod/go.uber.org/zap@v1.25.0/zapcore/entry.go:196 +0x54
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc005bd00d0, {0xc005b79200, 0x2, 0x2})
    /jenkins/GOPATH/pkg/mod/go.uber.org/zap@v1.25.0/zapcore/entry.go:262 +0x3ec
go.uber.org/zap.(*Logger).Panic(0x16636fb7b2560000?, {0x40af709?, 0x0?}, {0xc005b79200, 0x2, 0x2})
    /jenkins/GOPATH/pkg/mod/go.uber.org/zap@v1.25.0/logger.go:258 +0x51
code.vegaprotocol.io/vega/core/matching.(*OrderBook).ReSubmitSpecialOrders(0xc003c5d360, 0xc00679ad00)
    /jenkins/workspace/common/vega-market-sim-fuzzing-develop/extern/vega/core/matching/orderbook.go:948 +0x9b6
code.vegaprotocol.io/vega/core/execution/future.(*Market).reSubmitPeggedOrders(0xc003e6c700, {0x486f098, 0xc0070760f0}, {0xc002aec800, 0x4d, 0x0?})
    /jenkins/workspace/common/vega-market-sim-fuzzing-develop/extern/vega/core/execution/future/special_orders.go:112 +0xf3
code.vegaprotocol.io/vega/core/execution/future.(*Market).repriceAllSpecialOrders(0xc003e6c700, {0x486f098, 0xc0070760f0}, 0x40?, {0xc001b34300?, 0xc005b4be90?, 0xc003e5a9c0?})
    /jenkins/workspace/common/vega-market-sim-fuzzing-develop/extern/vega/core/execution/future/special_orders.go:151 +0x117
code.vegaprotocol.io/vega/core/execution/future.(*Market).checkForReferenceMoves(0xc003e6c700, {0x486f098, 0xc0070760f0}, {0xc001b34300, 0x20, 0x30}, 0x1)
    /jenkins/workspace/common/vega-market-sim-fuzzing-develop/extern/vega/core/execution/future/reference_price_moves.go:54 +0x2b1
code.vegaprotocol.io/vega/core/execution/future.(*Market).leaveAuction(0xc003e6c700, {0x486f098?, 0xc0070760f0}, {0xc005bc60c0?, 0x1?, 0x0?})
    /jenkins/workspace/common/vega-market-sim-fuzzing-develop/extern/vega/core/execution/future/market.go:1825 +0x1314
code.vegaprotocol.io/vega/core/execution/future.(*Market).checkAuction(0xc003e6c700, {0x486f098?, 0xc0070760f0}, {0x0?, 0xedbec1197?, 0x0?}, {0x483dcc0?, 0xc00cbdcdb0})
    /jenkins/workspace/common/vega-market-sim-fuzzing-develop/extern/vega/core/execution/future/auction.go:105 +0xb79
code.vegaprotocol.io/vega/core/execution/future.(*Market).OnTick(0xc003e6c700, {0x486f098, 0xc0070760f0}, {0x40?, 0xc005bd3a20?, 0x0?})
    /jenkins/workspace/common/vega-market-sim-fuzzing-develop/extern/vega/core/execution/future/market.go:1091 +0x6d8
code.vegaprotocol.io/vega/core/execution.(*Engine).OnTick(0xc0009dfc00, {0x486f098, 0xc0070760f0}, {0x1?, 0x488eec0?, 0x0?})
    /jenkins/workspace/common/vega-market-sim-fuzzing-develop/extern/vega/core/execution/engine.go:1223 +0x4f1
code.vegaprotocol.io/vega/core/vegatime.(*Svc).notify(...)
    /jenkins/workspace/common/vega-market-sim-fuzzing-develop/extern/vega/core/vegatime/service.go:111
code.vegaprotocol.io/vega/core/vegatime.(*Svc).SetTimeNow(0xc000892900, {0x486f098, 0xc0070760f0}, {0x40?, 0x6e29a20?, 0x0?})
    /jenkins/workspace/common/vega-market-sim-fuzzing-develop/extern/vega/core/vegatime/service.go:80 +0x263
code.vegaprotocol.io/vega/core/processor.(*App).OnBeginBlock(0xc0006a9500, 0x346, {0xc005baea00, 0x40}, {0xc004b33b00?, 0x453c69?, 0x0?}, {0x0, 0x0}, {0x6e29a20, ...})
    /jenkins/workspace/common/vega-market-sim-fuzzing-develop/extern/vega/core/processor/abci.go:1347 +0x976
code.vegaprotocol.io/vega/core/blockchain/abci.(*App).FinalizeBlock(0xc00071ea80, {0xb8?, 0x3cbc860?}, 0xc005921140)
    /jenkins/workspace/common/vega-market-sim-fuzzing-develop/extern/vega/core/blockchain/abci/abci.go:168 +0x449
code.vegaprotocol.io/vega/cmd/vega/commands/node.(*appW).FinalizeBlock(0xc00e140960?, {0x486f060?, 0x6e29a20?}, 0x3b9aca00?)
    /jenkins/workspace/common/vega-market-sim-fuzzing-develop/extern/vega/cmd/vega/commands/node/app_wrapper.go:91 +0x25
code.vegaprotocol.io/vega/core/blockchain/nullchain.(*NullBlockchain).processBlock(0xc0008dc000)
    /jenkins/workspace/common/vega-market-sim-fuzzing-develop/extern/vega/core/blockchain/nullchain/nullchain.go:212 +0x556
code.vegaprotocol.io/vega/core/blockchain/nullchain.(*NullBlockchain).ForwardTime(0xc0008dc000, 0x46c7cfe00)
    /jenkins/workspace/common/vega-market-sim-fuzzing-develop/extern/vega/core/blockchain/nullchain/nullchain.go:289 +0x1ce
code.vegaprotocol.io/vega/core/blockchain/nullchain.(*NullBlockchain).handleForwardTime.func1()
    /jenkins/workspace/common/vega-market-sim-fuzzing-develop/extern/vega/core/blockchain/nullchain/server.go:132 +0x25
created by code.vegaprotocol.io/vega/core/blockchain/nullchain.(*NullBlockchain).handleForwardTime in goroutine 17229
    /jenkins/workspace/common/vega-market-sim-fuzzing-develop/extern/vega/core/blockchain/nullchain/server.go:131 +0x22f
{
  "genesis_time": "2023-05-09T10:00:00.000000000Z",
  "chain_id": "CUSTOM",
  "initial_height": "0",
  "consensus_params": {
    "block": {
      "max_bytes": "22020096",
      "max_gas": "-1",
      "time_iota_ms": "1"
    },
    "evidence": {
      "max_age_num_blocks": "100000",
      "max_age_duration": "172800000000000",
      "max_bytes": "1048576"
    },
    "validator": {
      "pub_key_types": ["ed25519"]
    },
    "version": {}
  },
  "validators": [
    {
      "address": "7DFD3744A773C166B36F170E4F5EF3E998703F4E",
      "pub_key": {
        "type": "tendermint/PubKeyEd25519",
        "value": "2lPO6I0p5uCBZzeoA2+ECJ2+57LtLVMjXpxH5za9TvE="
      },
      "power": "10",
      "name": ""
    }
  ],
  "app_hash": "",
  "app_state": {
    "assets": {
      "0000000000000000000000000000000000000000000000000000000000000000": {
        "name": "VOTE",
        "symbol": "VOTE",
        "decimals": 5,
        "quantum": "1",
        "min_lp_stake": "1",
        "source": {
          "builtin_asset": {
            "max_faucet_amount_mint": "100000000000000000"
          }
        }
      }
    },
    "validators": {
      "6NemqXSfN4zfZvAQeIANflEFqQ98izX9C0KLI2dITSY=": {
        "id": "c2b0ab69fb1bd19862c04df401920ea62d3e0bce41a50148240f42017824d17a",
        "vega_pub_key": "dfcf0fa8d67cfa11b3a511e320f3c47807a5e0735cc06dd9c2bc00f7ccc67117",
        "vega_pub_key_index": 1,
        "ethereum_address": "0x9e5BEEC6E56B28cCbd02864840B0f1e0125e42Ce",
        "tm_pub_key": "6NemqXSfN4zfZvAQeIANflEFqQ98izX9C0KLI2dITSY=",
        "info_url": "",
        "country": "",
        "name": "",
        "avatar_url": ""
      }
    },
    "network": {
      "replay_attack_threshold": 150
    },
    "network_parameters": {
      "blockchains.ethereumConfig": "{\"network_id\": \"3\", \"chain_id\": \"3\", \"collateral_bridge_contract\": { \"address\": \"0xa6F1E140daC13002Dfd9789D6dBA59117c717D7a\" }, \"confirmations\": 50, \"staking_bridge_contract\": { \"address\": \"0xfce2CC92203A266a9C8e67461ae5067c78f67235\", \"deployment_block_height\": 11001702}, \"multisig_control_contract\": {\"address\": \"0xCF6d41235911184fe6F35D47207813bFF3B91601\", \"deployment_block_height\": 12710009 } }",
      "blockchains.evmBridgeConfigs": "{\"configs\":[{\"network_id\": \"3\", \"chain_id\": \"3\", \"collateral_bridge_contract\": { \"address\": \"0xa6F1E140daC13002Dfd9789D6dBA59117c717D7a\" }, \"confirmations\": 50, \"multisig_control_contract\": {\"address\": \"0xCF6d41235911184fe6F35D47207813bFF3B91601\", \"deployment_block_height\": 12710009 } }]}",
      "blockchains.ethereumRpcAndEvmCompatDataSourcesConfig": "{\"configs\": []}",
      "governance.proposal.asset.maxClose": "8760h0m0s",
      "governance.proposal.asset.maxEnact": "8760h0m0s",
      "governance.proposal.asset.minClose": "1s",
      "governance.proposal.asset.minEnact": "1s",
      "governance.proposal.asset.minProposerBalance": "1",
      "governance.proposal.asset.minVoterBalance": "1",
      "governance.proposal.asset.requiredMajority": "0.66",
      "governance.proposal.asset.requiredParticipation": "0.00001",
      "governance.proposal.freeform.maxClose": "8760h0m0s",
      "governance.proposal.freeform.minClose": "1s",
      "governance.proposal.freeform.minProposerBalance": "1",
      "governance.proposal.freeform.minVoterBalance": "1",
      "governance.proposal.freeform.requiredMajority": "0.66",
      "governance.proposal.freeform.requiredParticipation": "0.00001",
      "governance.proposal.market.maxClose": "8760h0m0s",
      "governance.proposal.market.maxEnact": "8760h0m0s",
      "governance.proposal.market.minClose": "1s",
      "governance.proposal.market.minEnact": "2s",
      "governance.proposal.market.minProposerBalance": "1",
      "governance.proposal.market.minVoterBalance": "1",
      "governance.proposal.market.requiredMajority": "0.66",
      "governance.proposal.market.requiredParticipation": "0.00001",
      "governance.proposal.updateAsset.maxClose": "720h",
      "governance.proposal.updateAsset.maxEnact": "720h",
      "governance.proposal.updateAsset.minClose": "1s",
      "governance.proposal.updateAsset.minEnact": "1s",
      "governance.proposal.updateAsset.minProposerBalance": "1",
      "governance.proposal.updateAsset.minVoterBalance": "1",
      "governance.proposal.updateAsset.requiredMajority": "0.66",
      "governance.proposal.updateAsset.requiredParticipation": "0.09",
      "governance.proposal.updateMarket.maxClose": "8760h0m0s",
      "governance.proposal.updateMarket.maxEnact": "8760h0m0s",
      "governance.proposal.updateMarket.minClose": "1s",
      "governance.proposal.updateMarket.minEnact": "1s",
      "governance.proposal.updateMarket.minProposerBalance": "1",
      "governance.proposal.updateMarket.minProposerEquityLikeShare": "0",
      "governance.proposal.updateMarket.minVoterBalance": "1",
      "governance.proposal.updateMarket.requiredMajority": "0.66",
      "governance.proposal.updateMarket.requiredMajorityLP": "0.66",
      "governance.proposal.updateMarket.requiredParticipation": "0.00001",
      "governance.proposal.updateMarket.requiredParticipationLP": "0",
      "governance.proposal.updateNetParam.maxClose": "8760h0m0s",
      "governance.proposal.updateNetParam.maxEnact": "8760h0m0s",
      "governance.proposal.updateNetParam.minClose": "1s",
      "governance.proposal.updateNetParam.minEnact": "1s",
      "governance.proposal.updateNetParam.minProposerBalance": "1",
      "governance.proposal.updateNetParam.minVoterBalance": "1",
      "governance.proposal.updateNetParam.requiredMajority": "0.5",
      "governance.proposal.updateNetParam.requiredParticipation": "0.00001",
      "governance.proposal.referralProgram.maxClose": "8760h0m0s",
      "governance.proposal.referralProgram.maxEnact": "8760h0m0s",
      "governance.proposal.referralProgram.minClose": "1s",
      "governance.proposal.referralProgram.minEnact": "1s",
      "governance.proposal.referralProgram.minProposerBalance": "1",
      "governance.proposal.referralProgram.minVoterBalance": "1",
      "governance.proposal.referralProgram.requiredMajority": "0.5",
      "governance.proposal.referralProgram.requiredParticipation": "0.00001",
      "governance.proposal.VolumeDiscountProgram.maxClose": "8760h0m0s",
      "governance.proposal.VolumeDiscountProgram.maxEnact": "8760h0m0s",
      "governance.proposal.VolumeDiscountProgram.minClose": "1s",
      "governance.proposal.VolumeDiscountProgram.minEnact": "1s",
      "governance.proposal.VolumeDiscountProgram.minProposerBalance": "1",
      "governance.proposal.VolumeDiscountProgram.minVoterBalance": "1",
      "governance.proposal.VolumeDiscountProgram.requiredMajority": "0.5",
      "governance.proposal.VolumeDiscountProgram.requiredParticipation": "0.00001",
      "governance.proposal.VolumeRebateProgram.maxClose": "8760h0m0s",
      "governance.proposal.VolumeRebateProgram.maxEnact": "8760h0m0s",
      "governance.proposal.VolumeRebateProgram.minClose": "1s",
      "governance.proposal.VolumeRebateProgram.minEnact": "1s",
      "governance.proposal.VolumeRebateProgram.minProposerBalance": "1",
      "governance.proposal.VolumeRebateProgram.minVoterBalance": "1",
      "governance.proposal.VolumeRebateProgram.requiredMajority": "0.5",
      "governance.proposal.VolumeRebateProgram.requiredParticipation": "0.00001",
      "governance.proposal.transfer.maxClose": "8760h0m0s",
      "governance.proposal.transfer.maxEnact": "8760h0m0s",
      "governance.proposal.transfer.minClose": "1s",
      "governance.proposal.transfer.minEnact": "1s",
      "governance.proposal.transfer.minProposerBalance": "1",
      "governance.proposal.transfer.minVoterBalance": "1",
      "governance.proposal.transfer.requiredMajority": "0.5",
      "governance.proposal.transfer.requiredParticipation": "0.00001",
      "limits.markets.maxPeggedOrders": "1500",
      "limits.markets.proposePerpetualEnabled": "1",
      "limits.markets.ammPoolEnabled": "1",
      "market.auction.maximumDuration": "168h",
      "market.auction.minimumDuration": "1s",
      "market.fee.factors.infrastructureFee": "0.0005",
      "market.fee.factors.makerFee": "0.0002",
      "market.fee.factors.treasuryFee": "0.0001",
      "market.fee.factors.buybackFee": "0.0001",
      "market.liquidity.bondPenaltyParameter": "0.1",
      "market.liquidity.earlyExitPenalty": "0.1",
      "market.liquidity.maximumLiquidityFeeFactorLevel": "0.03",
      "market.liquidity.minimum.probabilityOfTrading.lpOrders": "1e-6",
      "market.liquidity.probabilityOfTrading.tau.scaling": "10.0",
      "market.liquidity.providersFeeCalculationTimeStep": "1s",
      "market.liquidity.sla.nonPerformanceBondPenaltyMax": "0.5",
      "market.liquidity.sla.nonPerformanceBondPenaltySlope": "2",
      "market.liquidity.stakeToCcyVolume": "1.0",
      "market.liquidity.targetstake.triggering.ratio": "0.25",
      "market.liquidityProvision.minLpStakeQuantumMultiple": "5000",
      "market.liquidityProvision.shapes.maxSize": "100",
      "market.margin.scalingFactors": "{\"search_level\": 1.1, \"initial_margin\": 1.5, \"collateral_release\": 1.7}",
      "market.monitor.price.defaultParameters": "{\"triggers\": [{\"auction_extension\": 300, \"horizon\": 43200, \"probability\": \"0.9999999\"}] }",
      "market.stake.target.scalingFactor": "0.0001",
      "market.stake.target.timeWindow": "1h",
      "market.value.windowLength": "2h0m0s",
      "network.checkpoint.timeElapsedBetweenCheckpoints": "5m",
      "network.floatingPointUpdates.delay": "5m",
      "network.markPriceUpdateMaximumFrequency": "5s",
      "network.transaction.defaultgas": "1",
      "network.transactions.maxgasperblock": "3000000",
      "network.transactions.minBlockCapacity": "32",
      "network.validators.ersatz.multipleOfTendermintValidators": "0",
      "network.validators.ersatz.rewardFactor": "0.85",
      "network.validators.incumbentBonus": "0.05",
      "network.validators.minimumEthereumEventsForNewValidator": "3",
      "network.validators.multisig.numberOfSigners": "13",
      "network.validators.tendermint.number": "13",
      "reward.asset": "0000000000000000000000000000000000000000000000000000000000000000",
      "reward.staking.delegation.competitionLevel": "1.3",
      "reward.staking.delegation.delegatorShare": "0.8",
      "reward.staking.delegation.maxPayoutPerEpoch": "9890000000000000000000",
      "reward.staking.delegation.maxPayoutPerParticipant": "700000000000000000000",
      "reward.staking.delegation.minimumValidatorStake": "3000000000000000000000",
      "reward.staking.delegation.minValidators": "5",
      "reward.staking.delegation.optimalStakeMultiplier": "3.0",
      "reward.staking.delegation.payoutDelay": "0h",
      "reward.staking.delegation.payoutFraction": "1.0",
      "rewards.marketCreationQuantumMultiple": "1",
      "snapshot.interval.length": "300",
      "spam.pow.difficulty": "15",
      "spam.pow.hashFunction": "sha3_24_rounds",
      "spam.pow.increaseDifficulty": "0",
      "spam.pow.numberOfPastBlocks": "100",
      "spam.pow.numberOfTxPerBlock": "1000",
      "spam.protection.delegation.min.tokens": "100000000000000000",
      "spam.protection.max.batchSize": "30",
      "spam.protection.max.delegations": "360",
      "spam.protection.max.proposals": "300",
      "spam.protection.max.votes": "300",
      "spam.protection.maxUserTransfersPerEpoch": "10000",
      "spam.protection.minimumWithdrawalQuantumMultiple": "10",
      "spam.protection.minMultisigUpdates": "100000000000000000000",
      "spam.protection.proposal.min.tokens": "2000000000000000000000",
      "spam.protection.voting.min.tokens": "1000000000000000000",
      "spam.protection.applyReferral.min.funds": "0",
      "transfer.fee.factor": "0.001",
      "transfer.minTransferQuantumMultiple": "0",
      "transfer.fee.maxQuantumAmount": "100",
      "transfer.feeDiscountMinimumTrackedAmount": "0.001",
      "transfer.feeDiscountDecayFraction": "0.5",
      "validator.performance.scaling.factor": "0",
      "validators.delegation.minAmount": "1",
      "validators.epoch.length": "2m",
      "validators.vote.required": "0.67",
      "referralProgram.maxReferralTiers": "10",
      "referralProgram.maxReferralRewardFactor": "1",
      "referralProgram.maxReferralDiscountFactor": "1",
      "referralProgram.maxPartyNotionalVolumeByQuantumPerEpoch": "10000",
      "referralProgram.minStakedVegaTokens": "0",
      "referralProgram.maxReferralRewardProportion": "1",
      "market.aggressiveOrderBlockDelay": "1",
      "market.amm.minCommitmentQuantum": "1",
      "market.liquidity.maxAmmCalculationLevels": "100"
    },
    "network_limits": {
      "propose_market_enabled": true,
      "propose_asset_enabled": true,
      "bootstrap_block_count": 0
    },
    "checkpoint": {
      "load_hash": ""
    }
  }
}