search

vegaprotocol / vega

A Go implementation of the Vega Protocol, a protocol for creating and trading derivatives on a fully decentralised network.
https://vega.xyz
GNU Affero General Public License v3.0
36 stars 22 forks source link

SLA panic "decimal division by 0", "stack": "goroutine 340 #8993

Closed vega-paul closed 1 year ago

vega-paul commented 1 year ago

Vega branch origin/feature/integrate-sla-rebased-2 ST branch sla-future-market-proposal

Proposal

{"id":"37895-75","block":"476DF7628D55AFE3BB530304FB991233671C58F951D101034003BE1F22CBD8B3","type":"BUS_EVENT_TYPE_PROPOSAL","proposal":{"id":"ae2d74c7f332ec5755df73249a28f958da0e196222805b6b0edb042a9b4069e2","reference":"7D14472C2A","partyId":"e402221020c71c04e6bc6c4d6c0e188ed261ca4759ef38049f1d45b416e812d9","state":"STATE_ENACTED","timestamp":"1691163453115247000","terms":{"closingTimestamp":"1691163457","enactmentTimestamp":"1691163458","newMarket":{"changes":{"instrument":{"name":"Futures System-Tests Market","code":"CRYPTO:VEGA CORE SYS TEST","future":{"settlementAsset":"24a6f67ab6f7ba468dc363236460c9d05dcf09b2b11da1f9c46e90890499bf2e","quoteName":"USD","dataSourceSpecForSettlementData":{"external":{"oracle":{"signers":[{"pubKey":{"key":"e402221020c71c04e6bc6c4d6c0e188ed261ca4759ef38049f1d45b416e812d9"}}],"filters":[{"key":{"name":"oracle.test.settlementFD93D","type":"TYPE_INTEGER","numberDecimalPlaces":"5"},"conditions":[{"operator":"OPERATOR_GREATER_THAN_OR_EQUAL","value":"0"}]}]}}},"dataSourceSpecForTradingTermination":{"external":{"oracle":{"signers":[{"pubKey":{"key":"e402221020c71c04e6bc6c4d6c0e188ed261ca4759ef38049f1d45b416e812d9"}}],"filters":[{"key":{"name":"oracle.test.termination716EB","type":"TYPE_BOOLEAN"},"conditions":[{"operator":"OPERATOR_EQUALS","value":"True"}]}]}}},"dataSourceSpecBinding":{"settlementDataProperty":"oracle.test.settlementFD93D","tradingTerminationProperty":"oracle.test.termination716EB"}}},"decimalPlaces":"5","metadata":["asset_class:fx/crypto","product:futures"],"priceMonitoringParameters":{},"liquidityMonitoringParameters":{"targetStakeParameters":{"timeWindow":"10","scalingFactor":5},"triggeringRatio":"0","auctionExtension":"1"},"logNormal":{"riskAversionParameter":0.001,"tau":0.00011407711613050422,"params":{"r":0.016,"sigma":1.5}},"positionDecimalPlaces":"5","linearSlippageFactor":"0.01","quadraticSlippageFactor":"0","liquiditySlaParameters":{"priceRange":"1","commitmentMinTimeFraction":"1","providersFeeCalculationTimeStep":"1","performanceHysteresisEpochs":"1","slaCompetitionFactor":"1"}}}},"rationale":{"description":"Proposal submitted to enact a new Market","title":"Market Proposal"},"requiredParticipation":"0.00000015","requiredMajority":"0.66"},"version":1,"chainId":"testnet-001","txHash":"476DF7628D55AFE3BB530304FB991233671C58F951D101034003BE1F22CBD8B3"}

LP - did not submit any LP orders

{"id":"37897-76","block":"91EE86F698550B70C466EFF6FED51F4C0F055AC640FDC86EC6C932E803FB9C35","type":"BUS_EVENT_TYPE_LIQUIDITY_PROVISION","liquidityProvision":{"id":"f40b99b7f4ffd40d8d0ab5efcca21af283c8e203fe87fd667967d44d5c365cd7","partyId":"e402221020c71c04e6bc6c4d6c0e188ed261ca4759ef38049f1d45b416e812d9","createdAt":"1691163460152438000","updatedAt":"1691163460152438000","marketId":"ae2d74c7f332ec5755df73249a28f958da0e196222805b6b0edb042a9b4069e2","commitmentAmount":"390500000000000000000000","fee":"0.3","version":"1","status":"STATUS_ACTIVE","reference":"57A5464751F34BF"},"version":1,"chainId":"testnet-001","txHash":"9874A3D8A6AF3426236278DBCDC2C3DC04B7B3AFDAC176B384DD04A998654C55"}

Created traders/pubkeys/deposits

{"id":"37924-71","block":"50F5441320C2CE7A9E8BA597E6621D16F610FFF3D90CF652C4E0506912D7047D","type":"BUS_EVENT_TYPE_DEPOSIT","deposit":{"id":"43daffaaa412a7182c7c3c5b630a47a5b29fb16dd712a496586818fdcf9026b3","status":"STATUS_OPEN","partyId":"4ed84aeb4d79ed7bc61f6b206195ee8b2e4044f9bf05862421045336692f60aa","asset":"24a6f67ab6f7ba468dc363236460c9d05dcf09b2b11da1f9c46e90890499bf2e","amount":"100000000000000000000000000","createdTimestamp":"1691163474796561000"},"version":1,"chainId":"testnet-001","txHash":"C1F628E83BA794E7CE88C2C3A10103A066150470609562FA24E7C0D834EBE517"}

Panic

2023-08-04T16:37:56.282+0100    INFO    tendermint      txindex/indexer_service.go:97   indexed block exents    {"height": 37925}
2023-08-04T16:37:56.452+0100    INFO    tendermint      consensus/state.go:1850 received proposal       {"proposal": "Proposal{37926/0 (AE5DEE5C35A4DCF47F41757F2FF24043539945690E4B8E5EA6FAB49DDC607E42:1:1096B7EDD62F, -1) FE85EEC5048B @ 2023-08-04T15:37:56.391319Z}"}
2023-08-04T16:37:56.452+0100    INFO    tendermint      consensus/state.go:1917 received complete proposal block        {"height": 37926, "hash": "AE5DEE5C35A4DCF47F41757F2FF24043539945690E4B8E5EA6FAB49DDC607E42"}
2023-08-04T16:37:56.657+0100    INFO    tendermint      consensus/state.go:1604 finalizing commit of block      {"hash": "AE5DEE5C35A4DCF47F41757F2FF24043539945690E4B8E5EA6FAB49DDC607E42", "root": "83B8B00E494B63D4F73DB03B95AE599A4B87EECDCDC3156A56BC42382DF90779", "num_txs": 1}
2023-08-04T16:37:56.666+0100    INFO    tendermint      state/execution.go:333  executed block  {"height": 37926, "num_valid_txs": 1, "num_invalid_txs": 0}
2023-08-04T16:37:56.671+0100    INFO    tendermint      state/execution.go:235  committed state {"height": 37926, "num_txs": 1, "app_hash": "83B8B00E494B63D4F73DB03B95AE599A4B87EECDCDC3156A56BC42382DF90779"}
2023-08-04T16:37:56.675+0100    INFO    tendermint      consensus/ticker.go:124 Timed out       {"dur": "-18.352ms", "height": 37927, "round": 0, "step": "RoundStepNewHeight"}
2023-08-04T16:37:56.679+0100    INFO    tendermint      txindex/indexer_service.go:97   indexed block exents    {"height": 37926}
2023-08-04T16:37:56.691+0100    INFO    tendermint      consensus/state.go:1850 received proposal       {"proposal": "Proposal{37927/0 (1628CA9F45A02946FBF9ADC572C6C43D555765B0F24D56F813270E454D3D8167:1:A832301D3F3C, -1) C4D913D5C5FC @ 2023-08-04T15:37:56.683278Z}"}
2023-08-04T16:37:56.695+0100    INFO    tendermint      consensus/state.go:1917 received complete proposal block        {"height": 37927, "hash": "1628CA9F45A02946FBF9ADC572C6C43D555765B0F24D56F813270E454D3D8167"} 
2023-08-04T16:37:56.833+0100    INFO    core.protocol   ethcall/engine.go:220   tick    {"wallTime": "2023-08-04T16:37:56.831+0100", "ethBlock": "50959", "ethTime": "2023-08-04T16:37:56.000+0100"}  
2023-08-04T16:37:56.866+0100    INFO    tendermint      consensus/state.go:1604 finalizing commit of block      {"hash": "1628CA9F45A02946FBF9ADC572C6C43D555765B0F24D56F813270E454D3D8167", "root": "83B8B00E494B63D4F73DB03B95AE599A4B87EECDCDC3156A56BC42382DF90779", "num_txs": 1}

2023-08-04T16:37:56.977+0100    ERROR   tendermint  consensus/state.go:732  CONSENSUS FAILURE!!!    {"err": "decimal division by 0", "stack": "goroutine 340 [running]:\nruntime/debug.Stack()
/opt/homebrew/Cellar/go@1.19/1.19.10/libexec/src/runtime/debug/stack.go:24 +0x64\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine.func2()
/Users/paulwebb/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/consensus/state.go:732 +0x44\npanic({0x1073723e0, 0x107b65dd0})
/opt/homebrew/Cellar/go@1.19/1.19.10/libexec/src/runtime/panic.go:884 +0x204\ngithub.com/shopspring/decimal.Decimal.QuoRem({0x14005ca5c00, 0xfffffff0}, {0x14005ca5c20, 0x0}, 0x10)
/Users/paulwebb/go/pkg/mod/github.com/vegaprotocol/decimal@v1.3.1-uint256/decimal.go:590 +0x23c\ngithub.com/shopspring/decimal.Decimal.DivRound({0x14005ca5c00?, 0xdc5f1254?}, {0x14005ca5c20?, 0x50c4514?}, 0x10)
/Users/paulwebb/go/pkg/mod/github.com/vegaprotocol/decimal@v1.3.1-uint256/decimal.go:632 +0x3c\ngithub.com/shopspring/decimal.Decimal.Div(...)
/Users/paulwebb/go/pkg/mod/github.com/vegaprotocol/decimal@v1.3.1-uint256/decimal.go:577\ncode.vegaprotocol.io/vega/core/liquidity/v2.(*Engine).calculateCurrentFeePenalty(0x14004042fc0, {0x14005ca5b80?, 0x4bf9140?})
/Users/paulwebb/repos/vega/core/liquidity/v2/sla.go:165 +0xa0\ncode.vegaprotocol.io/vega/core/liquidity/v2.(*Engine).CalculateSLAPenalties(0x14004042fc0, {0x14000f2e4d0?, 0x1b749468?, 0x0?})
/Users/paulwebb/repos/vega/core/liquidity/v2/sla.go:82 +0x2d0\ncode.vegaprotocol.io/vega/core/execution/common.(*MarketLiquidity).calculateAndDistribute(0x14004043200, {0x107ba7b50, 0x140059d82a0}, {0x105224098?, 0x140005c4e48?, 0x0?})
/Users/paulwebb/repos/vega/core/execution/common/liquidity_provision.go:288 +0x48\ncode.vegaprotocol.io/vega/core/execution/common.(*MarketLiquidity).OnEpochEnd(...)
/Users/paulwebb/repos/vega/core/execution/common/liquidity_provision.go:280\ncode.vegaprotocol.io/vega/core/execution/future.(*Market).OnEpochEvent(0x140019f0480, {0x107ba7b50, 0x140059d82a0}, {0x1069, {0xf6a6558, 0xedc5f124e, 0x0}, {0xf6a6558, 0xedc5f1253, 0x0}, ...})
/Users/paulwebb/repos/vega/core/execution/future/market.go:336 +0xbc\ncode.vegaprotocol.io/vega/core/execution.(*Engine).OnEpochEvent(0x1400125cfc0?, {0x107ba7b50, 0x140059d82a0}, {0x1069, {0xf6a6558, 0xedc5f124e, 0x0}, {0xf6a6558, 0xedc5f1253, 0x0}, ...})
/Users/paulwebb/repos/vega/core/execution/engine.go:223 +0xdc\ncode.vegaprotocol.io/vega/core/epochtime.(*Svc).notify(0x14001594b40, {0x107ba7b50, 0x140059d82a0}, {0x1069, {0xf6a6558, 0xedc5f124e, 0x0}, {0xf6a6558, 0xedc5f1253, 0x0}, ...})
/Users/paulwebb/repos/vega/core/epochtime/service.go:201 +0xec\ncode.vegaprotocol.io/vega/core/epochtime.(*Svc).OnTick(0x14001594b40, {0x107ba7b50, 0x140059d82a0}, {0x1?, 0x107bc5fd8?, 0x0?})
/Users/paulwebb/repos/vega/core/epochtime/service.go:131 +0x568\ncode.vegaprotocol.io/vega/core/vegatime.(*Svc).notify(...)
/Users/paulwebb/repos/vega/core/vegatime/service.go:104\ncode.vegaprotocol.io/vega/core/vegatime.(*Svc).SetTimeNow(0x14000f2e480, {0x107ba7b50, 0x140059d82a0}, {0x40?, 0x107371460?, 0x0?})
/Users/paulwebb/repos/vega/core/vegatime/service.go:77 +0x278\ncode.vegaprotocol.io/vega/core/processor.(*App).OnBeginBlock(_, {{0x14005bbef80, 0x20, 0x20}, {{0xb, 0x1}, {0x140058b04d0, 0xb}, 0x9427, {0x1b749468, ...}, ...}, ...})
/Users/paulwebb/repos/vega/core/processor/abci.go:779 +0x554\ncode.vegaprotocol.io/vega/core/blockchain/abci.(*App).BeginBlock(_, {{0x14005bbef80, 0x20, 0x20}, {{0xb, 0x1}, {0x140058b04d0, 0xb}, 0x9427, {0x1b749468, ...}, ...}, ...})
/Users/paulwebb/repos/vega/core/blockchain/abci/abci.go:46 +0x60\ncode.vegaprotocol.io/vega/cmd/vega/commands/node.(*appW).BeginBlock(_, {{0x14005bbef80, 0x20, 0x20}, {{0xb, 0x1}, {0x140058b04d0, 0xb}, 0x9427, {0x1b749468, ...}, ...}, ...})
/Users/paulwebb/repos/vega/cmd/vega/commands/node/app_wrapper.go:64 +0x4c\ngithub.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync(_, {{0x14005bbef80, 0x20, 0x20}, {{0xb, 0x1}, {0x140058b04d0, 0xb}, 0x9427, {0x1b749468, ...}, ...}, ...})
/Users/paulwebb/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/abci/client/local_client.go:280 +0x108\ngithub.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(_, {{0x14005bbef80, 0x20, 0x20}, {{0xb, 0x1}, {0x140058b04d0, 0xb}, 0x9427, {0x1b749468, ...}, ...}, ...})
/Users/paulwebb/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/proxy/app_conn.go:81 +0x4c\ngithub.com/tendermint/tendermint/state.execBlockOnProxyApp({0x107ba5e70?, 0x14000010888}, {0x107bbc960, 0x1400180d4d0}, 0x140041df680, {0x107bcf118, 0x14000346288}, 0x9426?)
/Users/paulwebb/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/state/execution.go:307 +0x2b8\ngithub.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x1}, {0x1400166f100, 0x7}}, {0x1400166f110, 0xb}, 0x1, 0x9426, {{0x14005467f00, ...}, ...}, ...}, ...)
/Users/paulwebb/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/state/execution.go:140 +0xe4\ngithub.com/tendermint/tendermint/consensus.(*State).finalizeCommit(0x140016fa000, 0x9427)
/Users/paulwebb/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/consensus/state.go:1661 +0x94c\ngithub.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit(0x140016fa000, 0x9427)
/Users/paulwebb/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/consensus/state.go:1570 +0x288\ngithub.com/tendermint/tendermint/consensus.(*State).enterCommit.func1()
/Users/paulwebb/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/consensus/state.go:1505 +0xa0\ngithub.com/tendermint/tendermint/consensus.(*State).enterCommit(0x140016fa000, 0x9427, 0x0)
/Users/paulwebb/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/consensus/state.go:1543 +0xb8c\ngithub.com/tendermint/tendermint/consensus.(*State).addVote(0x140016fa000, 0x140043fa6e0, {0x14000326780, 0x28})
/Users/paulwebb/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/consensus/state.go:2165 +0x1678\ngithub.com/tendermint/tendermint/consensus.(*State).tryAddVote(0x140016fa000, 0x140043fa6e0, {0x14000326780?, 0x140063e4a90?})
/Users/paulwebb/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/consensus/state.go:1963 +0x28\ngithub.com/tendermint/tendermint/consensus.(*State).handleMsg(0x140016fa000, {{0x107b82a40?, 0x140046c6878?}, {0x14000326780?, 0x0?}})
/Users/paulwebb/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/consensus/state.go:861 +0x108\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0x140016fa000, 0x0)
/Users/paulwebb/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/consensus/state.go:768 +0x2bc\ncreated by github.com/tendermint/tendermint/consensus.(*State).OnStart
/Users/paulwebb/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/consensus/state.go:379 +0xf4\n"}
ze97286 commented 1 year ago

the problem is that we allow commitmentMinTimeFraction to be 1 and that leads to division by zero in the formula for calculating the penalty:

Let $t$ be fraction_of_time_on_book

Let $s$ be market.liquidity.commitmentMinTimeFraction.

Let $c$ be market.liquidity.slaCompetitionFactor.

$$ p_i = (1 - \frac{t - s}{1 - s}) \cdot c. $$

gordsport commented 1 year ago

spec issue - needs clarifying before we can fix

karlem commented 1 year ago

Fixed by #8996