search

vegaprotocol / vega

A Go implementation of the Vega Protocol, a protocol for creating and trading derivatives on a fully decentralised network.
https://vega.xyz
GNU Affero General Public License v3.0
37 stars 19 forks source link

Panic - updating oracle termination condition and value internal oracle spec #5329

Closed vega-paul closed 2 years ago

vega-paul commented 2 years ago

This submission on an enacted continuous market updates the oracle term condition to be 10s in the future. When we reach the time core throws a panic.

          conditions {
            operator: OPERATOR_EQUALS
            value: "1651598980"
          }
terms {
  update_market {
    market_id: "27587c64bcb66a0f7ec17b53159b7ad5861147f2aa28cf954b2b350112be85e4"
    changes {
      instrument {
        code: "CRYPTO:VEGA CORE SYS TEST"
        future {
          quote_name: "USD"
          oracle_spec_for_settlement_price {
            pub_keys: "c49d680fc28320abe5a0d36a0ab309cbe1a40bfaad2b0bf8d22adaad005cf1bd"
            filters {
              key {
                name: "oracle.test.settlement"
                type: TYPE_INTEGER
              }
              conditions {
                operator: OPERATOR_GREATER_THAN_OR_EQUAL
                value: "0"
              }
            }
          }
          oracle_spec_for_trading_termination {
            pub_keys: "c49d680fc28320abe5a0d36a0ab309cbe1a40bfaad2b0bf8d22adaad005cf1bd"
            filters {
              key {
                name: "vegaprotocol.builtin.timestamp"
                type: TYPE_TIMESTAMP
              }
              conditions {
                operator: OPERATOR_EQUALS
                value: "1651598980"
              }
            }
          }
          oracle_spec_binding {
            settlement_price_property: "oracle.test.settlement"
            trading_termination_property: "vegaprotocol.builtin.timestamp"
          }
          settlement_price_decimals: 5
        }
      }
      metadata: "asset_class:fx/crypto"
      metadata: "product:futures"
      log_normal {
        risk_aversion_parameter: 0.001
        tau: 0.00011407711613050422
        params {
          r: 0.016
          sigma: 1.5
        }
      }
    }
  }
}
rationale {
  description: "Update market"
}

Enacted market

{
  "id": "4725-33",
  "block": "F494941B6D7015F02EC59022B7D4A308A1975E79992ED226939728AA53BB2762",
  "type": "BUS_EVENT_TYPE_PROPOSAL",
  "proposal": {
    "id": "27587c64bcb66a0f7ec17b53159b7ad5861147f2aa28cf954b2b350112be85e4",
    "reference": "F94DE11BE1",
    "partyId": "c49d680fc28320abe5a0d36a0ab309cbe1a40bfaad2b0bf8d22adaad005cf1bd",
    "state": "STATE_ENACTED",
    "timestamp": "1651598932232997000",
    "terms": {
      "closingTimestamp": "1651598942",
      "enactmentTimestamp": "1651598944",
      "validationTimestamp": "1651598932",
      "newMarket": {
        "changes": {
          "instrument": {
            "name": "Futures System-Tests Market",
            "code": "CRYPTO:VEGA CORE SYS TEST",
            "future": {
              "settlementAsset": "b66b7c057f3b47508f8402cf8e1a7d34455e26b5f1eef98a318bd363ce5c0464",
              "quoteName": "USD",
              "oracleSpecForSettlementPrice": {
                "pubKeys": [
                  "c49d680fc28320abe5a0d36a0ab309cbe1a40bfaad2b0bf8d22adaad005cf1bd"
                ],
                "filters": [
                  {
                    "key": {
                      "name": "oracle.test.settlement",
                      "type": "TYPE_INTEGER"
                    },
                    "conditions": [
                      {
                        "operator": "OPERATOR_GREATER_THAN_OR_EQUAL",
                        "value": "0"
                      }
                    ]
                  }
                ]
              },
              "oracleSpecForTradingTermination": {
                "pubKeys": [
                  "c49d680fc28320abe5a0d36a0ab309cbe1a40bfaad2b0bf8d22adaad005cf1bd"
                ],
                "filters": [
                  {
                    "key": {
                      "name": "vegaprotocol.builtin.timestamp",
                      "type": "TYPE_TIMESTAMP"
                    },
                    "conditions": [
                      {
                        "operator": "OPERATOR_GREATER_THAN_OR_EQUAL",
                        "value": "1651599432"
                      }
                    ]
                  }
                ]
              },
              "oracleSpecBinding": {
                "settlementPriceProperty": "oracle.test.settlement",
                "tradingTerminationProperty": "vegaprotocol.builtin.timestamp"
              }
            }
          },
          "decimalPlaces": "5",
          "metadata": [
            "asset_class:fx/crypto",
            "product:futures"
          ],
          "priceMonitoringParameters": {},
          "liquidityMonitoringParameters": {
            "targetStakeParameters": {
              "timeWindow": "10",
              "scalingFactor": 5
            },
            "auctionExtension": "1"
          },
          "logNormal": {
            "riskAversionParameter": 0.001,
            "tau": 0.00011407711613050422,
            "params": {
              "r": 0.016,
              "sigma": 1.5
            }
          },
          "positionDecimalPlaces": "5"
        },
        "liquidityCommitment": {
          "commitmentAmount": "3905000000000000000000000",
          "fee": "0.3",
          "sells": [
            {
              "reference": "PEGGED_REFERENCE_BEST_ASK",
              "proportion": 13,
              "offset": "100000"
            }
          ],
          "buys": [
            {
              "reference": "PEGGED_REFERENCE_BEST_BID",
              "proportion": 2,
              "offset": "100000"
            }
          ],
          "reference": "2684B3671F"
        }
      }
    }
  },
  "version": 1,
  "chainId": "test-chain-9lCsDX"
}

Logs:

2022-05-03T15:48:13.498+0100    ERROR   tm.abci.socket-server   server/socket_server.go:142     Connection error [err recovered from panic: unable to start market opening auction
goroutine 118 [running]:
github.com/tendermint/tendermint/abci/server.(*SocketServer).handleRequests.func1()
        /Users/paulwebb/repos/vega/vendor/github.com/tendermint/tendermint/abci/server/socket_server.go:165 +0x85
panic({0x2296280, 0xc006b5b5b0})
        /usr/local/Cellar/go/1.18.1/libexec/src/runtime/panic.go:838 +0x207
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0002980c0, {0xc004773f00, 0x2, 0x2})
        /Users/paulwebb/repos/vega/vendor/go.uber.org/zap/zapcore/entry.go:232 +0x44c
go.uber.org/zap.(*Logger).Panic(0xc0000003c0?, {0x25c5457?, 0xc0026c2960?}, {0xc004773f00, 0x2, 0x2})
        /Users/paulwebb/repos/vega/vendor/go.uber.org/zap/logger.go:230 +0x59
code.vegaprotocol.io/vega/processor.(*App).onTick(0xc0001762c0, {0x299f498, 0xc0026c2960}, {0x29a6ea0?, 0x1?, 0x0?})
        /Users/paulwebb/repos/vega/processor/abci.go:1340 +0x369
code.vegaprotocol.io/vega/vegatime.(*Svc).notify(0xc000574000?, {0x299f498, 0xc0026c2960}, {0xc00007f201?, 0xc0000f8a60?, 0x0?})
        /Users/paulwebb/repos/vega/vegatime/service.go:84 +0x114
code.vegaprotocol.io/vega/vegatime.(*Svc).SetTimeNow(0xc0001658f0, {0x299f498, 0xc0026c2960}, {0x3510920?, 0x2295880?, 0x0?})
        /Users/paulwebb/repos/vega/vegatime/service.go:59 +0x227
code.vegaprotocol.io/vega/processor.(*App).OnBeginBlock(_, {{0xc00c0a9fc0, 0x20, 0x20}, {{0xb, 0x1}, {0xc004613038, 0x11}, 0x2211, {0x10d50178, ...}, ...}, ...})
        /Users/paulwebb/repos/vega/processor/abci.go:606 +0x5cb
code.vegaprotocol.io/vega/blockchain/abci.(*App).BeginBlock(_, {{0xc00c0a9fc0, 0x20, 0x20}, {{0xb, 0x1}, {0xc004613038, 0x11}, 0x2211, {0x10d50178, ...}, ...}, ...})
        /Users/paulwebb/repos/vega/blockchain/abci/abci.go:50 +0x75
code.vegaprotocol.io/vega/cmd/vega/node.(*appW).BeginBlock(_, {{0xc00c0a9fc0, 0x20, 0x20}, {{0xb, 0x1}, {0xc004613038, 0x11}, 0x2211, {0x10d50178, ...}, ...}, ...})
        /Users/paulwebb/repos/vega/cmd/vega/node/app_wrapper.go:60 +0x55
github.com/tendermint/tendermint/abci/server.(*SocketServer).handleRequest(0x2988ec0?, 0xc0006be4e0?, 0x299a6e0?)
        /Users/paulwebb/repos/vega/vendor/github.com/tendermint/tendermint/abci/server/socket_server.go:222 +0x975
github.com/tendermint/tendermint/abci/server.(*SocketServer).handleRequests(0xc00060cdc0, 0xc0007128a0, {0xa2d7008?, 0xc00073a008?}, 0xc0001c98b0?)
        /Users/paulwebb/repos/vega/vendor/github.com/tendermint/tendermint/abci/server/socket_server.go:189 +0x2a6
created by github.com/tendermint/tendermint/abci/server.(*SocketServer).acceptConnectionsRoutine
        /Users/paulwebb/repos/vega/vendor/github.com/tendermint/tendermint/abci/server/socket_server.go:127 +0x25f
vega-paul commented 2 years ago

this is the proposal:

{
  "proposal-id": "9c2ebbbba39c050df7ce3e8e2b3ac7efea542b1bf2087d4d3ebf47f2f8b616d7",
  "proposal-reference": "2AFA4D3D2F",
  "proposal-party": "6b5dd645c5fa7f56830ac4d49ccbfed5ac94b38a353607da5f5224f0bf680ed8",
  "proposal-terms": "closing_timestamp:1651588987 enactment_timestamp:1651588989 validation_timestamp:1651588977 new_market:{changes:{instrument:{name:\"Futures System-Tests Market\" code:\"CRYPTO:VEGA CORE SYS TEST\" future:{settlement_asset:\"d82b84a1cdc5e68433b31ec35fb68d982669b6104e0dffedca748748f744b31a\" quote_name:\"USD\" oracle_spec_for_settlement_price:{pub_keys:\"6b5dd645c5fa7f56830ac4d49ccbfed5ac94b38a353607da5f5224f0bf680ed8\" filters:{key:{name:\"oracle.test.settlement\" type:TYPE_INTEGER} conditions:{operator:OPERATOR_GREATER_THAN_OR_EQUAL value:\"0\"}}} oracle_spec_for_trading_termination:{pub_keys:\"6b5dd645c5fa7f56830ac4d49ccbfed5ac94b38a353607da5f5224f0bf680ed8\" filters:{key:{name:\"vegaprotocol.builtin.timestamp\" type:TYPE_TIMESTAMP} conditions:{operator:OPERATOR_GREATER_THAN_OR_EQUAL value:\"1651587958\"}}} oracle_spec_binding:{settlement_price_property:\"oracle.test.settlement\" trading_termination_property:\"vegaprotocol.builtin.timestamp\"} settlement_price_decimals:5}} decimal_places:5 metadata:\"asset_class:fx/crypto\" metadata:\"product:futures\" log_normal:{risk_aversion_parameter:0.001 tau:0.00011407711613050422 params:{r:0.016 sigma:1.5}} position_decimal_places:5} liquidity_commitment:{commitment_amount:\"3905000000000000000000000\" fee:\"0.3\" sells:{reference:PEGGED_REFERENCE_BEST_ASK proportion:13 offset:\"100000\"} buys:{reference:PEGGED_REFERENCE_BEST_BID proportion:2 offset:\"100000\"} reference:\"B52E111497\"}}"
}
vega-paul commented 2 years ago

logs

https://vegaprotocol.slack.com/archives/CD4AKTFQT/p1651591254964819?thread_ts=1651589959.354539&cid=CD4AKTFQT

vega-paul commented 2 years ago

blocking all tests with use a termination dts

ValentinTrinque commented 2 years ago

Strangely, there's no trace of the update market proposal in the logs.

EVODelavega commented 2 years ago

"proposal-terms": "closing_timestamp:1651588987 enactment_timestamp:1651588989

With the oracle setting the trading terminated timestamp to 1651598980, 9 seconds before the enactment timestamp just doesn't make all that much sense to me

AnExsomnis commented 2 years ago

@vega-paul , In order to be fully investigate the bug, can you please provide both full proposals - for creating and updating; as well as logs that contain the market ID, because the current log file mentioned in the slack discussion does not contain it. Since the issue is a few months old already, it will be helpful to run it again in order to have the correct files lines, so we can trace the path (currently there are changes in the files and the places mentioned in the logs do not match). Thank you!