Testnet out of memory (October 28th)

[edd]

Testnet had to be reset. The cause is currently unknown.

Circumstances

• @tamlyn10 was testing opening auctions transitioning in to continous trading
• The market AAPL/DAI seemed to spike block times, but Tamlyn seemed to see a successful transition
• @barnabee then creates a TSLA/DAI market, and during the auction the nodes ground to a halt.

   {
      "id": "P0000217756-0000000007",
      "tradableInstrument": {
        "instrument": {
          "id": "",
          "code": "CRYPTO:TSLADAI/DEC20",
          "name": "TSLA/DAI",
          "baseName": "TESLA",
          "quoteName": "DAI",
          "metadata": {
            "tags": [
            ]
          },
          "initialMarkPrice": "1",
          "future": {
            "maturity": "2020-12-31T22:59:59Z",
            "asset": "94beaabf8a99450f148e337c68357d800dcf52e2f08b50ad89ca37fb3c451fb1",
            "ethereumEvent": {
              "contractID": "0x0B484706fdAF3A4F24b2266446B1cb6d648E3cC1",
              "event": "price_changed",
              "value": "1500000"
            }
          }
        },
        "marginCalculator": {
          "scalingFactors": {
            "searchLevel": 1.1,
            "initialMargin": 1.2,
            "collateralRelease": 1.4
          }
        },
        "logNormalRiskModel": {
          "riskAversionParameter": 0.01,
          "tau": 0.00000190128526884173,
          "params": {
            "mu": 0,
            "r": 0.016,
            "sigma": 0.2
          }
        }
      },
      "decimalPlaces": "5",
      "fees": {
        "factors": {
          "makerFee": "0.004",
          "infrastructureFee": "0.001",
          "liquidityFee": "0.3"
        }
      },
      "openingAuction": {
        "duration": "1800",
        "volume": "0"
      },
      "continuous": {
        "tickSize": "0.00001"
      },
      "priceMonitoringSettings": {
        "priceMonitoringParameters": [
        ],
        "updateFrequency": "0"
      }
    },

https://vegaprotocol.slack.com/archives/C8W4JQ27P/p1603893167257300

[edd]

Checking the logs on n07 I see:

Oct 28 14:18:02 n07-n vega[7725]: 2020-10-28T14:18:02.682Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:02 n07-n vega[7725]: 2020-10-28T14:18:02.682Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761736\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"69cb3eb360cb595a448fe80d6ecf30d6d3aed5aff229e4788d7a981ddcae0987\" side:SIDE_SELL price:38137073 size:31 remaining:31 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895519853268109 reference:\"0b704f06-9c8f-407a-87ab-bbc2089e7a80\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:02 n07-n vega[7725]: 2020-10-28T14:18:02.697Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:02 n07-n vega[7725]: 2020-10-28T14:18:02.697Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761745\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"ee9ea61728e149b493110ec9ad9f432071b4a34b6de86e516df131d81caad993\" side:SIDE_BUY price:38136991 size:77 remaining:77 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895520537285103 reference:\"d1d84f3f-2279-4607-a545-adc4b4876d29\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:02 n07-n vega[7725]: 2020-10-28T14:18:02.710Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:02 n07-n vega[7725]: 2020-10-28T14:18:02.710Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761754\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"d8f565b31386330947bad44c8bc3b4206bd6c681737113440f6f0676097eb31f\" side:SIDE_SELL price:38137046 size:71 remaining:71 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895521186667565 reference:\"8df725f7-7deb-4364-a288-b7bc952cd958\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.864Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.864Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761763\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"6083e4f76c64ee54ce64781d25ea5e8bf8ded4f1c37b4a39f12b836bebc36747\" side:SIDE_BUY price:38137001 size:39 remaining:39 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895522063956571 reference:\"7065ec4b-3eba-4bab-b8df-9094942a7456\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.865Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.866Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761765\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"d96d1ce7d1e37c87cc7ae4d9e6821c140d2929d499c8c57015da71cff9250524\" side:SIDE_BUY price:38137001 size:71 remaining:71 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895522165204861 reference:\"74dc4ba2-9a42-4baa-bbbb-1823a9fde79e\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.867Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.867Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761766\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"3a56fcbcbfdc73d1cf8ab25a033954e12cb54c326a4930bf30547cdd140d42c5\" side:SIDE_BUY price:38137001 size:30 remaining:30 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895522237637313 reference:\"ffd52d95-4a08-4ec7-8c2d-7bfbc30a4666\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.867Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.867Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761769\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"fc7af01c1ed9278e7385752a1d81869167976860ccd265e5c575664080f0163d\" side:SIDE_BUY price:38136936 size:53 remaining:53 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895522475909490 reference:\"9fc7aac4-7582-4eb6-b3bc-442253041b36\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.868Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.869Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761773\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"1a22c510a3b99cae3c46e81a9ba5934e62ffa042db87ff74e0ec91bd4a44c4f3\" side:SIDE_SELL price:38136999 size:77 remaining:77 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895522656763605 reference:\"2e9e11b8-cebb-4e07-a0bb-149cbb86426d\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.876Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.876Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761779\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"4e84e17d749981d1afe6c70158cb49358b91aeb6beae4bf8ac1d9b832d2c06b9\" side:SIDE_SELL price:38137050 size:85 remaining:85 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895523362868159 reference:\"16323344-586b-4eed-b581-4c546f483c9c\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.886Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.887Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761791\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"b72c8e11c0a58813fffc4f0435e892cb3796921496aa81abbde872203635cfb1\" side:SIDE_BUY price:38136940 size:74 remaining:74 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895524688970244 reference:\"68304728-d420-404f-8183-7d02c52fd0e2\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.888Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.888Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761794\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"5d12753e04562e58ab2a7b7b7f4aa0e20606f58612425a75a401221a5679c475\" side:SIDE_SELL price:38137007 size:27 remaining:27 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895524960502559 reference:\"e13ab3de-9895-4571-84fd-15f958137e71\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.894Z        WARN        execution.position        positions/engine.go:329        removing trader from positions engine        {"party-id": "abcf4a76ab8d42bac6391fef5ccc53c1c153047650a54dfd276364a798bf4f4b"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.894Z        WARN        execution.position        positions/engine.go:341        removing trader from positions engine (cpy slice)        {"party-id": "abcf4a76ab8d42bac6391fef5ccc53c1c153047650a54dfd276364a798bf4f4b"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.896Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.896Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761811\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"e5bc4b6a81b9a76a2122fd7e5ca51f77918d043e636d3829b8aa43e16a0226d5\" side:SIDE_SELL price:38136999 size:96 remaining:96 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895526277207462 reference:\"294f918a-0c08-42c2-b937-572b7f3bc163\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.897Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.898Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761815\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"ee00d8d1a68811b235fdc50d913978f2693f36ac5bbfb201a50710905ee6c1bd\" side:SIDE_BUY price:38136972 size:84 remaining:84 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895526385503375 reference:\"664d93a2-bca5-414a-a043-849ec44c1d04\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.915Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.916Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761822\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"0cd3068f46543610651e092ecf395579cbb5becc8838e847718f33541bf658f4\" side:SIDE_BUY price:38136996 size:53 remaining:53 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895527077196692 reference:\"dd34de33-7af7-4efa-9c24-9c00008dce47\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.918Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.918Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761825\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"eec540978f94504570dc914466040bcc0777e8b5ff86c2921203834730143b37\" side:SIDE_SELL price:38136999 size:73 remaining:73 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895527225135619 reference:\"dde2ce7b-fb51-4450-b98e-467d41e02a7c\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.926Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.926Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761841\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"eea533b6e37941fe727f2a5fb43a0345d6acac6ba47b02116a2a62704fc4154f\" side:SIDE_SELL price:38137009 size:52 remaining:52 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895528401082036 reference:\"3ef07006-ee2e-4f0a-ab4b-7cfc79ec56b8\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.934Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.934Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761851\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"de7db7c6c3769967c46a400aaa49312eb68d95db39c0b2ad70b065088c32dec8\" side:SIDE_BUY price:38136953 size:70 remaining:70 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895529669296394 reference:\"f46790a2-0158-4852-b067-4965bf87c6cf\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.935Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.935Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761860\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"5bd0c39cc7a76ed42e2428156cb1e554ce630c0f1cac7e785b9befa11a02dbd1\" side:SIDE_BUY price:38136909 size:84 remaining:84 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895529106536485 reference:\"de0c740a-d0fd-45b8-b411-f3ca2bbd9782\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.936Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.936Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761865\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"c8ccf346bcb8ffa6317e7ba1d8c02d0052e8a1651dd996c48c8356b391b0052e\" side:SIDE_SELL price:38136999 size:54 remaining:54 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895529408611378 reference:\"ba0ede8c-ae65-4c4f-8768-80fff554e809\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.940Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.940Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761867\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"92ecf5191554716966f4f028fa930ec26edb4cfa81578d6550c70e56e0d44abe\" side:SIDE_BUY price:38136991 size:78 remaining:78 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895529606599620 reference:\"b8979a4f-3da6-471b-b683-cb7d34b19e23\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.940Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.940Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761868\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"4e41d3ecd6dea2875a80328e6a33156e29a5f6136321fd3d2291436fcdc7cdcb\" side:SIDE_BUY price:38136961 size:61 remaining:61 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895529789817575 reference:\"d0e7f2b7-63fa-48b0-ae2e-fc35836aa97d\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.941Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.941Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761875\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"ff569d14b7e47f4c63f6be57c7764de5c928a029fed2c3cd80ec495d630096b7\" side:SIDE_SELL price:38136999 size:77 remaining:77 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895530237562355 reference:\"3afc4eb4-5024-4e91-a461-f63293528668\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.944Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.944Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761876\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"048eb4437c2608dca4cd3e53b885196c236aeedeca374469c08ac243663db3a8\" side:SIDE_SELL price:38136999 size:69 remaining:69 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895530280908223 reference:\"447e6c0d-f4ef-47a9-855b-b1985135378b\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.947Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.947Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761880\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"e45d72c5c27462cdd5b3f80d7ff70ef51a5232672c69201c2cb008b62b635a3c\" side:SIDE_BUY price:38136999 size:88 remaining:88 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895530670019105 reference:\"48890103-cb3e-4121-8857-1e62cc6cc95a\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.952Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.952Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761889\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"11f4085b62cbd7095d5502209ba35a722aee62ce942c7dc38edec8a20d54957e\" side:SIDE_BUY price:38136996 size:92 remaining:92 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895531180035281 reference:\"111c4474-46ec-4c90-9959-08a266d57f21\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.954Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.954Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761893\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"361f4ffac0e6d14d2f6e2c7ae54021ae53d784cea8b3cb948b8ecbc1f8d13ef8\" side:SIDE_SELL price:38136999 size:52 remaining:52 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895531223119846 reference:\"d7759db0-dad5-431d-bbbf-1526a2345e6e\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.955Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.956Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761895\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"f466540b1a41aaa517f0def877ba67a1aafb439fa6fa1c705ce0125762f2b272\" side:SIDE_SELL price:38137009 size:94 remaining:94 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895531400143881 reference:\"c1f50f18-7a3b-4f06-9b38-b30204f2b917\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.956Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.956Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761896\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"186edc59070500cf7a60dd7da48952304847a64d9318c3599ce708f79882f8f6\" side:SIDE_BUY price:38137001 size:53 remaining:53 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895531472857470 reference:\"80a7da6c-170b-42f0-bc64-5cfb96a940a0\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.956Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.957Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761898\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"5b4a505d1f41436f90612cdaa4502cbbdc744aec608de317f66e2555cbfd99b7\" side:SIDE_BUY price:38136997 size:59 remaining:59 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895531570308487 reference:\"a8b86a4c-270b-4b74-9864-23fd23b45a25\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.960Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.960Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761903\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"ed79f401c6551180918ed373e7f96f444d7aa08b6374d7bbe96b981a3d8bb751\" side:SIDE_SELL price:38136999 size:79 remaining:79 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895531856069097 reference:\"1745284e-776b-402b-a262-c9358df51267\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.967Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.967Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761910\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"82e2739d166045059b5397d9bddfbb09e7e6542dd4a2abf26983c7404cf7ba96\" side:SIDE_BUY price:38136986 size:54 remaining:54 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895533308650802 reference:\"0d811b2b-8dc5-430e-9954-556501b7ffb8\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.967Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.968Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761913\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"23241e9861798fcf7a65ad648931bfb414a7f9ab99a6474ba28b354d2ba6a2b5\" side:SIDE_SELL price:38136999 size:101 remaining:101 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895533783225045 reference:\"46fd471a-e287-4df1-8d45-e3d202f547d5\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.987Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:12 n07-n vega[7725]: 2020-10-28T14:18:12.987Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761941\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"c3ccbf0cd83ed6cbeba03319d448b6937d17011730765235cf4ce338755d05a5\" side:SIDE_BUY price:38136998 size:96 remaining:96 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895534381904324 reference:\"1e975dcd-2df7-4f35-90b7-6fca67f42eee\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:13 n07-n vega[7725]: 2020-10-28T14:18:13.012Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:13 n07-n vega[7725]: 2020-10-28T14:18:13.012Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761964\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"f0fb942a8dc78a300168125a2f389e944ba3006fecfbbe3010be4e6c2f5d1913\" side:SIDE_BUY price:38137001 size:65 remaining:65 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895537195728146 reference:\"5fc4423a-4031-40d6-b8ea-48e0498f020a\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:13 n07-n vega[7725]: 2020-10-28T14:18:13.018Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:13 n07-n vega[7725]: 2020-10-28T14:18:13.018Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761972\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"b6f222a830ff609caaca2c2f066a21e414e38367c309c3d7e3bc14014666eaa1\" side:SIDE_SELL price:38137033 size:63 remaining:63 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895538288066711 reference:\"4116f90b-c8f0-4fe5-b35d-74a8fe3a780b\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:13 n07-n vega[7725]: 2020-10-28T14:18:13.029Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:13 n07-n vega[7725]: 2020-10-28T14:18:13.029Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761987\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"de7db7c6c3769967c46a400aaa49312eb68d95db39c0b2ad70b065088c32dec8\" side:SIDE_BUY price:38136968 size:81 remaining:81 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895539693937937 reference:\"44cebac5-63c7-4709-ae95-a6f8083241ad\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:13 n07-n vega[7725]: 2020-10-28T14:18:13.029Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:13 n07-n vega[7725]: 2020-10-28T14:18:13.029Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761988\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"4e41d3ecd6dea2875a80328e6a33156e29a5f6136321fd3d2291436fcdc7cdcb\" side:SIDE_BUY price:38136981 size:48 remaining:48 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895539812806406 reference:\"4146ba5b-0930-4f45-9260-fc41b8819fd2\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:13 n07-n vega[7725]: 2020-10-28T14:18:13.035Z        ERROR        execution        execution/market.go:708        Unable to check/add margin for trader        {"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:13 n07-n vega[7725]: 2020-10-28T14:18:13.035Z        ERROR        processor        processor/abci.go:307        error message on creating order        {"order": "id:\"V0000218145-0010761993\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"ff569d14b7e47f4c63f6be57c7764de5c928a029fed2c3cd80ec495d630096b7\" side:SIDE_SELL price:38136999 size:77 remaining:77 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895540260440070 reference:\"10ad7a97-75d1-4607-9031-5556c31a653e\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:20:00 n07-n vega[7725]: 2020-10-28T14:20:00.513Z        ERROR        gateway.gql        graphql/resolvers.go:2501        marketDepth: stream closed        {"error": "rpc error: code = Canceled desc = context canceled"}
Oct 28 14:20:00 n07-n vega[7725]: 2020-10-28T14:20:00.514Z        ERROR        gateway.gql        graphql/resolvers.go:2390        orders: stream closed        {"error": "rpc error: code = Canceled desc = context canceled"}
Oct 28 14:20:00 n07-n vega[7725]: 2020-10-28T14:20:00.514Z        ERROR        gateway.gql        graphql/resolvers.go:2576        candles: stream closed        {"error": "rpc error: code = Canceled desc = context canceled"}
Oct 28 14:20:00 n07-n vega[7725]: 2020-10-28T14:20:00.514Z        ERROR        gateway.gql        graphql/resolvers.go:2501        marketDepth: stream closed        {"error": "rpc error: code = Canceled desc = context canceled"}
Oct 28 14:20:00 n07-n vega[7725]: 2020-10-28T14:20:00.516Z        ERROR        api.grpc        api/trading_data.go:1223        Candles subscriber        {"error": "channel closed", "errorVerbose": "channel closed\ncode.vegaprotocol.io/vega/api.init\n\t/drone/src/api/errors.go:18\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373", "ref": 45}
Oct 28 14:26:37 n07-n vega[7725]: 2020-10-28T14:26:37.219Z        ERROR        gateway.gql        graphql/resolvers.go:2294        marketdata: stream closed        {"error": "rpc error: code = Canceled desc = context canceled"}
Oct 28 14:26:37 n07-n vega[7725]: 2020-10-28T14:26:37.219Z        ERROR        gateway.gql        graphql/resolvers.go:2294        marketdata: stream closed        {"error": "rpc error: code = Canceled desc = context canceled"}
Oct 28 14:26:37 n07-n vega[7725]: 2020-10-28T14:26:37.219Z        ERROR        gateway.gql        graphql/resolvers.go:2294        marketdata: stream closed        {"error": "rpc error: code = Canceled desc = context canceled"}
Oct 28 14:26:37 n07-n vega[7725]: 2020-10-28T14:26:37.219Z        ERROR        gateway.gql        graphql/resolvers.go:2576        candles: stream closed        {"error": "rpc error: code = Canceled desc = context canceled"}
Oct 28 14:26:37 n07-n vega[7725]: 2020-10-28T14:26:37.219Z        ERROR        gateway.gql        graphql/resolvers.go:2294        marketdata: stream closed        {"error": "rpc error: code = Canceled desc = context canceled"}
Oct 28 14:26:37 n07-n vega[7725]: 2020-10-28T14:26:37.219Z        ERROR        gateway.gql        graphql/resolvers.go:2294        marketdata: stream closed        {"error": "rpc error: code = Canceled desc = context canceled"}
Oct 28 14:26:37 n07-n vega[7725]: 2020-10-28T14:26:37.219Z        ERROR        gateway.gql        graphql/resolvers.go:2294        marketdata: stream closed        {"error": "rpc error: code = Canceled desc = context canceled"}
Oct 28 14:26:37 n07-n vega[7725]: 2020-10-28T14:26:37.219Z        ERROR        gateway.gql        graphql/resolvers.go:2501        marketDepth: stream closed        {"error": "rpc error: code = Canceled desc = context canceled"}
Oct 28 14:26:37 n07-n vega[7725]: 2020-10-28T14:26:37.219Z        ERROR        gateway.gql        graphql/resolvers.go:2591        stream closed        {"error": "rpc error: code = Canceled desc = context canceled"}
Oct 28 14:26:37 n07-n vega[7725]: 2020-10-28T14:26:37.219Z        ERROR        gateway.gql        graphql/resolvers.go:2294        marketdata: stream closed        {"error": "rpc error: code = Canceled desc = context canceled"}
Oct 28 14:26:37 n07-n vega[7725]: 2020-10-28T14:26:37.219Z        ERROR        gateway.gql        graphql/resolvers.go:2294        marketdata: stream closed        {"error": "rpc error: code = Canceled desc = context canceled"}
Oct 28 14:26:37 n07-n vega[7725]: 2020-10-28T14:26:37.220Z        ERROR        api.grpc        api/trading_data.go:883        Markets data subscriber        {"error": "channel closed", "errorVerbose": "channel closed\ncode.vegaprotocol.io/vega/api.init\n\t/drone/src/api/errors.go:18\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373", "ref": 70}
Oct 28 14:47:57 n07-n vega[7725]: 2020-10-28T14:47:57.462Z        ERROR        storage        storage/markets.go:112        unable to get market from badger store        {"error": "market does not exist", "errorVerbose": "market does not exist\ncode.vegaprotocol.io/vega/storage.init\n\t/drone/src/storage/markets.go:15\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373", "market-id": "P0000298094-0000000016"}
Oct 28 14:47:59 n07-n vega[7725]: 2020-10-28T14:47:59.012Z        ERROR        storage        storage/markets.go:112        unable to get market from badger store        {"error": "market does not exist", "errorVerbose": "market does not exist\ncode.vegaprotocol.io/vega/storage.init\n\t/drone/src/storage/markets.go:15\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373", "market-id": "P0000103713-0000000002"}
Oct 28 14:47:59 n07-n vega[7725]: 2020-10-28T14:47:59.021Z        ERROR        storage        storage/markets.go:112        unable to get market from badger store        {"error": "market does not exist", "errorVerbose": "market does not exist\ncode.vegaprotocol.io/vega/storage.init\n\t/drone/src/storage/markets.go:15\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373", "market-id": "P0000193236-0000000015"}

[ashleyvega]

I haven't seen unable to get market from badger store before.

[edd]

and n08:

Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.880Z#011WARN#011execution.position#011positions/engine.go:329#011removing trader from positions engine#011{"party-id": "abcf4a76ab8d42bac6391fef5ccc53c1c153047650a54dfd276364a798bf4f4b"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.880Z#011WARN#011execution.position#011positions/engine.go:341#011removing trader from positions engine (cpy slice)#011{"party-id": "abcf4a76ab8d42bac6391fef5ccc53c1c153047650a54dfd276364a798bf4f4b"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.881Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.882Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761811\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"e5bc4b6a81b9a76a2122fd7e5ca51f77918d043e636d3829b8aa43e16a0226d5\" side:SIDE_SELL price:38136999 size:96 remaining:96 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895526277207462 reference:\"294f918a-0c08-42c2-b937-572b7f3bc163\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.882Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.882Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761815\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"ee00d8d1a68811b235fdc50d913978f2693f36ac5bbfb201a50710905ee6c1bd\" side:SIDE_BUY price:38136972 size:84 remaining:84 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895526385503375 reference:\"664d93a2-bca5-414a-a043-849ec44c1d04\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.888Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.888Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761822\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"0cd3068f46543610651e092ecf395579cbb5becc8838e847718f33541bf658f4\" side:SIDE_BUY price:38136996 size:53 remaining:53 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895527077196692 reference:\"dd34de33-7af7-4efa-9c24-9c00008dce47\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.888Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.888Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761825\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"eec540978f94504570dc914466040bcc0777e8b5ff86c2921203834730143b37\" side:SIDE_SELL price:38136999 size:73 remaining:73 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895527225135619 reference:\"dde2ce7b-fb51-4450-b98e-467d41e02a7c\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.892Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.892Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761841\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"eea533b6e37941fe727f2a5fb43a0345d6acac6ba47b02116a2a62704fc4154f\" side:SIDE_SELL price:38137009 size:52 remaining:52 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895528401082036 reference:\"3ef07006-ee2e-4f0a-ab4b-7cfc79ec56b8\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.895Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.895Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761851\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"de7db7c6c3769967c46a400aaa49312eb68d95db39c0b2ad70b065088c32dec8\" side:SIDE_BUY price:38136953 size:70 remaining:70 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895529669296394 reference:\"f46790a2-0158-4852-b067-4965bf87c6cf\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.896Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.896Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761860\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"5bd0c39cc7a76ed42e2428156cb1e554ce630c0f1cac7e785b9befa11a02dbd1\" side:SIDE_BUY price:38136909 size:84 remaining:84 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895529106536485 reference:\"de0c740a-d0fd-45b8-b411-f3ca2bbd9782\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.897Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.897Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761865\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"c8ccf346bcb8ffa6317e7ba1d8c02d0052e8a1651dd996c48c8356b391b0052e\" side:SIDE_SELL price:38136999 size:54 remaining:54 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895529408611378 reference:\"ba0ede8c-ae65-4c4f-8768-80fff554e809\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.898Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.898Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761867\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"92ecf5191554716966f4f028fa930ec26edb4cfa81578d6550c70e56e0d44abe\" side:SIDE_BUY price:38136991 size:78 remaining:78 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895529606599620 reference:\"b8979a4f-3da6-471b-b683-cb7d34b19e23\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.898Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.898Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761868\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"4e41d3ecd6dea2875a80328e6a33156e29a5f6136321fd3d2291436fcdc7cdcb\" side:SIDE_BUY price:38136961 size:61 remaining:61 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895529789817575 reference:\"d0e7f2b7-63fa-48b0-ae2e-fc35836aa97d\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.899Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.899Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761875\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"ff569d14b7e47f4c63f6be57c7764de5c928a029fed2c3cd80ec495d630096b7\" side:SIDE_SELL price:38136999 size:77 remaining:77 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895530237562355 reference:\"3afc4eb4-5024-4e91-a461-f63293528668\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.899Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.899Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761876\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"048eb4437c2608dca4cd3e53b885196c236aeedeca374469c08ac243663db3a8\" side:SIDE_SELL price:38136999 size:69 remaining:69 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895530280908223 reference:\"447e6c0d-f4ef-47a9-855b-b1985135378b\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.901Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.901Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761880\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"e45d72c5c27462cdd5b3f80d7ff70ef51a5232672c69201c2cb008b62b635a3c\" side:SIDE_BUY price:38136999 size:88 remaining:88 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895530670019105 reference:\"48890103-cb3e-4121-8857-1e62cc6cc95a\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.903Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.903Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761889\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"11f4085b62cbd7095d5502209ba35a722aee62ce942c7dc38edec8a20d54957e\" side:SIDE_BUY price:38136996 size:92 remaining:92 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895531180035281 reference:\"111c4474-46ec-4c90-9959-08a266d57f21\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.904Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.904Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761893\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"361f4ffac0e6d14d2f6e2c7ae54021ae53d784cea8b3cb948b8ecbc1f8d13ef8\" side:SIDE_SELL price:38136999 size:52 remaining:52 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895531223119846 reference:\"d7759db0-dad5-431d-bbbf-1526a2345e6e\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.905Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.905Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761895\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"f466540b1a41aaa517f0def877ba67a1aafb439fa6fa1c705ce0125762f2b272\" side:SIDE_SELL price:38137009 size:94 remaining:94 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895531400143881 reference:\"c1f50f18-7a3b-4f06-9b38-b30204f2b917\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.905Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.905Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761896\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"186edc59070500cf7a60dd7da48952304847a64d9318c3599ce708f79882f8f6\" side:SIDE_BUY price:38137001 size:53 remaining:53 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895531472857470 reference:\"80a7da6c-170b-42f0-bc64-5cfb96a940a0\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.905Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.905Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761898\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"5b4a505d1f41436f90612cdaa4502cbbdc744aec608de317f66e2555cbfd99b7\" side:SIDE_BUY price:38136997 size:59 remaining:59 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895531570308487 reference:\"a8b86a4c-270b-4b74-9864-23fd23b45a25\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.906Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.906Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761903\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"ed79f401c6551180918ed373e7f96f444d7aa08b6374d7bbe96b981a3d8bb751\" side:SIDE_SELL price:38136999 size:79 remaining:79 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895531856069097 reference:\"1745284e-776b-402b-a262-c9358df51267\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.910Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.911Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761910\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"82e2739d166045059b5397d9bddfbb09e7e6542dd4a2abf26983c7404cf7ba96\" side:SIDE_BUY price:38136986 size:54 remaining:54 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895533308650802 reference:\"0d811b2b-8dc5-430e-9954-556501b7ffb8\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.911Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.911Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761913\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"23241e9861798fcf7a65ad648931bfb414a7f9ab99a6474ba28b354d2ba6a2b5\" side:SIDE_SELL price:38136999 size:101 remaining:101 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895533783225045 reference:\"46fd471a-e287-4df1-8d45-e3d202f547d5\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.919Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.919Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761941\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"c3ccbf0cd83ed6cbeba03319d448b6937d17011730765235cf4ce338755d05a5\" side:SIDE_BUY price:38136998 size:96 remaining:96 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895534381904324 reference:\"1e975dcd-2df7-4f35-90b7-6fca67f42eee\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.932Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.933Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761964\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"f0fb942a8dc78a300168125a2f389e944ba3006fecfbbe3010be4e6c2f5d1913\" side:SIDE_BUY price:38137001 size:65 remaining:65 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895537195728146 reference:\"5fc4423a-4031-40d6-b8ea-48e0498f020a\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.935Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.935Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761972\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"b6f222a830ff609caaca2c2f066a21e414e38367c309c3d7e3bc14014666eaa1\" side:SIDE_SELL price:38137033 size:63 remaining:63 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895538288066711 reference:\"4116f90b-c8f0-4fe5-b35d-74a8fe3a780b\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.938Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.938Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761987\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"de7db7c6c3769967c46a400aaa49312eb68d95db39c0b2ad70b065088c32dec8\" side:SIDE_BUY price:38136968 size:81 remaining:81 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895539693937937 reference:\"44cebac5-63c7-4709-ae95-a6f8083241ad\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.938Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.938Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761988\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"4e41d3ecd6dea2875a80328e6a33156e29a5f6136321fd3d2291436fcdc7cdcb\" side:SIDE_BUY price:38136981 size:48 remaining:48 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895539812806406 reference:\"4146ba5b-0930-4f45-9260-fc41b8819fd2\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.939Z#011ERROR#011execution#011execution/market.go:708#011Unable to check/add margin for trader#011{"market-id": "VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67", "error": "insufficient margin", "errorVerbose": "insufficient margin\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:43\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:18:31 n08-n vega[15521]: 2020-10-28T14:18:31.939Z#011ERROR#011processor#011processor/abci.go:307#011error message on creating order#011{"order": "id:\"V0000218145-0010761993\" marketID:\"VHSRA2G5MDFKREFJ5TOAGHZBBDGCYS67\" partyID:\"ff569d14b7e47f4c63f6be57c7764de5c928a029fed2c3cd80ec495d630096b7\" side:SIDE_SELL price:38136999 size:77 remaining:77 timeInForce:TIF_GTT type:TYPE_LIMIT createdAt:1603894623512127405 status:STATUS_REJECTED expiresAt:1603895540260440070 reference:\"10ad7a97-75d1-4607-9031-5556c31a653e\" reason:ORDER_ERROR_MARGIN_CHECK_FAILED version:1 ", "error": "margin check failed", "errorVerbose": "margin check failed\ncode.vegaprotocol.io/vega/execution.init\n\t/drone/src/execution/market.go:41\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"}
Oct 28 14:20:15 n08-n vega[15521]: 2020-10-28T14:20:15.489Z#011ERROR#011api.grpc#011api/trading_data.go:1153#011Trades subscriber#011{"error": "channel closed", "errorVerbose": "channel closed\ncode.vegaprotocol.io/vega/api.init\n\t/drone/src/api/errors.go:18\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5420\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5415\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373", "ref": 92}

This feels pretty similar to https://github.com/vegaprotocol/vega/wiki/TestnetOutage20201009 for me - poor handling of closeouts. @ashleyvega does that look reasonable to you?

[edd]

@ashleyvega tells me that this looks pretty familiar log-wise, in terms of the number of margin checks. So it may be a red herring. It's due to the 🤖 s being a bit dumb rather than a flood of closeouts.

[edd]

Oct 28 14:18:25 n01-n vega[32424]: fatal error: runtime: out of memory
Oct 28 14:18:25 n01-n vega[32424]: runtime stack:
Oct 28 14:18:25 n01-n vega[32424]: runtime.throw(0x1b5806a, 0x16)
Oct 28 14:18:25 n01-n vega[32424]: #011/usr/local/go/src/runtime/panic.go:1116 +0x72
Oct 28 14:18:25 n01-n vega[32424]: runtime.sysMap(0xc1b0000000, 0xf8000000, 0x2dfcc38)
Oct 28 14:18:25 n01-n vega[32424]: #011/usr/local/go/src/runtime/mem_linux.go:169 +0xc5
Oct 28 14:18:25 n01-n vega[32424]: runtime.(*mheap).sysAlloc(0x2de7740, 0xf6800000, 0x2de7748, 0x7b400)
Oct 28 14:18:25 n01-n vega[32424]: #011/usr/local/go/src/runtime/malloc.go:715 +0x1cd
Oct 28 14:18:25 n01-n vega[32424]: runtime.(*mheap).grow(0x2de7740, 0x7b400, 0x0)
Oct 28 14:18:25 n01-n vega[32424]: #011/usr/local/go/src/runtime/mheap.go:1286 +0x11c
Oct 28 14:18:25 n01-n vega[32424]: runtime.(*mheap).allocSpan(0x2de7740, 0x7b400, 0x7f00f3ff0100, 0x2dfcc48, 0x11)
Oct 28 14:18:25 n01-n vega[32424]: #011/usr/local/go/src/runtime/mheap.go:1124 +0x6a0
Oct 28 14:18:25 n01-n vega[32424]: runtime.(*mheap).alloc.func1()
Oct 28 14:18:25 n01-n vega[32424]: #011/usr/local/go/src/runtime/mheap.go:871 +0x64
Oct 28 14:18:25 n01-n vega[32424]: runtime.(*mheap).alloc(0x2de7740, 0x7b400, 0x470101, 0x2de7740)
Oct 28 14:18:25 n01-n vega[32424]: #011/usr/local/go/src/runtime/mheap.go:865 +0x81
Oct 28 14:18:25 n01-n vega[32424]: runtime.largeAlloc(0xf6800000, 0x470101, 0xc0332de3c0)
Oct 28 14:18:25 n01-n vega[32424]: #011/usr/local/go/src/runtime/malloc.go:1152 +0x92
Oct 28 14:18:25 n01-n vega[32424]: runtime.mallocgc.func1()
Oct 28 14:18:25 n01-n vega[32424]: #011/usr/local/go/src/runtime/malloc.go:1047 +0x46
Oct 28 14:18:25 n01-n vega[32424]: runtime.systemstack(0x0)
Oct 28 14:18:25 n01-n vega[32424]: #011/usr/local/go/src/runtime/asm_amd64.s:370 +0x66
Oct 28 14:18:25 n01-n vega[32424]: runtime.mstart()
Oct 28 14:18:25 n01-n vega[32424]: #011/usr/local/go/src/runtime/proc.go:1041
Oct 28 14:18:25 n01-n vega[32424]: goroutine 144 [running]:
Oct 28 14:18:25 n01-n vega[32424]: runtime.systemstack_switch()
Oct 28 14:18:25 n01-n vega[32424]: #011/usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc0074f84e8 sp=0xc0074f84e0 pc=0x4756f0
Oct 28 14:18:25 n01-n vega[32424]: runtime.mallocgc(0xf6800000, 0x19e46a0, 0xc000237801, 0x7f01a36b17d0)
Oct 28 14:18:25 n01-n vega[32424]: #011/usr/local/go/src/runtime/malloc.go:1046 +0x895 fp=0xc0074f8588 sp=0xc0074f84e8 pc=0x41c9e5
Oct 28 14:18:25 n01-n vega[32424]: runtime.newarray(0x19e46a0, 0x880000, 0x4d770dbb3cc10f65)
Oct 28 14:18:25 n01-n vega[32424]: #011/usr/local/go/src/runtime/malloc.go:1187 +0x63 fp=0xc0074f85b8 sp=0xc0074f8588 pc=0x41ce23
Oct 28 14:18:25 n01-n vega[32424]: runtime.makeBucketArray(0x18dc6c0, 0xc0074f8617, 0x0, 0x0, 0xb1801)
Oct 28 14:18:25 n01-n vega[32424]: #011/usr/local/go/src/runtime/map.go:362 +0x183 fp=0xc0074f85f0 sp=0xc0074f85b8 pc=0x41dd03
Oct 28 14:18:25 n01-n vega[32424]: runtime.hashGrow(0x18dc6c0, 0xc014c6e1e0)
Oct 28 14:18:25 n01-n vega[32424]: #011/usr/local/go/src/runtime/map.go:1027 +0x89 fp=0xc0074f8640 sp=0xc0074f85f0 pc=0x41f8c9
Oct 28 14:18:25 n01-n vega[32424]: runtime.mapassign_fast64(0x18dc6c0, 0xc014c6e1e0, 0x15af080, 0xc033342f48)
Oct 28 14:18:25 n01-n vega[32424]: #011/usr/local/go/src/runtime/map_fast64.go:156 +0x10d fp=0xc0074f8680 sp=0xc0074f8640 pc=0x42196d
Oct 28 14:18:25 n01-n vega[32424]: code.vegaprotocol.io/vega/matching.(*OrderBook).buildCumulativePriceLevels(0xc00a918770, 0x2faf080, 0x7a120, 0xc00a2d2760)
Oct 28 14:18:25 n01-n vega[32424]: #011/drone/src/matching/orderbook.go:344 +0x103 fp=0xc0074f8740 sp=0xc0074f8680 pc=0x1350263
Oct 28 14:18:25 n01-n vega[32424]: code.vegaprotocol.io/vega/matching.(*OrderBook).GetIndicativePriceAndVolume(0xc00a918770, 0xc9, 0xc016ee21a0, 0xc01c1b7d40)
Oct 28 14:18:25 n01-n vega[32424]: #011/drone/src/matching/orderbook.go:256 +0x10a fp=0xc0074f8a00 sp=0xc0074f8740 pc=0x134f74a
Oct 28 14:18:25 n01-n vega[32424]: code.vegaprotocol.io/vega/execution.(*Market).GetMarketData(0xc00be9c000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Oct 28 14:18:25 n01-n vega[32424]: #011/drone/src/execution/market.go:287 +0x3e1 fp=0xc0074f8cb0 sp=0xc0074f8a00 pc=0x1374621
Oct 28 14:18:25 n01-n vega[32424]: code.vegaprotocol.io/vega/execution.(*Engine).onChainTimeUpdate(0xc000282780, 0x1fcfb00, 0xc014c6e0c0, 0xb1c0976, 0xed72b7473, 0x0)
Oct 28 14:18:25 n01-n vega[32424]: #011/drone/src/execution/engine.go:366 +0x1c0 fp=0xc0074f8f58 sp=0xc0074f8cb0 pc=0x13716f0
Oct 28 14:18:25 n01-n vega[32424]: code.vegaprotocol.io/vega/execution.(*Engine).onChainTimeUpdate-fm(0x1fcfb00, 0xc014c6e0c0, 0xb1c0976, 0xed72b7473, 0x0)
Oct 28 14:18:25 n01-n vega[32424]: #011/drone/src/execution/engine.go:361 +0x5c fp=0xc0074f8f98 sp=0xc0074f8f58 pc=0x138bf1c
Oct 28 14:18:25 n01-n vega[32424]: code.vegaprotocol.io/vega/vegatime.(*Svc).notify(0xc0002ea180, 0x1fcfb00, 0xc014c6e0c0, 0xb1c0976, 0xed72b7473, 0x0)
Oct 28 14:18:25 n01-n vega[32424]: #011/drone/src/vegatime/service.go:73 +0xc9 fp=0xc0074f9008 sp=0xc0074f8f98 pc=0x10b42f9
Oct 28 14:18:25 n01-n vega[32424]: code.vegaprotocol.io/vega/vegatime.(*Svc).SetTimeNow(0xc0002ea180, 0x1fcfb00, 0xc014c6e0c0, 0xb1c0976, 0xed72b7473, 0x0)
Oct 28 14:18:25 n01-n vega[32424]: #011/drone/src/vegatime/service.go:48 +0x170 fp=0xc0074f9060 sp=0xc0074f9008 pc=0x10b4010
Oct 28 14:18:25 n01-n vega[32424]: code.vegaprotocol.io/vega/processor.(*App).OnBeginBlock(0xc0003bb980, 0xc028748120, 0x20, 0x20, 0xa, 0x0, 0xc014214cc0, 0xe, 0x35421, 0xb1c0976, ...)
Oct 28 14:18:25 n01-n vega[32424]: #011/drone/src/processor/abci.go:197 +0x295 fp=0xc0074f9320 sp=0xc0074f9060 pc=0x13b9265
Oct 28 14:18:25 n01-n vega[32424]: code.vegaprotocol.io/vega/processor.(*App).OnBeginBlock-fm(0xc028748120, 0x20, 0x20, 0xa, 0x0, 0xc014214cc0, 0xe, 0x35421, 0xb1c0976, 0xed72b7473, ...)
Oct 28 14:18:25 n01-n vega[32424]: #011/drone/src/processor/abci.go:192 +0x5f fp=0xc0074f9500 sp=0xc0074f9320 pc=0x13c410f
Oct 28 14:18:25 n01-n vega[32424]: code.vegaprotocol.io/vega/blockchain/abci.(*App).BeginBlock(0xc00031c070, 0xc028748120, 0x20, 0x20, 0xa, 0x0, 0xc014214cc0, 0xe, 0x35421, 0xb1c0976, ...)
Oct 28 14:18:25 n01-n vega[32424]: #011/drone/src/blockchain/abci/abci.go:44 +0xad fp=0xc0074f96d8 sp=0xc0074f9500 pc=0x13ab0ad
Oct 28 14:18:25 n01-n vega[32424]: github.com/tendermint/tendermint/abci/server.(*SocketServer).handleRequest(0xc000e2c210, 0xc01b291790, 0xc0072c31a0)
Oct 28 14:18:25 n01-n vega[32424]: #011/drone/src/vendor/github.com/tendermint/tendermint/abci/server/socket_server.go:222 +0xb4a fp=0xc0074f9ea8 sp=0xc0074f96d8 pc=0x13a95aa
Oct 28 14:18:25 n01-n vega[32424]: github.com/tendermint/tendermint/abci/server.(*SocketServer).handleRequests(0xc000e2c210, 0xc0072c3140, 0x1f9e740, 0xc00019e4e0, 0xc0072c31a0)
Oct 28 14:18:25 n01-n vega[32424]: #011/drone/src/vendor/github.com/tendermint/tendermint/abci/server/socket_server.go:189 +0x173 fp=0xc0074f9fb8 sp=0xc0074f9ea8 pc=0x13a8763
Oct 28 14:18:25 n01-n vega[32424]: runtime.goexit()

edd_vegaprotocol_io@n02-n:/home/vega/backup/log/2020-10-28-14-51-40$ grep '14:18:18' vega.log
Oct 28 14:18:18 n02-n vega[11594]: fatal error: runtime: out of memory
Oct 28 14:18:18 n02-n vega[11594]: runtime stack:
Oct 28 14:18:18 n02-n vega[11594]: runtime.throw(0x1b5806a, 0x16)
Oct 28 14:18:18 n02-n vega[11594]: #011/usr/local/go/src/runtime/panic.go:1116 +0x72
Oct 28 14:18:18 n02-n vega[11594]: runtime.sysMap(0xc1b0000000, 0xf8000000, 0x2dfcc38)
Oct 28 14:18:18 n02-n vega[11594]: #011/usr/local/go/src/runtime/mem_linux.go:169 +0xc5
Oct 28 14:18:18 n02-n vega[11594]: runtime.(*mheap).sysAlloc(0x2de7740, 0xf6800000, 0x2de7748, 0x7b400)
Oct 28 14:18:18 n02-n vega[11594]: #011/usr/local/go/src/runtime/malloc.go:715 +0x1cd
Oct 28 14:18:18 n02-n vega[11594]: runtime.(*mheap).grow(0x2de7740, 0x7b400, 0x0)
Oct 28 14:18:18 n02-n vega[11594]: #011/usr/local/go/src/runtime/mheap.go:1286 +0x11c
Oct 28 14:18:18 n02-n vega[11594]: runtime.(*mheap).allocSpan(0x2de7740, 0x7b400, 0x7f44c3ff0100, 0x2dfcc48, 0x11)
Oct 28 14:18:18 n02-n vega[11594]: #011/usr/local/go/src/runtime/mheap.go:1124 +0x6a0
Oct 28 14:18:18 n02-n vega[11594]: runtime.(*mheap).alloc.func1()
Oct 28 14:18:18 n02-n vega[11594]: #011/usr/local/go/src/runtime/mheap.go:871 +0x64
Oct 28 14:18:18 n02-n vega[11594]: runtime.(*mheap).alloc(0x2de7740, 0x7b400, 0x470101, 0x2de7740)
Oct 28 14:18:18 n02-n vega[11594]: #011/usr/local/go/src/runtime/mheap.go:865 +0x81
Oct 28 14:18:18 n02-n vega[11594]: runtime.largeAlloc(0xf6800000, 0x470101, 0xc034b5a000)
Oct 28 14:18:18 n02-n vega[11594]: #011/usr/local/go/src/runtime/malloc.go:1152 +0x92
Oct 28 14:18:18 n02-n vega[11594]: runtime.mallocgc.func1()
Oct 28 14:18:18 n02-n vega[11594]: #011/usr/local/go/src/runtime/malloc.go:1047 +0x46
Oct 28 14:18:18 n02-n vega[11594]: runtime.systemstack(0x0)
Oct 28 14:18:18 n02-n vega[11594]: #011/usr/local/go/src/runtime/asm_amd64.s:370 +0x66
Oct 28 14:18:18 n02-n vega[11594]: runtime.mstart()
Oct 28 14:18:18 n02-n vega[11594]: #011/usr/local/go/src/runtime/proc.go:1041
Oct 28 14:18:18 n02-n vega[11594]: goroutine 149 [running]:
Oct 28 14:18:18 n02-n vega[11594]: runtime.systemstack_switch()
Oct 28 14:18:18 n02-n vega[11594]: #011/usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc0276e24e8 sp=0xc0276e24e0 pc=0x4756f0
Oct 28 14:18:18 n02-n vega[11594]: runtime.mallocgc(0xf6800000, 0x19e46a0, 0xc0003ae001, 0x7f4575726108)
Oct 28 14:18:18 n02-n vega[11594]: #011/usr/local/go/src/runtime/malloc.go:1046 +0x895 fp=0xc0276e2588 sp=0xc0276e24e8 pc=0x41c9e5
Oct 28 14:18:18 n02-n vega[11594]: runtime.newarray(0x19e46a0, 0x880000, 0xc126a07520)
Oct 28 14:18:18 n02-n vega[11594]: #011/usr/local/go/src/runtime/malloc.go:1187 +0x63 fp=0xc0276e25b8 sp=0xc0276e2588 pc=0x41ce23
Oct 28 14:18:18 n02-n vega[11594]: runtime.makeBucketArray(0x18dc6c0, 0xc0276e2617, 0x0, 0x0, 0xb1801)
Oct 28 14:18:18 n02-n vega[11594]: #011/usr/local/go/src/runtime/map.go:362 +0x183 fp=0xc0276e25f0 sp=0xc0276e25b8 pc=0x41dd03
Oct 28 14:18:18 n02-n vega[11594]: runtime.hashGrow(0x18dc6c0, 0xc01b7bad50)
Oct 28 14:18:18 n02-n vega[11594]: #011/usr/local/go/src/runtime/map.go:1027 +0x89 fp=0xc0276e2640 sp=0xc0276e25f0 pc=0x41f8c9
Oct 28 14:18:18 n02-n vega[11594]: runtime.mapassign_fast64(0x18dc6c0, 0xc01b7bad50, 0x15af080, 0xc034c21c68)
Oct 28 14:18:18 n02-n vega[11594]: #011/usr/local/go/src/runtime/map_fast64.go:156 +0x10d fp=0xc0276e2680 sp=0xc0276e2640 pc=0x42196d
Oct 28 14:18:18 n02-n vega[11594]: code.vegaprotocol.io/vega/matching.(*OrderBook).buildCumulativePriceLevels(0xc00d00f650, 0x2faf080, 0x7a120, 0xc012a31ae0)
Oct 28 14:18:18 n02-n vega[11594]: #011/drone/src/matching/orderbook.go:344 +0x103 fp=0xc0276e2740 sp=0xc0276e2680 pc=0x1350263
Oct 28 14:18:18 n02-n vega[11594]: code.vegaprotocol.io/vega/matching.(*OrderBook).GetIndicativePriceAndVolume(0xc00d00f650, 0xc9, 0xc014050e00, 0xc00ae8c5d0)
Oct 28 14:18:18 n02-n vega[11594]: #011/drone/src/matching/orderbook.go:256 +0x10a fp=0xc0276e2a00 sp=0xc0276e2740 pc=0x134f74a
Oct 28 14:18:18 n02-n vega[11594]: code.vegaprotocol.io/vega/execution.(*Market).GetMarketData(0xc0103ebe10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Oct 28 14:18:18 n02-n vega[11594]: #011/drone/src/execution/market.go:287 +0x3e1 fp=0xc0276e2cb0 sp=0xc0276e2a00 pc=0x1374621
Oct 28 14:18:18 n02-n vega[11594]: code.vegaprotocol.io/vega/execution.(*Engine).onChainTimeUpdate(0xc0006e4180, 0x1fcfb00, 0xc01b7bac90, 0xb1c0976, 0xed72b7473, 0x0)
Oct 28 14:18:18 n02-n vega[11594]: #011/drone/src/execution/engine.go:366 +0x1c0 fp=0xc0276e2f58 sp=0xc0276e2cb0 pc=0x13716f0
Oct 28 14:18:18 n02-n vega[11594]: code.vegaprotocol.io/vega/execution.(*Engine).onChainTimeUpdate-fm(0x1fcfb00, 0xc01b7bac90, 0xb1c0976, 0xed72b7473, 0x0)
Oct 28 14:18:18 n02-n vega[11594]: #011/drone/src/execution/engine.go:361 +0x5c fp=0xc0276e2f98 sp=0xc0276e2f58 pc=0x138bf1c
Oct 28 14:18:18 n02-n vega[11594]: code.vegaprotocol.io/vega/vegatime.(*Svc).notify(0xc000131860, 0x1fcfb00, 0xc01b7bac90, 0xb1c0976, 0xed72b7473, 0x0)
Oct 28 14:18:18 n02-n vega[11594]: #011/drone/src/vegatime/service.go:73 +0xc9 fp=0xc0276e3008 sp=0xc0276e2f98 pc=0x10b42f9
Oct 28 14:18:18 n02-n vega[11594]: code.vegaprotocol.io/vega/vegatime.(*Svc).SetTimeNow(0xc000131860, 0x1fcfb00, 0xc01b7bac90, 0xb1c0976, 0xed72b7473, 0x0)