search

vegaprotocol / vega

A Go implementation of the Vega Protocol, a protocol for creating and trading derivatives on a fully decentralised network.
https://vega.xyz
GNU Affero General Public License v3.0
38 stars 22 forks source link

Api: GraphQL server (suspect candles subscriptions) uses all remaining RAM on node and causes VM to become unresponsive until OOMKiller #115

Closed ashleyvega closed 4 years ago

ashleyvega commented 4 years ago

In GitLab by @cdm on Dec 6, 2018, 10:09

ashleyvega commented 4 years ago

In GitLab by @cdm on Dec 11, 2018, 17:37

closed

ashleyvega commented 4 years ago

In GitLab by @cdm on Dec 18, 2018, 16:33

reopened

ashleyvega commented 4 years ago

In GitLab by @cdm on Dec 18, 2018, 16:37

This is still happening on devnet, candle store hasn't received a sweep of the production-iser stick (unlike trade/order stores).

Debugging the OOM event at 2pm has led to the following changes:

1) fixing misleading debugging, hard to work out what's going on in prod, improved this and added more. 2) general tidying up and closing/discarding of transactions (they all look like they're left open) 3) trying to work out best way to simulate blocking channels and effect on memory/GC - we log if we cant push out on a channel e.g. tradestore/candlestore subs, but I dont think we clean up (or we cant from deep in a channel - suicidal snail)

ashleyvega commented 4 years ago

In GitLab by @cdm on Dec 19, 2018, 18:48

More work:

  1. Re-written the handler in the CandleService that pushes out the correct candle for the requested subscribing interval
  2. Worked with @mattrussell36 to ensure we validate timestamps and that correct format for timestamps was being passed down into the core from API entry points.
  3. Added more error handling and go-doc comments
  4. Change get candles graphql endpoint to return candle pointers (test in the wild and do the same to orders/trades if successful at reducing ram overhead)
  5. Testing at 1K orders per second
ashleyvega commented 4 years ago

In GitLab by @cdm on Dec 21, 2018, 10:32

Removed candles pointers code as it wasn't causing the memory leak issue AFAICT.

ashleyvega commented 4 years ago

In GitLab by @cdm on Dec 21, 2018, 10:33

closed