threefoldtech / grid_deployment

Deploy a full Grid backend with docker-compose and snapshots
Apache License 2.0
2 stars 0 forks source link

[graphql] test cockroach cache en max-sql-memory #55

Closed coesensbert closed 1 week ago

coesensbert commented 2 weeks ago

especially for a mainnet stack, if a server has not enough disk iops available, the processor crashes regularly with a timeout to the indexer. I have setup a spare mainnet stack with sata ssd's where I noticed this issue again. Will make some temporary changes there to increase the indexer db (cockroach) memory usage since the default cache is only 128MiB.

https://www.cockroachlabs.com/docs/stable/recommended-production-settings#cache-and-sql-memory-size

--cache=.45
--max-sql-memory=.45
coesensbert commented 2 weeks ago

applied to test to

coesensbert commented 2 weeks ago

nice! https://mon.grid.tf/d/rYdddlPWk/node-exporter-full?orgId=1&var-DS_PROMETHEUS=default&var-job=node_exporter&var-node=prod-02:9100&var-diskdevices=%5Ba-z%5D%2B%7Cnvme%5B0-9%5D%2Bn%5B0-9%5D%2B&from=now-24h&to=now&refresh=30s

more mem usage: image less iops: image

coesensbert commented 2 weeks ago

No issues found so far. On the mainnet snapshot server noticed the processor restarting less, so that's a good sign. This srv has too few iops for a mainnet stack, so the processor issue was not going away but since it was very present there it was a good place to test. Will apply the change to devnet and continue testing there before going to the other nets.

coesensbert commented 1 week ago

no issues detected or reported

coesensbert commented 1 week ago

seems good, will continue with qa, test and mainnet