status-im / infra-status

Infrastructure for Status fleets
https://github.com/status-im/nim-waku
0 stars 2 forks source link

Increase the shared memory allowed by docker #39

Closed Ivansete-status closed 1 week ago

Ivansete-status commented 3 weeks ago

I've come across the following error when trying to perform VACUUM in a database:

nim-waku=# VACUUM (VERBOSE, ANALYZE) public.messages_lookup;

INFO:  vacuuming "nim-waku.public.messages_lookup"
ERROR:  could not resize shared memory segment "/PostgreSQL.160802088" to 490671040 bytes: No space left on device

The reason for that is that Docker, by default, reserves 64MB for shared memory but we need to make it larger in our Postgres setup.

To overcome this and possible future issues, I suggest configuring it to 1 GB. For that, we need to have the docker-compose file with sth like:

version: '3.7'
services:
  db:
    container_name: 'nim-waku-store-db'
    image: 'postgres:15.1-alpine'
    restart: always
    shm_size: '1g'  # Set shared memory size to 1 GB
  ...
  ...
apentori commented 3 weeks ago

PRs ready, waiting for approval before applying them.

Should the staging environment have the same value ?

Ivansete-status commented 3 weeks ago

PRs ready, waiting for approval before applying them.

Should the staging environment have the same value ?

Good question. Aligned with that, a good approach would be to have the very same hw specs in status.staging than in status.prod. Sth to consider in a separate PR of course ( wdyt @jakubgs .)

jakubgs commented 3 weeks ago

That kinda sounds like a waste of money to me, we should rather make the config dynamic dependent on memory available.

apentori commented 2 weeks ago

@Ivansete-status I updated the PR to use 1/16th of the memory available on the host, that would be 1 GB for the prod env. Is that good for you ?

Ivansete-status commented 2 weeks ago

@Ivansete-status I updated the PR to use 1/16th of the memory available on the host, that would be 1 GB for the prod env. Is that good for you ?

@apentori - I think that's fine for now yes. We can fine-tune if required in the future. Thanks !

Ivansete-status commented 1 week ago

@apentori - after applying the change in status.staging I could manually perform the vacuum in the ams database node. Therefore, I think this can be applied into status.prod as well. Nevertheless, before applying the change there we need to make sure the Status QA frens are not using them.

apentori commented 1 week ago

Deployed on both environment