The CPU and RAM raised to 100% regularly

doubaozia commented 1 month ago

Self-Hosted Version

23.4.0

CPU Architecture

x86_64

Docker Version

23.0.1

Docker Compose Version

2.16.0

Steps to Reproduce

Install the sentry as the document specified.
Integrate some projects, each of them set tracesSampleRate to 0.2
Wait until the cpu raised to 100%(maybe it will go as a few days...) My host configuration is as follows: 4 cores CPU 8G RAM

Expected Result

It will run as expected and the cpu and ram is normally under 100%.

Actual Result

The cpu usage is:

We can see it raised to 100% regularly... And the coresponding in and out network package count:

After this happened, I restart the host and the services, and go to Sentry web to see the following errors in the "internal" project:

This is one of them:

I encontered this for several months and tried everything metioned in the issues, but is leaves the same!!!

Event ID

No response

hubertdeng123 commented 1 month ago

We recommend minimum requirements of 16 GB of RAM and you are at half of that requirement. If you'd like to, you can try out errors-only self-hosted since it is more lightweight.

doubaozia commented 1 month ago

We recommend minimum requirements of 16 GB of RAM and you are at half of that requirement. If you'd like to, you can try out errors-only self-hosted since it is more lightweight.

It is said maybe the high cpu cost is caused by the RAM? Thx for the reply, I'll try it later.

henri9813 commented 1 month ago

Hello,

Sorry, but the documentation mentioned:

[Recommended system resource](https://develop.sentry.dev/self-hosted/#recommended-system-resource)

Depending on your usage, required system resource to run Sentry may varies. Minimum system specification for running Sentry looks like this:

2 CPU cores
4 GB RAM

Why sentry consume 16GB of ram ?

doubaozia commented 1 month ago

Hello,

Sorry, but the documentation mentioned:

[Recommended system resource](https://develop.sentry.dev/self-hosted/#recommended-system-resource)

Depending on your usage, required system resource to run Sentry may varies. Minimum system specification for running Sentry looks like this:

2 CPU cores
4 GB RAM

Why sentry consume 16GB of ram ?

The newest document mentions this:

Depending on your usage, required system resource to run Sentry may varies. Minimum system specification for running Sentry looks like this:

2 CPU cores
4 GB RAM
Having a machine that have lower system specification than that will fail the installation script.

Current recommended system specification for running Sentry is:

4 CPU cores
16 GB RAM
20 GB free disk storage space

And my host monitor also shows the RAM cost is normaly at 7G

henri9813 commented 1 month ago

Hello,

on a fresh install, I consume

I don't understand why :/

How reduce the sentry memory footprint ?

sneycampos commented 1 month ago

We recommend minimum requirements of 16 GB of RAM and you are at half of that requirement. If you'd like to, you can try out errors-only self-hosted since it is more lightweight.

Could you give some tips how to do this errors-only step?

hubertdeng123 commented 1 month ago

The instructions for doing so are here! https://develop.sentry.dev/self-hosted/#errors-only-self-hosted-sentry

Alena86 commented 2 weeks ago

I would like to add to this...

I have just setup the basic Sentry setup, using the tag 24.8.0 I have Kafka/Postgres/Redis connected to our external services, which I setup by setting the parameters on the sentry.config.py file, so the containers/volumes with it are still being spined up (need to see how to disable those). I also added the ldap-auth from the other repo.

The machine specs are 8 CPU, and 16 GB RAM. Originally was 4 CPU and it was hitting the high load mark so I updated it to 8CPU and now CPU seams fine. I left it running over the weekend, without anyone using it, with just a basic setup, and CPU is being fine, with load under 1... But memory... This morning I checked, it was using 10 GB or memory, 4 SWAP memory!?!? and the rest 6 GB were yellow in htop.

Not sure if this is helping anything when it comes to narrowing down how to lower the consumption of the memory/CPU.

I was thinking to set it up with error-only but I'm not sure what features would be disabled with it? Is there a link with those so I can see if I should even look into that for our needs?

Thank you.

afonsosantos commented 2 weeks ago

@Alena86 Just opened a new issue (on the memory usage topic) here: https://github.com/getsentry/self-hosted/issues/3324

sneycampos commented 2 weeks ago

@afonsosantos could you see the memory usage by service with docker stats?

afonsosantos commented 2 weeks ago

Yes, here it is:

sneycampos commented 2 weeks ago

Well, it looks normal for a full features enabled. Mine are using the same amount of ram but being used by a few projects. We just upgraded to a 64GB RAM (Hetzner) instance to avoid problems. If you can't upgrade it give a try to errors only mode.

Alena86 commented 2 weeks ago

What I found interesting, yesterday for the sake of it I tried to disable kafka/postgres/redis containers from spinning up in docker-compose since I'm using external ones. First surprising thing was that I was getting bunch of errors which I'll try to see why? But the interesting part related to this, is that it was actually using waaaaaaaaaaaaaaay less memory...

sneycampos commented 2 weeks ago

What I found interesting, yesterday for the sake of it I tried to disable kafka/postgres/redis containers from spinning up in docker-compose since I'm using external ones. First surprising thing was that I was getting bunch of errors which I'll try to see why? But the interesting part related to this, is that it was actually using waaaaaaaaaaaaaaay less memory...

These are crucial services and uses a lot of ram 🤣🤣🤣

Alena86 commented 2 weeks ago

I disabled them in docker-compose, because I have EXTERNAL CONNECTIONS to those systems outside, that are working fine. All the Sentry data is generated in the external ones!

Alena86 commented 2 weeks ago

Also those systems (Kafka/Redis/Postgres) are not using that much resources as these containers, and they are HEAVILY used.

sneycampos commented 2 weeks ago

Could you share how did you configure the sentry to use these external services?

Alena86 commented 2 weeks ago

In 2 places... sentry.config.py:

Kafka settings:


DEFAULT_KAFKA_OPTIONS = {
"common": {"bootstrap.servers": "<kafka_server_URL>:<port>",
                "sasl.username": "<kafka_user>",
                "sasl.mechanism": "<auth_mechanism>",
                "security.protocol": "<auth_protocol>",
                "sasl.password": "<kafka_password>",
    }

}

- Postgres settings:

DATABASES = { "default": { "ENGINE": "sentry.db.postgres", "NAME": "", "USER": "", "PASSWORD": "", "HOST": "", "PORT": "", } }


- Redis settinsg:

SENTRY_OPTIONS["redis.clusters"] = { "default": { "hosts": { 0: { "host": "", "password": "", "port": "", "db": "" } } } }


And the same settings in the **relay's config**:

kafka_config:

{name: "bootstrap.servers", value: ":"}
{name: "sasl.username", value: ""}
{name: "sasl.password", value: ""}
{name: "security.protocol", value: ""}
{name: "sasl.mechanism", value: """}
{name: "message.max.bytes", value: 50000000} # 50MB redis: redis://redis_user>:<redis_password>@<redis_host_url:

getsentry / self-hosted