search

getsentry / self-hosted

Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept
https://develop.sentry.dev/self-hosted/
Other
7.89k stars 1.77k forks source link

Container sentry-self-hosted_web_1 (web service) consumes a ot of RAM #2064

Closed vsfomin closed 1 year ago

vsfomin commented 1 year ago

Self-Hosted Version

23.3.1

CPU Architecture

x86_64

Docker Version

23.0.2

Docker Compose Version

2.17.2

Steps to Reproduce

  1. execute install.sh
  2. Waited until the installation process ends and containers will become in Up state
  3. Container sentry-self-hosted_web_1 didn't become healthy. respawned uWSGI http 1 (pid: ) in logs.

Expected Result

All containers are in Up state and 127.0.0.1:9000 works fine.

Actual Result

sentry_install_logs.txt docker-compose_logs.txt sentry-self-hosted_web_1_logs.txt

image

web_1  | Updating certificates in /etc/ssl/certs...
web_1  | 0 added, 0 removed; done.
web_1  | Running hooks in /etc/ca-certificates/update.d...
web_1  | done.
web_1  | 08:59:11 [INFO] sentry.plugins.github: apps-not-configured
web_1  | *** Starting uWSGI 2.0.20 (64bit) on [Tue Apr  4 08:59:11 2023] ***
web_1  | compiled with version: 8.3.1 20190311 (Red Hat 8.3.1-3) on 11 November 2021 18:13:24
web_1  | os: Linux-5.14.0-162.22.2.el9_1.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 30 17:01:52 UTC 2023
web_1  | nodename: 1d350b263a41
web_1  | machine: x86_64
web_1  | clock source: unix
web_1  | pcre jit disabled
web_1  | detected number of CPU cores: 4
web_1  | current working directory: /
web_1  | detected binary path: /usr/local/bin/python
web_1  | your memory page size is 4096 bytes
web_1  | detected max file descriptor number: 1073741816
web_1  | lock engine: pthread robust mutexes
web_1  | thunder lock: enabled
web_1  | uWSGI http bound on 0.0.0.0:9000 fd 4
web_1  | uwsgi socket 0 bound to TCP address 127.0.0.1:45111 (port auto-assigned) fd 3
web_1  | Python version: 3.8.13 (default, Aug 23 2022, 09:42:19)  [GCC 10.2.1 20210110]
web_1  | --- Python VM already initialized ---
web_1  | Python main interpreter initialized at 0x55bd25f3f870
web_1  | python threads support enabled
web_1  | your server socket listen backlog is limited to 100 connections
web_1  | your mercy for graceful operations on workers is 60 seconds
web_1  | setting request body buffering size to 65536 bytes
web_1  | mapped 1924224 bytes (1879 KB) for 12 cores
web_1  | *** Operational MODE: preforking+threaded ***
web_1  | spawned uWSGI master process (pid: 686)
web_1  | spawned uWSGI worker 1 (pid: 688, cores: 4)
web_1  | spawned uWSGI worker 2 (pid: 689, cores: 4)
web_1  | spawned uWSGI worker 3 (pid: 690, cores: 4)
web_1  | spawned uWSGI http 1 (pid: 691)
web_1  | respawned uWSGI http 1 (pid: 695)
web_1  | respawned uWSGI http 1 (pid: 703)
web_1  | respawned uWSGI http 1 (pid: 704)
web_1  | /usr/local/lib/python3.8/site-packages/memcache.py:1303: SyntaxWarning: "is" with a literal. Did you mean "=="?
web_1  |   if key is '':
web_1  | /usr/local/lib/python3.8/site-packages/memcache.py:1304: SyntaxWarning: "is" with a literal. Did you mean "=="?
web_1  |   if key_extra_len is 0:
web_1  | /usr/local/lib/python3.8/site-packages/memcache.py:1303: SyntaxWarning: "is" with a literal. Did you mean "=="?
web_1  |   if key is '':
web_1  | /usr/local/lib/python3.8/site-packages/memcache.py:1304: SyntaxWarning: "is" with a literal. Did you mean "=="?
web_1  |   if key_extra_len is 0:
web_1  | /usr/local/lib/python3.8/site-packages/memcache.py:1303: SyntaxWarning: "is" with a literal. Did you mean "=="?
web_1  |   if key is '':
web_1  | /usr/local/lib/python3.8/site-packages/memcache.py:1304: SyntaxWarning: "is" with a literal. Did you mean "=="?
web_1  |   if key_extra_len is 0:
web_1  | 08:59:28 [INFO] sentry.plugins.github: apps-not-configured
web_1  | 08:59:28 [INFO] sentry.plugins.github: apps-not-configured
web_1  | 08:59:28 [INFO] sentry.plugins.github: apps-not-configured
web_1  | WSGI app 0 (mountpoint='') ready in 17 seconds on interpreter 0x55bd25f3f870 pid: 690 (default app)
web_1  | WSGI app 0 (mountpoint='') ready in 18 seconds on interpreter 0x55bd25f3f870 pid: 689 (default app)
web_1  | WSGI app 0 (mountpoint='') ready in 17 seconds on interpreter 0x55bd25f3f870 pid: 688 (default app)
web_1  | respawned uWSGI http 1 (pid: 714)
web_1  | respawned uWSGI http 1 (pid: 715)
web_1  | respawned uWSGI http 1 (pid: 716)
web_1  | respawned uWSGI http 1 (pid: 717)
web_1  | respawned uWSGI http 1 (pid: 718)
web_1  | respawned uWSGI http 1 (pid: 719)
web_1  | respawned uWSGI http 1 (pid: 720)
web_1  | respawned uWSGI http 1 (pid: 721)
web_1  | respawned uWSGI http 1 (pid: 722)
web_1  | respawned uWSGI http 1 (pid: 723)
web_1  | respawned uWSGI http 1 (pid: 724)
web_1  | respawned uWSGI http 1 (pid: 725)
web_1  | respawned uWSGI http 1 (pid: 726)
web_1  | respawned uWSGI http 1 (pid: 727)
web_1  | respawned uWSGI http 1 (pid: 728)
web_1  | respawned uWSGI http 1 (pid: 729)
web_1  | respawned uWSGI http 1 (pid: 730)
web_1  | respawned uWSGI http 1 (pid: 731)
web_1  | respawned uWSGI http 1 (pid: 732)
web_1  | respawned uWSGI http 1 (pid: 733)
web_1  | respawned uWSGI http 1 (pid: 734)
web_1  | respawned uWSGI http 1 (pid: 735)
web_1  | respawned uWSGI http 1 (pid: 736)
web_1  | respawned uWSGI http 1 (pid: 737)
web_1  | respawned uWSGI http 1 (pid: 738)
web_1  | respawned uWSGI http 1 (pid: 739)
web_1  | respawned uWSGI http 1 (pid: 740)
web_1  | respawned uWSGI http 1 (pid: 741)
web_1  | respawned uWSGI http 1 (pid: 742)
web_1  | respawned uWSGI http 1 (pid: 743)
web_1  | respawned uWSGI http 1 (pid: 744)
web_1  | respawned uWSGI http 1 (pid: 745)
web_1  | respawned uWSGI http 1 (pid: 746)
web_1  | respawned uWSGI http 1 (pid: 747)
web_1  | respawned uWSGI http 1 (pid: 748)
web_1  | respawned uWSGI http 1 (pid: 749)
web_1  | respawned uWSGI http 1 (pid: 750)
web_1  | respawned uWSGI http 1 (pid: 751)
web_1  | respawned uWSGI http 1 (pid: 759)
web_1  | respawned uWSGI http 1 (pid: 760)
web_1  | respawned uWSGI http 1 (pid: 761)
web_1  | respawned uWSGI http 1 (pid: 762)
web_1  | respawned uWSGI http 1 (pid: 763)
web_1  | respawned uWSGI http 1 (pid: 764)
web_1  | respawned uWSGI http 1 (pid: 765)
web_1  | respawned uWSGI http 1 (pid: 766)
web_1  | respawned uWSGI http 1 (pid: 767)
web_1  | respawned uWSGI http 1 (pid: 768)
web_1  | respawned uWSGI http 1 (pid: 769)
web_1  | respawned uWSGI http 1 (pid: 770)
web_1  | respawned uWSGI http 1 (pid: 771)
web_1  | respawned uWSGI http 1 (pid: 772)
web_1  | respawned uWSGI http 1 (pid: 773)
web_1  | respawned uWSGI http 1 (pid: 774)
web_1  | respawned uWSGI http 1 (pid: 775)
web_1  | respawned uWSGI http 1 (pid: 776)
web_1  | respawned uWSGI http 1 (pid: 777)
web_1  | respawned uWSGI http 1 (pid: 778)
web_1  | respawned uWSGI http 1 (pid: 779)
web_1  | respawned uWSGI http 1 (pid: 780)
web_1  | respawned uWSGI http 1 (pid: 781)
web_1  | respawned uWSGI http 1 (pid: 782)
web_1  | respawned uWSGI http 1 (pid: 783)
web_1  | respawned uWSGI http 1 (pid: 784)
web_1  | respawned uWSGI http 1 (pid: 785)
web_1  | respawned uWSGI http 1 (pid: 786)
web_1  | respawned uWSGI http 1 (pid: 787)
web_1  | respawned uWSGI http 1 (pid: 788)
web_1  | respawned uWSGI http 1 (pid: 789)
web_1  | respawned uWSGI http 1 (pid: 790)
web_1  | respawned uWSGI http 1 (pid: 791)
web_1  | respawned uWSGI http 1 (pid: 792)
web_1  | respawned uWSGI http 1 (pid: 800)
web_1  | respawned uWSGI http 1 (pid: 801)
web_1  | respawned uWSGI http 1 (pid: 802)
web_1  | respawned uWSGI http 1 (pid: 803)
web_1  | respawned uWSGI http 1 (pid: 804)
web_1  | respawned uWSGI http 1 (pid: 805)
web_1  | respawned uWSGI http 1 (pid: 806)
web_1  | respawned uWSGI http 1 (pid: 807)
web_1  | respawned uWSGI http 1 (pid: 808)
web_1  | respawned uWSGI http 1 (pid: 809)
web_1  | respawned uWSGI http 1 (pid: 810)
web_1  | respawned uWSGI http 1 (pid: 811)
web_1  | respawned uWSGI http 1 (pid: 812)
web_1  | respawned uWSGI http 1 (pid: 813)
web_1  | respawned uWSGI http 1 (pid: 814)
web_1  | respawned uWSGI http 1 (pid: 815)
web_1  | respawned uWSGI http 1 (pid: 816)
web_1  | respawned uWSGI http 1 (pid: 817)
web_1  | respawned uWSGI http 1 (pid: 818)
web_1  | respawned uWSGI http 1 (pid: 819)
web_1  | respawned uWSGI http 1 (pid: 820)
web_1  | respawned uWSGI http 1 (pid: 821)
web_1  | respawned uWSGI http 1 (pid: 822)
web_1  | respawned uWSGI http 1 (pid: 823)
web_1  | respawned uWSGI http 1 (pid: 824)
web_1  | respawned uWSGI http 1 (pid: 825)
web_1  | respawned uWSGI http 1 (pid: 826)
web_1  | respawned uWSGI http 1 (pid: 827)
web_1  | respawned uWSGI http 1 (pid: 828)
web_1  | respawned uWSGI http 1 (pid: 829)
web_1  | respawned uWSGI http 1 (pid: 830)
web_1  | respawned uWSGI http 1 (pid: 831)
web_1  | respawned uWSGI http 1 (pid: 832)
web_1  | respawned uWSGI http 1 (pid: 833)
web_1  | respawned uWSGI http 1 (pid: 834)
web_1  | respawned uWSGI http 1 (pid: 835)
web_1  | respawned uWSGI http 1 (pid: 836)
web_1  | respawned uWSGI http 1 (pid: 837)
web_1  | respawned uWSGI http 1 (pid: 838)
web_1  | respawned uWSGI http 1 (pid: 839)
web_1  | respawned uWSGI http 1 (pid: 847)
web_1  | respawned uWSGI http 1 (pid: 848)
web_1  | respawned uWSGI http 1 (pid: 849)
web_1  | respawned uWSGI http 1 (pid: 850)
web_1  | respawned uWSGI http 1 (pid: 851)
web_1  | respawned uWSGI http 1 (pid: 852)
web_1  | respawned uWSGI http 1 (pid: 853)
web_1  | respawned uWSGI http 1 (pid: 854)
web_1  | respawned uWSGI http 1 (pid: 855)
web_1  | respawned uWSGI http 1 (pid: 856)
web_1  | respawned uWSGI http 1 (pid: 857)
web_1  | respawned uWSGI http 1 (pid: 858)
web_1  | respawned uWSGI http 1 (pid: 859)
web_1  | respawned uWSGI http 1 (pid: 860)
web_1  | respawned uWSGI http 1 (pid: 861)
web_1  | respawned uWSGI http 1 (pid: 862)
web_1  | respawned uWSGI http 1 (pid: 863)
web_1  | respawned uWSGI http 1 (pid: 864)
web_1  | respawned uWSGI http 1 (pid: 865)
web_1  | respawned uWSGI http 1 (pid: 866)
web_1  | respawned uWSGI http 1 (pid: 867)
web_1  | respawned uWSGI http 1 (pid: 868)
web_1  | respawned uWSGI http 1 (pid: 869)
web_1  | respawned uWSGI http 1 (pid: 870)
web_1  | respawned uWSGI http 1 (pid: 871)
web_1  | respawned uWSGI http 1 (pid: 872)
web_1  | respawned uWSGI http 1 (pid: 873)
web_1  | respawned uWSGI http 1 (pid: 874)
web_1  | respawned uWSGI http 1 (pid: 875)
web_1  | respawned uWSGI http 1 (pid: 876)
web_1  | respawned uWSGI http 1 (pid: 877)
web_1  | respawned uWSGI http 1 (pid: 878)
web_1  | respawned uWSGI http 1 (pid: 879)
web_1  | respawned uWSGI http 1 (pid: 880)
web_1  | respawned uWSGI http 1 (pid: 881)
web_1  | respawned uWSGI http 1 (pid: 882)
web_1  | respawned uWSGI http 1 (pid: 883)
web_1  | respawned uWSGI http 1 (pid: 884)
web_1  | respawned uWSGI http 1 (pid: 885)
web_1  | respawned uWSGI http 1 (pid: 886)
web_1  | respawned uWSGI http 1 (pid: 893)

CONTAINER ID   NAME                                                            CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
e4be6cc44a50   sentry-self-hosted_sentry-cleanup_1                             0.00%     364KiB / 9.707GiB     0.00%     3.82kB / 0B       37.3MB / 682kB    1
ec57e4d363f4   sentry-self-hosted_ingest-replay-recordings_1                   0.10%     86.83MiB / 9.707GiB   0.87%     3.14MB / 2.24MB   904MB / 46.3MB    6
3f58b0442216   sentry-self-hosted_worker_1                                     1.45%     356.7MiB / 9.707GiB   3.59%     5.07MB / 41.2MB   4.3GB / 192MB     12
9bc5732d875a   sentry-self-hosted_post-process-forwarder-errors_1              0.18%     102.3MiB / 9.707GiB   1.03%     6.08MB / 4.36MB   3GB / 37.6MB      11
bb9082f614fd   sentry-self-hosted_subscription-consumer-transactions_1         0.23%     97.25MiB / 9.707GiB   0.98%     3.21MB / 2.32MB   2.47GB / 39.2MB   6
53ae21f40536   sentry-self-hosted_post-process-forwarder-transactions_1        0.28%     88.5MiB / 9.707GiB    0.89%     6.22MB / 4.47MB   3.28GB / 45.8MB   11
099c51ba6e37   sentry-self-hosted_subscription-consumer-events_1               0.21%     71.2MiB / 9.707GiB    0.72%     3.17MB / 2.29MB   2.21GB / 66.3MB   6
d193cb1b1a43   sentry-self-hosted_ingest-consumer_1                            0.19%     95.45MiB / 9.707GiB   0.96%     5.89MB / 3.48MB   858MB / 40MB      6
b664a8dc4b55   sentry-self-hosted_cron_1                                       1.97%     80.91MiB / 9.707GiB   0.81%     346kB / 2.88MB    1.24GB / 88.5MB   3
1d350b263a41   sentry-self-hosted_web_1                                        99.70%    5.618GiB / 9.707GiB   57.88%    25.5kB / 21.1kB   2.21GB / 2.33GB   58
4977419de981   sentry-self-hosted_snuba-outcomes-consumer_1                    0.09%     63.14MiB / 9.707GiB   0.64%     3.03MB / 2.13MB   937MB / 28.6MB    5
92ab8af080f9   sentry-self-hosted_snuba-subscription-consumer-sessions_1       0.24%     54.52MiB / 9.707GiB   0.55%     3.15MB / 2.27MB   2.22GB / 40.5MB   9
ca3eb80a6425   sentry-self-hosted_snuba-consumer_1                             0.39%     40.48MiB / 9.707GiB   0.41%     3.01MB / 2.11MB   1.05GB / 60.4MB   11
6e571806b330   sentry-self-hosted_snuba-transactions-consumer_1                0.30%     27.07MiB / 9.707GiB   0.27%     3.05MB / 2.17MB   765MB / 68.8MB    8
514b96e68d07   sentry-self-hosted_snuba-replacer_1                             0.20%     2.16MiB / 9.707GiB    0.02%     3.1MB / 2.19MB    805MB / 94.7MB    5
4d0928b0c1c5   sentry-self-hosted_snuba-subscription-consumer-transactions_1   0.25%     11.05MiB / 9.707GiB   0.11%     3.17MB / 2.31MB   2.23GB / 85.1MB   9
9d615c943fb3   sentry-self-hosted_snuba-api_1                                  0.01%     44.93MiB / 9.707GiB   0.45%     44.7kB / 75.5kB   1.64GB / 68.2MB   6
b7223ddb4756   sentry-self-hosted_snuba-replays-consumer_1                     0.23%     3.301MiB / 9.707GiB   0.03%     3.12MB / 2.22MB   2.37GB / 91.7MB   9
61c26dc72303   sentry-self-hosted_snuba-sessions-consumer_1                    0.29%     2.441MiB / 9.707GiB   0.02%     3.08MB / 2.18MB   790MB / 92.8MB    8
be8ef685d95b   sentry-self-hosted_snuba-subscription-consumer-events_1         0.23%     2.668MiB / 9.707GiB   0.03%     3.08MB / 2.22MB   2.33GB / 94.3MB   9
1c820bdccf3f   sentry-self-hosted_kafka_1                                      1.16%     364MiB / 9.707GiB     3.66%     39MB / 55.5MB     20.2GB / 409MB    74
b79c71f62e18   sentry-self-hosted_memcached_1                                  0.02%     2.109MiB / 9.707GiB   0.02%     363kB / 236kB     2.25GB / 5.46MB   10
b92822cfb944   sentry-self-hosted_symbolicator-cleanup_1                       0.00%     320KiB / 9.707GiB     0.00%     4.3kB / 0B        71.9MB / 445kB    1
88460a3408af   sentry-self-hosted_clickhouse_1                                 0.07%     1.113GiB / 9.707GiB   11.47%    18.1kB / 9.72kB   31.7GB / 2.09GB   57
d3e2ab915cd2   sentry-self-hosted_symbolicator_1                               0.02%     2.02MiB / 9.707GiB    0.02%     4.3kB / 0B        7.57GB / 82.5MB   18
82be3c0bd559   sentry-self-hosted_redis_1                                      0.28%     2.992MiB / 9.707GiB   0.03%     42.7MB / 4.02MB   8.31GB / 15.2MB   5
c425971977d7   sentry-self-hosted_postgres_1                                   0.02%     30.1MiB / 9.707GiB    0.30%     1.18MB / 1.24MB   11.7GB / 42.8MB   16
17f0cfcfc5e6   sentry-self-hosted_zookeeper_1                                  0.06%     50.89MiB / 9.707GiB   0.51%     135kB / 118kB     6.61GB / 111MB    48
d6d0bd5fc68f   sentry-self-hosted_smtp_1                                       0.03%     796KiB / 9.707GiB     0.01%     4.7kB / 164B      701MB / 1.01MB    2

Event ID

No response

vsfomin commented 1 year ago

Ok, the problem was related to not enough RAM on the host where Sentry was installed. It needs 11Gb+ for working in our case. No other application installed on this server. Now all containers are UP but when I try to enter via web UI, I see this: image image

Can you suggest what I need to do to resolve this?

hubertdeng123 commented 1 year ago

Seems like your free memory is still really low, wondering if that might be the issue here. I'd guess there's something that's going wrong with the health of your containers

vsfomin commented 1 year ago

image Is it normal that web-1 consumes over 8 Gb of RAM or something went wrong?

UPD: I tried to do docker compose stop && docker system prune than install another version [22.7.0] -- same thing.

hubertdeng123 commented 1 year ago

Yeah, that's not normal, mine does not use nearly as much. Is your web container constantly restarting still?

hubertdeng123 commented 1 year ago

Is this a clean install?

vsfomin commented 1 year ago

Is your web container constantly restarting still? No, now sentry works fine after I added additional RAM. Now I have 16Gb. Is this a clean install? I tried to install Sentry on fresh Rocky Linux 9 but did it several times because sentry didn't start correctly. I always do docker compose stop, docker system prune when I want to install another version (not 22.7.0 in my case) or reinstall the same version (23.3.1). When I do docker copose stop, free -h shows me 14Gb available. (total 16Gb) When I try to install it first time, sentry didn't start correctly. Therefore I can make a conclusion that web-1 consumed a lot of memory when I installed it first time too.
Have you some ideas what I can do with that?

Leonardomav commented 1 year ago

I am facing the same problem: Clean install with same Sentry and Docker version and same architecture and RAM (16GB). Sentry web service instantly starts consuming around 8.4GB of RAM. Right now I am using RHEL but I have an older installation with the same sentry version using Debian, where this doesn't happen. Which distro are you using?

Edit: Just noticed you are using Rocky Linux 9, which is based in RHEL. Could there be an issue with the OS used to install docker and sentry?

vsfomin commented 1 year ago

@Leonardomav I'm using Rocky Linux (RHEL). I have older installation of Sentry 20.11.14 on CentOS7 where application doesn't consume a lot of memory. Web-1 consumes there only about 400 Mb of RAM. 

b9c336b13cd6   sentry_onpremise_web_1                                        0.01%     398.2MiB / 43.23GiB   0.90% 
cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)
Leonardomav commented 1 year ago

@vsfomin Same scenario here but with Debian instead of centOS. Have you tried installing the latest sentry 22 release on your rocky Linux instance?

vsfomin commented 1 year ago

@Leonardomav I tried to install 22.7.0 and other versions of sentry. I've faced this issue in all cases.

Leonardomav commented 1 year ago

@vsfomin Changed my setup from RHEL to a new Debian VM with the same specs and versions, and the problem was fixed. I don't fully understand what is causing the issue but it may be something to do with Fedora based linux with the current version of docker.

github-actions[bot] commented 1 year ago

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

hubertdeng123 commented 1 year ago

I'm curious to see if this is an issue after ulimits were added for zookeeper, kafka, and web

https://github.com/getsentry/self-hosted/pull/2123

gsilantyev commented 1 year ago

Hey @hubertdeng123 yes, adding ulimits for web resolve this issue in my case