How to monitor Sentry freezes with command line tools?

[its-apashnin]

Environment

self-hosted (https://develop.sentry.dev/self-hosted/)

What are you trying to accomplish?

There's a problem while Sentry doesn't show issues. The Sentry is containerized and the only way to fix the problem is restart the container. It looks like Sentry receives data and doesn't send data to the monitor pages. Primary goal is to monitor that problem and manually restart container while it occurs. Secondary goal is to prevent such behaviour and avoid "freezes".

How are you getting stuck?

-

Where in the product are you?

Unknown

Link

No response

DSN

No response

Version

No response

[getsantry[bot]]

Assigning to @getsentry/support for routing ⏲️

[joshuarli]

Can you provide more information? What version of self-hosted are you on, and what do sentry's container logs say when you try and load the issues page?

[streicherishere]

Hi, I have the same problem. It happens every 2-3 days. The Sentry is containerised and the only way to fix it is to stop and start the container.

Sentry Version: self-hosted-24.6.0

load page: https://sentry.kunde.com/organizations/kunde/issues/
docker logs sentry-self-hosted-web-1 -f

 token_type='None' is_frontend_request='True' organization_id='1' auth_id='None' path='/api/0/organizations/kunde/issues/' caller_ip='178.19.0.1' user_agent='Mozilla/5.0 (X11; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0' rate_limited='False' rate_limit_category='None' request_duration_seconds=1.1726279258728027 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
[uwsgi-http key: localhost client_addr: 172.18.0.31 client_port: 39087] hr_write(): Broken pipe [plugins/http/http.c line 565]
12:35:38 [INFO] sentry.access.api: api.access (method='GET' view='sentry.issues.endpoints.organization_group_index.OrganizationGroupIndexEndpoint' response=200 user_id='1' is_app='False' token_type='None' is_frontend_request='True' organization_id='1' auth_id='None' path='/api/0/organizations/kunde/issues/' caller_ip='178.19.0.1' user_agent='Mozilla/5.0 (X11; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.9935879707336426 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
12:35:38 [INFO] sentry.access.api: api.access (method='GET' view='sentry.replays.endpoints.organization_replay_count.OrganizationReplayCountEndpoint' response=200 user_id='1' is_app='False' token_type='None' is_frontend_request='True' organization_id='1' auth_id='None' path='/api/0/organizations/kunde/replay-count/' caller_ip='178.19.0.1' user_agent='Mozilla/5.0 (X11; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.06168317794799805 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
12:35:38 [INFO] sentry.access.api: api.access (method='GET' view='sentry.api.endpoints.organization_issues_count.OrganizationIssuesCountEndpoint' response=200 user_id='1' is_app='False' token_type='None' is_frontend_request='True' organization_id='1' auth_id='None' path='/api/0/organizations/kunde/issues-count/' caller_ip='178.19.0.1' user_agent='Mozilla/5.0 (X11; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.445507287979126 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
12:35:38 [INFO] sentry.access.api: api.access (method='GET' view='sentry.api.endpoints.organization_group_index_stats.OrganizationGroupIndexStatsEndpoint' response=200 user_id='1' is_app='False' token_type='None' is_frontend_request='True' organization_id='1' auth_id='None' path='/api/0/organizations/kunde/issues-stats/' caller_ip='178.19.0.1' user_agent='Mozilla/5.0 (X11; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.5527019500732422 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')

env.custom

COMPOSE_PROJECT_NAME=sentry-self-hosted
SENTRY_EVENT_RETENTION_DAYS=30
# You can either use a port number or an IP:PORT combo for SENTRY_BIND
# See https://docs.docker.com/compose/compose-file/#ports for more
SENTRY_BIND=9000
# Set SENTRY_MAIL_HOST to a valid FQDN (host/domain name) to be able to send emails!
# SENTRY_MAIL_HOST=example.com
SENTRY_IMAGE=getsentry/sentry:24.6.0
SNUBA_IMAGE=getsentry/snuba:24.6.0
RELAY_IMAGE=getsentry/relay:24.6.0
SYMBOLICATOR_IMAGE=getsentry/symbolicator:24.6.0
VROOM_IMAGE=getsentry/vroom:24.6.0
WAL2JSON_VERSION=latest
HEALTHCHECK_INTERVAL=5m
HEALTHCHECK_TIMEOUT=30s
HEALTHCHECK_RETRIES=3
# Mail Settings
SENTRY_MAIL_HOST=sentry2.kunde.com
# Caution: Raising max connections of postgres increases CPU and RAM usage
# see https://github.com/getsentry/self-hosted/pull/2740 for more information
POSTGRES_MAX_CONNECTIONS=100

[aldy505]

Hi, I have the same problem. It happens every 2-3 days. The Sentry is containerised and the only way to fix it is to stop and start the container.

Hi @streicherishere, can you specify more details on what happened, since you said that "you have the same problem"? Do you also have the need to monitor Sentry freezes via CLI?

[joshuarli]

Can you post more logs? Is there a python exception involved anywhere, or is the webserver really just hanging and not doing anything (the uwsgi broken pipe there is usually the underlying webserver timing out)

[getsantry[bot]]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀