INTERNAL: processing_issue.bad_report

hheexx commented 1 year ago

Self-Hosted Version

22.12.0

CPU Architecture

x86_64

Docker Version

20.10.21

Docker Compose Version

2.14.1

Steps to Reproduce

This issue started appearing in INTERNAL project in large quantities:

Expected Result

No errors.

Actual Result

Error

Event ID

No response

hubertdeng123 commented 1 year ago

I'd assume this is some sort of problem with your setup. Is this a fresh install? Can you post symbolicator logs?

Rassilion commented 1 year ago

We have this issue too. I think it's appeared after upgrade (all events on 22.12)

We are using addtional redis config to lower ram and disk usage, maybe these effecting somethink in new version?

 redis-server  --maxmemory 4294967296 --maxmemory-policy volatile-ttl --save 120 100000

Also symbolicator logs empty (sudo docker compose logs symbolicator shows no result), is this normal? do I need to set some enviroment variable?

event json

```js { "event_id":"d7d22f9624de4332b0d4d1cad64cb84d", "project":1, "release":"backend@22.12.0+aa3ab3371076eef2aebec38654fba457e4bb16ad", "dist":null, "platform":"python", "message":"processing_issue.bad_report", "datetime":"2023-01-18T20:52:11.858924Z", "tags":[ [ "celery_task_id", "96fc7d2b-ccfa-445a-bc42-691591bba4ce" ], [ "delete_default_cache", "true" ], [ "environment", "production" ], [ "get_default_cache", "true" ], [ "level", "error" ], [ "logger", "sentry.events" ], [ "processing_event_for_project", "6" ], [ "project", "6" ], [ "runtime", "CPython 3.8.13" ], [ "runtime.name", "CPython" ], [ "release", "backend@22.12.0+aa3ab3371076eef2aebec38654fba457e4bb16ad" ], [ "server_name", "92ab049b0999" ], [ "task_name", "sentry.tasks.store.symbolicate_event" ], [ "transaction", "sentry.tasks.store.symbolicate_event" ], [ "used_default_cache", "true" ] ], "_metrics":{ "bytes.ingested.event":6306, "bytes.stored.event":7068 }, "breadcrumbs":{ "values":[ { "timestamp":1674075131.738312, "type":"redis", "category":"redis", "level":"info", "message":"GET 'c:1:e:4b3494587e1b4426a17b95b3b671b71e:6'", "data":{ "redis.command":"GET", "redis.is_cluster":false, "redis.key":"c:1:e:4b3494587e1b4426a17b95b3b671b71e:6" } }, { "timestamp":1674075131.752605, "type":"redis", "category":"redis", "level":"info", "message":"GET 'c:1:symbolicator:4b3494587e1b4426a17b95b3b671b71e:6'", "data":{ "redis.command":"GET", "redis.is_cluster":false, "redis.key":"c:1:symbolicator:4b3494587e1b4426a17b95b3b671b71e:6" } }, { "timestamp":1674075131.763135, "type":"http", "category":"httplib", "level":"info", "data":{ "method":"POST", "reason":"OK", "status_code":200, "url":"http://symbolicator:3021/symbolicate?timeout=5&scope=6" } }, { "timestamp":1674075131.765405, "type":"redis", "category":"redis", "level":"info", "message":"DEL 'c:1:symbolicator:4b3494587e1b4426a17b95b3b671b71e:6'", "data":{ "redis.command":"DEL", "redis.is_cluster":false, "redis.key":"c:1:symbolicator:4b3494587e1b4426a17b95b3b671b71e:6" } }, { "timestamp":1674075131.765996, "type":"log", "category":"sentry.events", "level":"error", "message":"processing_issue.bad_report", "data":{ "platform":"csharp" } }, { "timestamp":1674075131.782073, "type":"log", "category":"sentry.events", "level":"error", "message":"processing_issue.bad_report", "data":{ "platform":"csharp" } }, { "timestamp":1674075131.801192, "type":"log", "category":"sentry.events", "level":"error", "message":"processing_issue.bad_report", "data":{ "platform":"csharp" } }, { "timestamp":1674075131.819046, "type":"log", "category":"sentry.events", "level":"error", "message":"processing_issue.bad_report", "data":{ "platform":"csharp" } }, { "timestamp":1674075131.83831, "type":"log", "category":"sentry.events", "level":"error", "message":"processing_issue.bad_report", "data":{ "platform":"csharp" } } ] }, "contexts":{ "runtime":{ "name":"CPython", "version":"3.8.13", "build":"3.8.13 (default, Aug 23 2022, 09:42:19) \n[GCC 10.2.1 20210110]", "type":"runtime" }, "trace":{ "trace_id":"d769c9612d7f4ef2ae8ae8dfbd064112", "span_id":"a272fcf5d4a35703", "parent_span_id":"bddaf7e43b2e9a9d", "op":"tasks.store.symbolicate_event.process_payload", "status":"unknown", "type":"trace" } }, "culprit":"sentry.tasks.store.symbolicate_event", "environment":"production", "extra":{ "celery-job":{ "args":[ ], "kwargs":{ "cache_key":"e:4b3494587e1b4426a17b95b3b671b71e:6", "event_id":"4b3494587e1b4426a17b95b3b671b71e", "has_attachments":false, "queue_switches":0, "start_time":1674075130.0 }, "task_name":"sentry.tasks.store.symbolicate_event" }, "platform":"csharp", "sys.argv":[ "/usr/local/bin/sentry", "run", "worker" ] }, "fingerprint":[ "{{ default }}" ], "grouping_config":{ "enhancements":"eJybzDRxY3J-bm5-npWRgaGlroGxrpHxBABcYgcZ", "id":"newstyle:2019-10-29" }, "hashes":[ "ed4de006a7407a1f717dcaed6486f8c6" ], "key_id":"1", "level":"error", "logentry":{ "formatted":"processing_issue.bad_report", "params":[ ] }, "logger":"sentry.events", "metadata":{ "title":"processing_issue.bad_report" }, "modules":{ "amqp":"2.6.1", "async-generator":"1.10", "attrs":"19.2.0", "avalara":"20.9.0", "beautifulsoup4":"4.7.1", "billiard":"3.6.4.0", "boto3":"1.22.12", "botocore":"1.25.12", "brotli":"1.0.9", "cachetools":"4.2.4", "celery":"4.4.7", "certifi":"2022.5.18.1", "cffi":"1.15.1", "chardet":"4.0.0", "click":"8.0.4", "confluent-kafka":"1.9.2", "croniter":"0.3.37", "cryptography":"38.0.3", "cssselect":"1.0.3", "cssutils":"2.4.0", "datadog":"0.29.3", "decorator":"5.1.1", "django":"2.2.28", "django-crispy-forms":"1.14.0", "django-pg-zero-downtime-migrations":"0.11", "djangorestframework":"3.12.4", "drf-spectacular":"0.22.1", "email-reply-parser":"0.5.12", "fido2":"0.9.2", "google-api-core":"2.10.1", "google-auth":"1.35.0", "google-cloud-bigtable":"2.11.3", "google-cloud-core":"2.3.2", "google-cloud-functions":"1.8.1", "google-cloud-pubsub":"2.13.6", "google-cloud-spanner":"3.20.0", "google-cloud-storage":"2.5.0", "google-crc32c":"1.3.0", "google-resumable-media":"2.3.3", "googleapis-common-protos":"1.56.4", "grpc-google-iam-v1":"0.12.4", "grpcio":"1.48.1", "grpcio-status":"1.48.1", "h11":"0.13.0", "hiredis":"0.3.1", "idna":"2.10", "inflection":"0.5.1", "isodate":"0.6.1", "jmespath":"0.10.0", "jsonschema":"3.2.0", "kombu":"4.6.11", "lxml":"4.6.5", "maxminddb":"2.0.3", "milksnake":"0.1.5", "mistune":"2.0.4", "mmh3":"3.0.0", "msgpack":"1.0.4", "natsort":"8.1.0", "oauthlib":"3.1.0", "outcome":"1.2.0", "packaging":"21.3", "parsimonious":"0.8.0", "petname":"2.6", "phabricator":"0.7.0", "phonenumberslite":"8.12.0", "pillow":"9.2.0", "pip":"22.0.4", "planout":"0.6.0", "progressbar2":"3.41.0", "proto-plus":"1.22.1", "protobuf":"4.21.6", "psycopg2-binary":"2.8.6", "pyasn1":"0.4.5", "pyasn1-modules":"0.2.4", "pycountry":"17.5.14", "pycparser":"2.21", "pyjwt":"2.4.0", "pyopenssl":"22.0.0", "pyparsing":"3.0.9", "pyrsistent":"0.18.1", "pysocks":"1.7.1", "python-dateutil":"2.8.1", "python-memcached":"1.59", "python-rapidjson":"1.4", "python-u2flib-server":"5.0.0", "python-utils":"3.3.3", "python3-saml":"1.14.0", "pytz":"2018.9", "pyuwsgi":"2.0.20", "pyvat":"1.3.15", "pyyaml":"5.4", "rb":"1.9.0", "redis":"3.4.1", "redis-py-cluster":"2.1.0", "reportlab":"3.6.1", "requests":"2.25.1", "requests-oauthlib":"1.2.0", "rfc3339-validator":"0.1.2", "rfc3986-validator":"0.1.1", "rsa":"4.8", "s3transfer":"0.5.2", "selenium":"4.3.0", "sentry":"22.12.0", "sentry-arroyo":"2.2.0", "sentry-relay":"0.8.15", "sentry-sdk":"1.11.0", "setuptools":"56.0.0", "simplejson":"3.17.6", "six":"1.16.0", "sniffio":"1.2.0", "snuba-sdk":"1.0.1", "sortedcontainers":"2.4.0", "soupsieve":"2.3.2.post1", "sqlparse":"0.3.0", "statsd":"3.3.0", "stripe":"2.61.0", "structlog":"21.1.0", "symbolic":"10.2.0", "toronado":"0.1.0", "trio":"0.21.0", "trio-websocket":"0.9.2", "typing-extensions":"3.10.0.2", "ua-parser":"0.10.0", "unidiff":"0.7.4", "uritemplate":"4.1.1", "urllib3":"1.26.11", "vine":"1.3.0", "wheel":"0.37.1", "wsproto":"1.1.0", "xmlsec":"1.3.12", "zstandard":"0.18.0" }, "nodestore_insert":1674075133.611053, "received":1674075131.928474, "sdk":{ "name":"sentry.python.django", "version":"1.11.0", "integrations":[ "argv", "atexit", "celery", "dedupe", "django", "django_atomic", "excepthook", "logging", "modules", "redis", "rust_info", "stdlib", "threading" ], "packages":[ { "name":"pypi:sentry-sdk", "version":"1.11.0" } ] }, "timestamp":1674075131.858924, "title":"processing_issue.bad_report", "transaction":"sentry.tasks.store.symbolicate_event", "transaction_info":{ }, "type":"default", "version":"7", "location":null } ```

github-actions[bot] commented 1 year ago

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

pstumpe commented 1 year ago

I have this issue too after upgrading the docker stack from 22.12.0 to 23.1.1. Not quite sure where to actually look for details. The logs from the symbolicator container show a lot of this:

2023-02-09T08:18:20.004491914Z ERROR symbolicator_service::services::objects::data_cache: Error while downloading file stderr=download was cancelled

RobotGizmo commented 1 year ago

We are also having this issue after upgrading, if you let me know what you need to help troubleshoot this I'll be happy to look it up.

roalter commented 1 year ago

Same issue here, still working with 22.12.0, no upgrade.

Any updates?

roalter commented 1 year ago

Maybe related to https://github.com/getsentry/self-hosted/issues/1580. I added the proxy-environment with a NO_PROXY: sentry to the symbolicator. Maybe this could be the clue.

getsantry[bot] commented 1 year ago

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

hubertdeng123 commented 1 year ago

No updates here. Are you all still experiencing this issue? Wondering what RAM and disk space usage are looking like

getsantry[bot] commented 1 year ago

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

getsentry / self-hosted