getsentry / sentry

Developer-first error tracking and performance monitoring
https://sentry.io
Other
39.18k stars 4.2k forks source link

Multipart-HTTP upload of minidump is failing due to a UnreadablePostError #16596

Closed maxp1256 closed 4 years ago

maxp1256 commented 4 years ago

I'm trying to upload a crashdump with the example_crashpad project from the sentry-native sdk to a fresh on premise (docker) instance of sentry v10 (latest version).

I'm getting the following error in the web_1 docker instance and the dump isn't showing up in sentry.

web_1                     | 14:49:34 [INFO] sentry_sdk.errors: event processor (<function processor at 0x7fdbca6f48c0>) dropped event ({'exception': {'values': [{'stacktrace': {'frames': [{'function': '__call__', 'abs_path': '/usr/local/lib/python2.7/site-packages/sentry/wsgi.py', 'pre_context': ['', '    Note: this was added natively into Django 1.8, so if by some reason,', '    we upgraded, this wouldn\'t be relevant anymore."""', '', '    def __call__(self, environ, start_response):'], 'lineno': 36, 'vars': {'start_response': <built-in function uwsgi_spit>, 'self': <sentry.wsgi.FileWrapperWSGIHandler object at 0x7fdbcd577f10>, 'environ': {'wsgi.multiprocess': False, 'SCRIPT_NAME': u'', 'HTTP_TRANSFER_ENCODING': 'chunked', 'REQUEST_METHOD': 'POST', 'UWSGI_ROUTER': 'http', 'SERVER_PROTOCOL': 'HTTP/1.1', 'QUERY_STRING': 'sentry_key=b93d050b55354134b53e4faf42b637bf&guid=ef74196a-9488-46db-88c7-1b93503bfad1', 'CONTENT_LENGTH': '4294967295', 'HTTP_USER_AGENT': 'Crashpad/0.8.0 WinHTTP/10.0.17134.590 Windows_NT/10.0.17134.753 (x86; WoW64)', 'HTTP_CONNECTION': 'Keep-Alive', 'SERVER_NAME': 'b0e10b9cd9d9', 'REMOTE_ADDR': '172.21.0.1', 'wsgi.url_scheme': 'http', 'SERVER_PORT': '9000', 'uwsgi.node': 'b0e10b9cd9d9', 'uwsgi.core': 0, 'wsgi.input': <uwsgi._Input object at 0x7fdbc09de030>, 'HTTP_HOST': 'localhost:9000', 'wsgi.multithread': True, 'REQUEST_URI': '/api/2/minidump/?sentry_key=b93d050b55354134b53e4faf42b637bf&guid=ef74196a-9488-46db-88c7-1b93503bfad1', 'wsgi.version': (1, 0), 'wsgi.run_once': False, 'wsgi.errors': <open file 'wsgi_errors', mode 'w' at 0x7fdbc25d5150>, 'REMOTE_PORT': '15024', 'uwsgi.version': '2.0.18', 'CONTENT_TYPE': 'multipart/form-data; boundary=---MultipartBoundary-xJxM7ASBnI9nYvUX0428SZemdMmw42Mb---', 'wsgi.file_wrapper': <built-in function uwsgi_sendfile>, 'PATH_INFO': u'/api/2/minidump/'}}, 'context_line': '        response = super(FileWrapperWSGIHandler, self).__call__(environ, start_response)', 'post_context': ['        if hasattr(response, "streaming") and response.streaming:', '            try:', '                response = environ["wsgi.file_wrapper"](response.streaming_content)', '            except KeyError:', "                # In our case, we're shipping with uwsgi, so it's safer to assume"], 'module': 'sentry.wsgi', 'filename': 'sentry/wsgi.py'}, {'function': '__call__', 'abs_path': '/usr/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py', 'pre_context': ['', '    def __call__(self, environ, start_response):', '        set_script_prefix(get_script_name(environ))', '        signals.request_started.send(sender=self.__class__, environ=environ)', '        request = self.request_class(environ)'], 'lineno': 157, 'vars': {'start_response': <functools.partial object at 0x7fdbc06c56d8>, 'environ': {'wsgi.multiprocess': False, 'SCRIPT_NAME': u'', 'HTTP_TRANSFER_ENCODING': 'chunked', 'REQUEST_METHOD': 'POST', 'UWSGI_ROUTER': 'http', 'SERVER_PROTOCOL': 'HTTP/1.1', 'QUERY_STRING': 'sentry_key=b93d050b55354134b53e4faf42b637bf&guid=ef74196a-9488-46db-88c7-1b93503bfad1', 'CONTENT_LENGTH': '4294967295', 'HTTP_USER_AGENT': 'Crashpad/0.8.0 WinHTTP/10.0.17134.590 Windows_NT/10.0.17134.753 (x86; WoW64)', 'HTTP_CONNECTION': 'Keep-Alive', 'SERVER_NAME': 'b0e10b9cd9d9', 'REMOTE_ADDR': '172.21.0.1', 'wsgi.url_scheme': 'http', 'SERVER_PORT': '9000', 'uwsgi.node': 'b0e10b9cd9d9', 'uwsgi.core': 0, 'wsgi.input': <uwsgi._Input object at 0x7fdbc09de030>, 'HTTP_HOST': 'localhost:9000', 'wsgi.multithread': True, 'REQUEST_URI': '/api/2/minidump/?sentry_key=b93d050b55354134b53e4faf42b637bf&guid=ef74196a-9488-46db-88c7-1b93503bfad1', 'wsgi.version': (1, 0), 'wsgi.run_once': False, 'wsgi.errors': <open file 'wsgi_errors', mode 'w' at 0x7fdbc25d5150>, 'REMOTE_PORT': '15024', 'uwsgi.version': '2.0.18', 'CONTENT_TYPE': 'multipart/form-data; boundary=---MultipartBoundary-xJxM7ASBnI9nYvUX0428SZemdMmw42Mb---', 'wsgi.file_wrapper': <built-in function uwsgi_sendfile>, 'PATH_INFO': u'/api/2/minidump/'}, 'self': <sentry.wsgi.FileWrapperWSGIHandler object at 0x7fdbcd577f10>, 'request': <WSGIRequest: POST u'/api/2/minidump/?sentry_key=b93d050b55354134b53e4faf42b637bf&guid=ef74196a-9488-46db-88c7-1b93503bfad1'>}, 'context_line': '        response = self.get_response(request)', 'post_context': ['', '        response._handler_class = self.__class__', '', "        status = '%d %s' % (response.status_code, response.reason_phrase)", '        response_headers = [(str(k), str(v)) for k, v in response.items()]'], 'module': 'django.core.handlers.wsgi', 'filename': 'django/core/handlers/wsgi.py'}, {'function': 'get_response', 'abs_path': '/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py', 'pre_context': ['    def get_response(self, request):', '        """Return an HttpResponse object for the given HttpRequest."""', '        # Setup default url resolver for this thread', '        set_urlconf(settings.ROOT_URLCONF)', ''], 'lineno': 124, 'vars': {'self': <sentry.wsgi.FileWrapperWSGIHandler object at 0x7fdbcd577f10>, 'request': <WSGIRequest: POST u'/api/2/minidump/?sentry_key=b93d050b55354134b53e4faf42b637bf&guid=ef74196a-9488-46db-88c7-1b93503bfad1'>}, 'context_line': '        response = self._middleware_chain(request)', 'post_context': ['', '        # This block is only needed for legacy MIDDLEWARE_CLASSES; if', '        # MIDDLEWARE is used, self._response_middleware will be empty.', '        try:', '            # Apply response middleware, regardless of the response'], 'module': 'django.core.handlers.base', 'filename': 'django/core/handlers/base.py'}, {'function': 'inner', 'abs_path': '/usr/local/lib/python2.7/site-packages/django/core/handlers/exception.py', 'pre_context': ['    @wraps(get_response, assigned=available_attrs(get_response))', '    def inner(request):', '        try:', '            response = get_response(request)', '        except Exception as exc:'], 'lineno': 43, 'vars': {'get_response': <bound method FileWrapperWSGIHandler._legacy_get_response of <sentry.wsgi.FileWrapperWSGIHandler object at 0x7fdbcd577f10>>, 'request': <WSGIRequest: POST u'/api/2/minidump/?sentry_key=b93d050b55354134b53e4faf42b637bf&guid=ef74196a-9488-46db-88c7-1b93503bfad1'>, 'exc': UnreadablePostError('unable to receive chunked part',)}, 'context_line': '            response = response_for_exception(request, exc)', 'post_context': ['        return response', '    return inner', '', '', 'def response_for_exception(request, exc):'], 'module': 'django.core.handlers.exception', 'filename': 'django/core/handlers/exception.py'}, {'function': 'response_for_exception', 'abs_path': '/usr/local/lib/python2.7/site-packages/django/core/handlers/exception.py', 'pre_context': ['        # Allow sys.exit() to actually exit. See tickets #1023 and #4701', '        raise', '', '    else:', '        signals.got_request_exception.send(sender=None, request=request)'], 'lineno': 93, 'vars': {'request': <WSGIRequest: POST u'/api/2/minidump/?sentry_key=b93d050b55354134b53e4faf42b637bf&guid=ef74196a-9488-46db-88c7-1b93503bfad1'>, 'exc': UnreadablePostError('unable to receive chunked part',)}, 'context_line': '        response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())', 'post_context': ['', '    # Force a TemplateResponse to be rendered.', "    if not getattr(response, 'is_rendered', True) and callable(getattr(response, 'render', None)):", '        response = response.render()', ''], 'module': 'django.core.handlers.exception', 'filename': 'django/core/handlers/exception.py'}, {'function': 'handle_uncaught_exception', 'abs_path': '/usr/local/lib/python2.7/site-packages/django/core/handlers/exception.py', 'pre_context': ['    if settings.DEBUG:', '        return debug.technical_500_response(request, *exc_info)', '', '    # Return an HttpResponse that displays a friendly error message.', '    callback, param_dict = resolver.resolve_error_handler(500)'], 'lineno': 143, 'vars': {'param_dict': {}, 'callback': <function Error500View at 0x7fdbc1767758>, 'request': <WSGIRequest: POST u'/api/2/minidump/?sentry_key=b93d050b55354134b53e4faf42b637bf&guid=ef74196a-9488-46db-88c7-1b93503bfad1'>, 'resolver': <RegexURLResolver 'sentry.conf.urls' (None:None) ^/>, 'exc_info': (<class 'django.http.request.UnreadablePostError'>, UnreadablePostError('unable to receive chunked part',), <traceback object at 0x7fdbc0796bd8>)}, 'context_line': '    return callback(request, **param_dict)', 'post_context': [], 'module': 'django.core.handlers.exception', 'filename': 'django/core/handlers/exception.py'}, {'function': 'view', 'abs_path': '/usr/local/lib/python2.7/site-packages/django/views/generic/base.py', 'pre_context': ["            if hasattr(self, 'get') and not hasattr(self, 'head'):", '                self.head = self.get', '            self.request = request', '            self.args = args', '            self.kwargs = kwargs'], 'lineno': 68, 'vars': {'initkwargs': {}, 'self': <sentry.web.frontend.error_500.Error500View object at 0x7fdbc0851d50>, 'args': (), 'request': <WSGIRequest: POST u'/api/2/minidump/?sentry_key=b93d050b55354134b53e4faf42b637bf&guid=ef74196a-9488-46db-88c7-1b93503bfad1'>, 'kwargs': {}, 'cls': <class 'sentry.web.frontend.error_500.Error500View'>}, 'context_line': '            return self.dispatch(request, *args, **kwargs)', 'post_context': ['        view.view_class = cls', '        view.view_initkwargs = initkwargs', '', '        # take name and docstring from class', '        update_wrapper(view, cls, updated=())'], 'module': 'django.views.generic.base', 'filename': 'django/views/generic/base.py'}, {'function': 'dispatch', 'abs_path': '/usr/local/lib/python2.7/site-packages/sentry/web/frontend/error_500.py', 'pre_context': ['        context = {}', '        embed_config = self.get_embed_config(request)', '        if embed_config:', '            context["embed_config"] = json.dumps_htmlsafe(embed_config)', ''], 'lineno': 38, 'vars': {'request': <WSGIRequest: POST u'/api/2/minidump/?sentry_key=b93d050b55354134b53e4faf42b637bf&guid=ef74196a-9488-46db-88c7-1b93503bfad1'>, 'self': <sentry.web.frontend.error_500.Error500View object at 0x7fdbc0851d50>, 'embed_config': None, 'context': {}}, 'context_line': '        return render_to_response("sentry/500.html", status=500, context=context, request=request)', 'post_context': [], 'module': 'sentry.web.frontend.error_500', 'filename': 'sentry/web/frontend/error_500.py'}, {'function': 'render_to_response', 'abs_path': '/usr/local/lib/python2.7/site-packages/sentry/web/helpers.py', 'pre_context': ['', '    return loader.render_to_string(template, context=context, request=request)', '', '', 'def render_to_response(template, context=None, request=None, status=200, content_type="text/html"):'], 'lineno': 97, 'vars': {'status': 500, 'request': <WSGIRequest: POST u'/api/2/minidump/?sentry_key=b93d050b55354134b53e4faf42b637bf&guid=ef74196a-9488-46db-88c7-1b93503bfad1'>, 'context': {}, 'content_type': 'text/html', 'template': 'sentry/500.html'}, 'context_line': '    response = HttpResponse(render_to_string(template, context, request))', 'post_context': ['    response.status_code = status', '    response["Content-Type"] = content_type', '    return response'], 'module': 'sentry.web.helpers', 'filename': 'sentry/web/helpers.py'}, {'function': 'render_to_string', 'abs_path': '/usr/local/lib/python2.7/site-packages/sentry/web/helpers.py', 'pre_context': ['    if context and "team" in context and isinstance(context["team"], Team):', '        team = context["team"]', '    else:', '        team = None', ''], 'lineno': 85, 'vars': {'request': <WSGIRequest: POST u'/api/2/minidump/?sentry_key=b93d050b55354134b53e4faf42b637bf&guid=ef74196a-9488-46db-88c7-1b93503bfad1'>, 'template': 'sentry/500.html', 'context': {}, 'team': None}, 'context_line': '    default_context = get_default_context(request, context, team=team)', 'post_context': ['', '    if context is None:', '        context = default_context', '    else:', '        context = dict(context)'], 'module': 'sentry.web.helpers', 'filename': 'sentry/web/helpers.py'}, {'function': 'get_default_context', 'abs_path': '/usr/local/lib/python2.7/site-packages/sentry/web/helpers.py', 'pre_context': ['        if (not existing_context or "TEAM_LIST" not in existing_context) and team:', '            context["TEAM_LIST"] = Team.objects.get_for_user(', '                organization=team.organization, user=request.user, with_projects=True', '            )', ''], 'lineno': 53, 'vars': {'plugins': <sentry.plugins.base.manager.PluginManager object at 0x7fdbc5ef7610>, 'request': <WSGIRequest: POST u'/api/2/minidump/?sentry_key=b93d050b55354134b53e4faf42b637bf&guid=ef74196a-9488-46db-88c7-1b93503bfad1'>, 'project': None, 'existing_context': {}, 'context': {'URL_PREFIX': u'http://localhost:9000', 'SINGLE_ORGANIZATION': True, 'ONPREMISE': True, 'PLUGINS': <sentry.plugins.base.manager.PluginManager object at 0x7fdbc5ef7610>}, 'team': None, 'organization': None, 'options': <module 'sentry.options' from '/usr/local/lib/python2.7/site-packages/sentry/options/__init__.py'>}, 'context_line': '        user = request.user', 'post_context': ['    else:', '        user = AnonymousUser()', '', '    if organization:', '        context["selectedOrganization"] = serialize(organization, user)'], 'module': 'sentry.web.helpers', 'filename': 'sentry/web/helpers.py'}]}, 'type': 'AttributeError', 'value': u"'WSGIRequest' object has no attribute 'user'", 'module': 'exceptions', 'mechanism': {'type': 'excepthook', 'handled': False}}]}, 'level': 'error', 'event_id': 'c227eb9962aa40498b4b609e512d63d9', 'timestamp': datetime.datetime(2020, 1, 23, 14, 49, 34, 995911), 'extra': {'sys.argv': ['uwsgi']}, 'modules': {'googleapis-common-protos': '1.6.0', 'django-crispy-forms': '1.7.2', 'cffi': '1.13.2', 'percy': '2.0.2', 'isodate': '0.6.0', 'querystring-parser': '1.2.4', 'jmespath': '0.9.4', 'python-u2flib-server': '5.0.0', 'confluent-kafka': '0.11.5', 'sqlparse': '0.2.4', 'oauthlib': '3.1.0', 'selenium': '3.141.0', 'celery': '3.1.18', 'djangorestframework': '3.6.4', 'mmh3': '2.3.1', 'pyjwt': '1.5.3', 'statsd': '3.1', 'cryptography': '2.8', 'python': '2.7.16', 'pycparser': '2.19', 'structlog': '16.1.0', 'maxminddb': '1.4.1', 'billiard': '3.3.0.23', 'httplib2': '0.16.0', 'decorator': '4.4.1', 'unidiff': '0.5.5', 'requests-oauthlib': '0.3.3', 'funcsigs': '1.0.2', 'setproctitle': '1.1.10', 'jsonschema': '2.6.0', 'phonenumberslite': '7.7.5', 'google-cloud-bigtable': '0.32.2', 'datadog': '0.30.0', 'msgpack': '0.6.2', 'sentry-sdk': '0.14.1', 'redis': '2.10.5', 'google-api-core': '1.14.3', 'docutils': '0.16', 'rb': '1.7', 'milksnake': '0.1.5', 'google-auth': '1.6.3', 'uwsgi': '2.0.18', 'google-cloud-storage': '1.13.3', 'grpc-google-iam-v1': '0.11.4', 'cached-property': '1.5.1', 'soupsieve': '1.9.5', 'mistune': '0.8.4', 'pyyaml': '3.11', 'django-sudo': '3.1.0', 'cachetools': '3.1.1', 'wsgiref': '0.1.2', 'cssselect': '1.1.0', 'requests': '2.20.1', 'defusedxml': '0.5.0', 'pyasn1': '0.4.8', 'pillow': '6.2.2', 'phabricator': '0.7.0', 'cssutils': '1.0.2', 'symbolic': '7.0.0', 'boto3': '1.4.5', 'ua-parser': '0.7.3', 'qrcode': '5.3', 'petname': '2.6', 'botocore': '1.5.70', 'python-utils': '2.3.0', 'redis-py-cluster': '1.3.4', 'ipaddress': '1.0.23', 'pkgconfig': '1.5.1', 'semaphore': '0.4.65', 'certifi': '2019.11.28', 'simplejson': '3.8.2', 'rsa': '4.0', 'pytz': '2019.3', 'parsimonious': '0.8.0', 'functools32': '3.2.3.post2', 'python-dateutil': '2.8.1', 'google-cloud-pubsub': '0.35.4', 'oauth2': '1.9.0.post1', 'sentry': '10.1.0.dev0', 'backports.functools-lru-cache': '1.6.1', 'chardet': '3.0.4', 'hiredis': '0.1.6', 'setuptools': '41.4.0', 'google-resumable-media': '0.4.1', 'grpcio': '1.26.0', 'progressbar2': '3.10.1', 'loremipsum': '1.0.5', 'pip': '19.3.1', 'lxml': '4.3.5', 'amqp': '1.4.9', 'six': '1.10.0', 'django-picklefield': '1.0.0', 'click': '6.7', 'kombu': '3.0.35', 'croniter': '0.3.31', 'mock': '3.0.5', 'wheel': '0.33.6', 'python3-saml': '1.4.1', 'protobuf': '3.11.2', 'librabbitmq': '1.6.1', 'email-reply-parser': '0.2.0', 'urllib3': '1.24.2', 'strict-rfc3339': '0.7', 'google-cloud-core': '0.29.1', 'beautifulsoup4': '4.7.1', 'pyasn1-modules': '0.2.8', 'anyjson': '0.3.3', 'xmlsec': '1.3.3', 'psycopg2-binary': '2.8.4', 'enum34': '1.1.6', 'futures': '3.3.0', 's3transfer': '0.1.13', 'toronado': '0.0.11', 'django': '1.11.27', 'pyopenssl': '19.1.0', 'python-memcached': '1.59', 'idna': '2.7'}, 'contexts': {'runtime': {'version': '2.7.16', 'name': 'CPython', 'build': '2.7.16 (default, Oct 17 2019, 07:39:30) \n[GCC 8.3.0]'}}, 'breadcrumbs': []})
web_1                     | Traceback (most recent call last):
web_1                     |   File "/usr/local/lib/python2.7/site-packages/sentry/wsgi.py", line 36, in __call__
web_1                     |     response = super(FileWrapperWSGIHandler, self).__call__(environ, start_response)
web_1                     |   File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/django/__init__.py", line 108, in sentry_patched_wsgi_handler
web_1                     |     return SentryWsgiMiddleware(bound_old_app)(environ, start_response)
web_1                     |   File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/wsgi.py", line 126, in __call__
web_1                     |     reraise(*_capture_exception(hub))
web_1                     |   File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/wsgi.py", line 122, in __call__
web_1                     |     _sentry_start_response, start_response, span
web_1                     |   File "/usr/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 157, in __call__
web_1                     |     response = self.get_response(request)
web_1                     |   File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/django/__init__.py", line 142, in sentry_patched_get_response
web_1                     |     return old_get_response(self, request)
web_1                     |   File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 124, in get_response
web_1                     |     response = self._middleware_chain(request)
web_1                     |   File "/usr/local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 43, in inner
web_1                     |     response = response_for_exception(request, exc)
web_1                     |   File "/usr/local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 93, in response_for_exception
web_1                     |     response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
web_1                     |   File "/usr/local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 143, in handle_uncaught_exception
web_1                     |     return callback(request, **param_dict)
web_1                     |   File "/usr/local/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
web_1                     |     return self.dispatch(request, *args, **kwargs)
web_1                     |   File "/usr/local/lib/python2.7/site-packages/sentry/web/frontend/error_500.py", line 38, in dispatch
web_1                     |     return render_to_response("sentry/500.html", status=500, context=context, request=request)
web_1                     |   File "/usr/local/lib/python2.7/site-packages/sentry/web/helpers.py", line 97, in render_to_response
web_1                     |     response = HttpResponse(render_to_string(template, context, request))
web_1                     |   File "/usr/local/lib/python2.7/site-packages/sentry/web/helpers.py", line 85, in render_to_string
web_1                     |     default_context = get_default_context(request, context, team=team)
web_1                     |   File "/usr/local/lib/python2.7/site-packages/sentry/web/helpers.py", line 53, in get_default_context
web_1                     |     user = request.user
web_1                     | AttributeError: 'WSGIRequest' object has no attribute 'user'
web_1                     | 172.21.0.1 - - [23/Jan/2020:14:49:30 +0000] "POST /api/2/minidump/?sentry_key=b93d050b55354134b53e4faf42b637bf&guid=ef74196a-9488-46db-88c7-1b93503bfad1 HTTP/1.1" 500 0 "-" "Crashpad/0.8.0 WinHTTP/10.0.17134.590 Windows_NT/10.0.17134.753 (x86; WoW64)"

I've reduced the sentry-configuration in example_crashpad.c to the following:

  const char *handler_path = "crashpad_handler.exe";
  sentry_options_t *options = sentry_options_new();
  sentry_options_set_dsn(options, "http://b93d050b55354134b53e4faf42b637bf@localhost:9000/2");
  sentry_options_set_handler_path(options, handler_path);
  sentry_init(options);

  *((unsigned int*)0) = 0xDEAD;
  sentry_shutdown();

The relevant line for the first part seems to be this one: https://github.com/getsentry/sentry/blob/825661ff3ba6aa64d2fd3ef8abd0b2961fe45cba/src/sentry/web/helpers.py#L53

I've found another error which seems to be the primary problem.

I've created a sentry.io account and uploaded the minidump successfully so it seems to be a problem with the on-prem installation or a bug in the on-prem version.

What's wrong with my configuration?

maxp1256 commented 4 years ago

There's another error in the Sentry Event-List which seems to be the primary problem. The first one (above) seems to be a problem while generating the HTTP-500 Error.

UnreadablePostError: unable to receive chunked part
  File "django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "django/core/handlers/base.py", line 244, in _legacy_get_response
    response = middleware_method(request)
  File "sentry/middleware/auth.py", line 46, in process_request
    user = process_signature(request)
  File "sentry/utils/linksign.py", line 44, in process_signature
    sig = request.GET.get("_") or request.POST.get("_sentry_request_signature")
  File "django/core/handlers/wsgi.py", line 126, in _get_post
    self._load_post_and_files()
  File "django/http/request.py", line 300, in _load_post_and_files
    self._post, self._files = self.parse_file_upload(self.META, data)
  File "django/http/request.py", line 259, in parse_file_upload
    return parser.parse()
  File "django/http/multipartparser.py", line 156, in parse
    for item_type, meta_data, field_stream in Parser(stream, self._boundary):
  File "django/http/multipartparser.py", line 652, in __iter__
    for sub_stream in boundarystream:
  File "django/utils/six.py", line 558, in next
    return type(self).__next__(self)
  File "django/http/multipartparser.py", line 473, in __next__
    return LazyStream(BoundaryIter(self._stream, self._boundary))
  File "django/http/multipartparser.py", line 500, in __init__
    unused_char = self._stream.read(1)
  File "django/http/multipartparser.py", line 369, in read
    out = b''.join(parts())
  File "django/http/multipartparser.py", line 360, in parts
    chunk = next(self)
  File "django/utils/six.py", line 558, in next
    return type(self).__next__(self)
  File "django/http/multipartparser.py", line 384, in __next__
    output = next(self._producer)
  File "django/utils/six.py", line 558, in next
    return type(self).__next__(self)
  File "django/http/multipartparser.py", line 448, in __next__
    data = self.flo.read(self.chunk_size)
  File "django/http/request.py", line 334, in read
    six.reraise(UnreadablePostError, UnreadablePostError(*e.args), sys.exc_info()[2])
  File "django/http/request.py", line 332, in read
    return self._stream.read(*args, **kwargs)
  File "sentry/middleware/proxy.py", line 32, in readinto
    self._internal_buffer = uwsgi.chunked_read()
maxp1256 commented 4 years ago

The problem is caused by the multipart upload feature (chunked requests). You can reproduce the error with curl -X POST "http://localhost:9000/api/6/minidump/?sentry_key=14c00235201d43a7af58ac9fdc368334&guid=9e3f0f2f-cdd0-4229-8ef8-a06c22bd391f" -F guid="9e3f0f2f-cdd0-4229-8ef8-a06c22bd391f" -F upload_file_minidump=@"/C/Users/maxp/Source/Repos/sentrysdk/gen_windows/bin/Debug/.sentry-native/reports/9e3f0f2f-cdd0-4229-8ef8-a06c22bd391f.dmp" -H transfer-encoding:chunked

while the following curl upload without multipart succeeds:

curl -X POST "http://localhost:9000/api/6/minidump/?sentry_key=14c00235201d43a7af58ac9fdc368334&guid=9e3f0f2f-cdd0-4229-8ef8-a06c22bd391f" -F guid="9e3f0f2f-cdd0-4229-8ef8-a06c22bd391f" -F upload_file_minidump=@"/C/Users/maxp/Source/Repos/sentrysdk/gen_windows/bin/Debug/.sentry-native/reports/9e3f0f2f-cdd0-4229-8ef8-a06c22bd391f.dmp"

I didn't find a possibility to disable in in the crashpad client and I don't think that it's a good idea to handle it on the client side because it would make problems with large dumpfiles and slow internet connections.

joshuarli commented 4 years ago

@maxp1256 Thanks for the detailed report, I'll probably have a fix for the 500 thing soon.

Regarding chunked uploads, what do you have in SENTRY_WEB_OPTIONS in your sentry.conf.py?

BYK commented 4 years ago

@joshuarli

Regarding chunked uploads, what do you have in SENTRY_WEB_OPTIONS in your sentry.conf.py?

Most probably these: https://github.com/getsentry/onpremise/blob/bc6d3b47e257057587e29153947c1ba223160416/sentry/sentry.conf.example.py#L153-L163

Which should be fine.

joshuarli commented 4 years ago

I'm still working on reproduction, but it's possible that uwsgi might be misconfigured. We have some tricky middleware in place (ChunkedMiddleware) and devserver's configured with

'protocol': 'http',
'http-chunked-input': True,

See https://github.com/getsentry/sentry/commit/8f9648df60040b89c1c95ca07b70b1007e5ed31b#diff-ca47edb6f8d573fb92cb4991460e5c99R89.

maxp1256 commented 4 years ago

@joshuarli my SENTRY_WEB_OPTIONS are default. I'm currently testing it with your settings. Thanks for your help, I'm hoping that there'll be a fix soon :-)

SENTRY_WEB_OPTIONS = {
    "http": "%s:%s" % (SENTRY_WEB_HOST, SENTRY_WEB_PORT),
    "protocol": "uwsgi",
    # This is needed to prevent https://git.io/fj7Lw
    "uwsgi-socket": None,
    "http-keepalive": True,
    "memory-report": False,
    # 'workers': 3,  # the number of web workers
}
franckchateau commented 4 years ago

Hello, I am facing the same issue, so I am following the thread

maxp1256 commented 4 years ago

I've completely cleared my SENTRY_WEB_OPTIONS and it's working now. But I think it should be fixed anyway to get it working with the default settings.

SENTRY_WEB_HOST = "0.0.0.0"
SENTRY_WEB_PORT = 9000
SENTRY_WEB_OPTIONS = {
    #"http": "%s:%s" % (SENTRY_WEB_HOST, SENTRY_WEB_PORT),
    #"protocol": "uwsgi",
    # This is needed to prevent https://git.io/fj7Lw
    #"uwsgi-socket": None,
    #"http-keepalive": True,
    #"memory-report": False,
    # 'workers': 3,  # the number of web workers
}
joshuarli commented 4 years ago

Hmm, curious. Sadly I wasn't able to reproduce with devserver, but I have yet to stand up an onpremise installation to test against... I'm gonna guess uwsgi is configured differently between the two, somehow.

Amphaal commented 4 years ago

I can also confirm that disabling everything in SENTRY_WEB_OPTIONS "fixes" it ! Many thx to @maxp1256 for the workaround, I can now work on my debug information file workflow ;)

BYK commented 4 years ago

Quite sure this is due to lack of 'http-chunked-input': True,. @Amphaal, @maxp1256 can you bring back those optons but add 'http-chunked-input': True, to them and see if it fixes the issue?

Amphaal commented 4 years ago

Re-running ./install.sh with default parameters in sentry.conf.py + "http-chunked-input": True seems to work fine as well for me. @maxp1256, can you confirm it works with this setup ?

joshuarli commented 4 years ago

That would make sense, since uwsgi.chunked_read() should only be available if "http-chunked-input": True. I however removed that while testing with devserver, and it didn't error on me... maybe it's not actually being configured, will have to look more closely soon.