Closed maxp1256 closed 4 years ago
There's another error in the Sentry Event-List which seems to be the primary problem. The first one (above) seems to be a problem while generating the HTTP-500 Error.
UnreadablePostError: unable to receive chunked part
File "django/core/handlers/exception.py", line 41, in inner
response = get_response(request)
File "django/core/handlers/base.py", line 244, in _legacy_get_response
response = middleware_method(request)
File "sentry/middleware/auth.py", line 46, in process_request
user = process_signature(request)
File "sentry/utils/linksign.py", line 44, in process_signature
sig = request.GET.get("_") or request.POST.get("_sentry_request_signature")
File "django/core/handlers/wsgi.py", line 126, in _get_post
self._load_post_and_files()
File "django/http/request.py", line 300, in _load_post_and_files
self._post, self._files = self.parse_file_upload(self.META, data)
File "django/http/request.py", line 259, in parse_file_upload
return parser.parse()
File "django/http/multipartparser.py", line 156, in parse
for item_type, meta_data, field_stream in Parser(stream, self._boundary):
File "django/http/multipartparser.py", line 652, in __iter__
for sub_stream in boundarystream:
File "django/utils/six.py", line 558, in next
return type(self).__next__(self)
File "django/http/multipartparser.py", line 473, in __next__
return LazyStream(BoundaryIter(self._stream, self._boundary))
File "django/http/multipartparser.py", line 500, in __init__
unused_char = self._stream.read(1)
File "django/http/multipartparser.py", line 369, in read
out = b''.join(parts())
File "django/http/multipartparser.py", line 360, in parts
chunk = next(self)
File "django/utils/six.py", line 558, in next
return type(self).__next__(self)
File "django/http/multipartparser.py", line 384, in __next__
output = next(self._producer)
File "django/utils/six.py", line 558, in next
return type(self).__next__(self)
File "django/http/multipartparser.py", line 448, in __next__
data = self.flo.read(self.chunk_size)
File "django/http/request.py", line 334, in read
six.reraise(UnreadablePostError, UnreadablePostError(*e.args), sys.exc_info()[2])
File "django/http/request.py", line 332, in read
return self._stream.read(*args, **kwargs)
File "sentry/middleware/proxy.py", line 32, in readinto
self._internal_buffer = uwsgi.chunked_read()
The problem is caused by the multipart upload feature (chunked requests). You can reproduce the error with
curl -X POST "http://localhost:9000/api/6/minidump/?sentry_key=14c00235201d43a7af58ac9fdc368334&guid=9e3f0f2f-cdd0-4229-8ef8-a06c22bd391f" -F guid="9e3f0f2f-cdd0-4229-8ef8-a06c22bd391f" -F upload_file_minidump=@"/C/Users/maxp/Source/Repos/sentrysdk/gen_windows/bin/Debug/.sentry-native/reports/9e3f0f2f-cdd0-4229-8ef8-a06c22bd391f.dmp" -H transfer-encoding:chunked
while the following curl upload without multipart succeeds:
curl -X POST "http://localhost:9000/api/6/minidump/?sentry_key=14c00235201d43a7af58ac9fdc368334&guid=9e3f0f2f-cdd0-4229-8ef8-a06c22bd391f" -F guid="9e3f0f2f-cdd0-4229-8ef8-a06c22bd391f" -F upload_file_minidump=@"/C/Users/maxp/Source/Repos/sentrysdk/gen_windows/bin/Debug/.sentry-native/reports/9e3f0f2f-cdd0-4229-8ef8-a06c22bd391f.dmp"
I didn't find a possibility to disable in in the crashpad client and I don't think that it's a good idea to handle it on the client side because it would make problems with large dumpfiles and slow internet connections.
@maxp1256 Thanks for the detailed report, I'll probably have a fix for the 500 thing soon.
Regarding chunked uploads, what do you have in SENTRY_WEB_OPTIONS
in your sentry.conf.py
?
@joshuarli
Regarding chunked uploads, what do you have in SENTRY_WEB_OPTIONS in your sentry.conf.py?
Most probably these: https://github.com/getsentry/onpremise/blob/bc6d3b47e257057587e29153947c1ba223160416/sentry/sentry.conf.example.py#L153-L163
Which should be fine.
I'm still working on reproduction, but it's possible that uwsgi might be misconfigured. We have some tricky middleware in place (ChunkedMiddleware
) and devserver's configured with
'protocol': 'http',
'http-chunked-input': True,
@joshuarli my SENTRY_WEB_OPTIONS are default. I'm currently testing it with your settings. Thanks for your help, I'm hoping that there'll be a fix soon :-)
SENTRY_WEB_OPTIONS = {
"http": "%s:%s" % (SENTRY_WEB_HOST, SENTRY_WEB_PORT),
"protocol": "uwsgi",
# This is needed to prevent https://git.io/fj7Lw
"uwsgi-socket": None,
"http-keepalive": True,
"memory-report": False,
# 'workers': 3, # the number of web workers
}
Hello, I am facing the same issue, so I am following the thread
I've completely cleared my SENTRY_WEB_OPTIONS
and it's working now. But I think it should be fixed anyway to get it working with the default settings.
SENTRY_WEB_HOST = "0.0.0.0"
SENTRY_WEB_PORT = 9000
SENTRY_WEB_OPTIONS = {
#"http": "%s:%s" % (SENTRY_WEB_HOST, SENTRY_WEB_PORT),
#"protocol": "uwsgi",
# This is needed to prevent https://git.io/fj7Lw
#"uwsgi-socket": None,
#"http-keepalive": True,
#"memory-report": False,
# 'workers': 3, # the number of web workers
}
Hmm, curious. Sadly I wasn't able to reproduce with devserver, but I have yet to stand up an onpremise installation to test against... I'm gonna guess uwsgi is configured differently between the two, somehow.
I can also confirm that disabling everything in SENTRY_WEB_OPTIONS "fixes" it ! Many thx to @maxp1256 for the workaround, I can now work on my debug information file workflow ;)
Quite sure this is due to lack of 'http-chunked-input': True,
. @Amphaal, @maxp1256 can you bring back those optons but add 'http-chunked-input': True,
to them and see if it fixes the issue?
Re-running ./install.sh
with default parameters in sentry.conf.py
+ "http-chunked-input": True
seems to work fine as well for me. @maxp1256, can you confirm it works with this setup ?
That would make sense, since uwsgi.chunked_read()
should only be available if "http-chunked-input": True
. I however removed that while testing with devserver, and it didn't error on me... maybe it's not actually being configured, will have to look more closely soon.
I'm trying to upload a crashdump with the example_crashpad project from the sentry-native sdk to a fresh on premise (docker) instance of sentry v10 (latest version).
I'm getting the following error in the web_1 docker instance and the dump isn't showing up in sentry.
I've reduced the sentry-configuration in example_crashpad.c to the following:
The relevant line for the first part seems to be this one: https://github.com/getsentry/sentry/blob/825661ff3ba6aa64d2fd3ef8abd0b2961fe45cba/src/sentry/web/helpers.py#L53
I've found another error which seems to be the primary problem.
I've created a sentry.io account and uploaded the minidump successfully so it seems to be a problem with the on-prem installation or a bug in the on-prem version.
What's wrong with my configuration?