search

DefectDojo / django-DefectDojo

DevSecOps, ASPM, Vulnerability Management. All on one platform.
https://defectdojo.com
BSD 3-Clause "New" or "Revised" License
3.69k stars 1.55k forks source link

Frequent crashes, serialization.py stack trace #2397

Closed nadavr-12 closed 4 years ago

nadavr-12 commented 4 years ago

Bug description Following a Kubernetes deployment of both 1.5.4 and the 1.6 release candidate, I'm seeing a lot of stack traces (adding console logs). When trying to import findings the pods are crashing. Anyone has seen this behaviour before?

Deployment method (select with an X)

Console logs ` Starting uWSGI 2.0.18 (64bit) on [Tue May 19 10:40:23 2020] compiled with version: 8.3.0 on 19 May 2020 10:20:34 os: Linux-3.10.0-957.12.2.el7.x86_64 #1 SMP Tue May 14 21:24:32 UTC 2019 nodename: defectdojo-django-67cd7ccc57-cvbgl machine: x86_64 clock source: unix pcre jit disabled detected number of CPU cores: 88 current working directory: /app detected binary path: /usr/local/bin/uwsgi WARNING: you are running uWSGI without its master process manager your processes number limit is 1048576 your memory page size is 4096 bytes detected max file descriptor number: 1048576 lock engine: pthread robust mutexes thunder lock: disabled (you can enable it with --thunder-lock) uwsgi socket 0 bound to UNIX address /run/uwsgi.sock fd 3 Python version: 3.5.9 (default, Jan 4 2020, 04:03:14) [GCC 8.3.0] Python main interpreter initialized at 0x5555a01e48b0 python threads support enabled your server socket listen backlog is limited to 100 connections your mercy for graceful operations on workers is 60 seconds mapped 166752 bytes (162 KB) for 4 cores Operational MODE: preforking+threaded enabling audit logging patching TagDescriptor WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0x5555a01e48b0 pid: 1 (default app) uWSGI is running in multiple interpreter mode spawned uWSGI worker 1 (pid: 1, cores: 2) spawned uWSGI worker 2 (pid: 74, cores: 2) [pid: 1|app: 0|req: 1/1] 172.20.63.1 () {34 vars in 388 bytes} [Tue May 19 10:43:01 2020] GET / => generated 0 bytes in 6 msecs (HTTP/1.1 302) 7 headers in 215 bytes (1 switches on core 0) Popen(['git', 'version'], cwd=/app, universal_newlines=False, shell=None, istream=None) Popen(['git', 'version'], cwd=/app, universal_newlines=False, shell=None, istream=None) Tue May 19 10:43:02 2020 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) !!!

Tue May 19 10:43:02 2020 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 306] during GET /login?next=/ (172.20.63.1) OSError: write error [pid: 74|app: 0|req: 1/2] 172.20.63.1 () {36 vars in 453 bytes} [Tue May 19 10:43:01 2020] GET /login?next=/ => generated 0 bytes in 1217 msecs (HTTP/1.1 200) 9 headers in 0 bytes (0 switches on core 0) [pid: 74|app: 0|req: 2/3] 172.20.63.1 () {34 vars in 388 bytes} [Tue May 19 10:43:09 2020] GET / => generated 0 bytes in 1 msecs (HTTP/1.1 302) 7 headers in 215 bytes (1 switches on core 1) Popen(['git', 'version'], cwd=/app, universal_newlines=False, shell=None, istream=None) Popen(['git', 'version'], cwd=/app, universal_newlines=False, shell=None, istream=None) Tue May 19 10:43:10 2020 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 306] during GET /login?next=/ (172.20.63.1) OSError: write error [pid: 1|app: 0|req: 2/4] 172.20.63.1 () {36 vars in 453 bytes} [Tue May 19 10:43:09 2020] GET /login?next=/ => generated 0 bytes in 1008 msecs (HTTP/1.1 200) 9 headers in 0 bytes (0 switches on core 1) [pid: 74|app: 0|req: 3/5] 172.20.63.1 () {34 vars in 388 bytes} [Tue May 19 10:43:11 2020] GET / => generated 0 bytes in 1 msecs (HTTP/1.1 302) 7 headers in 215 bytes (1 switches on core 0) [pid: 1|app: 0|req: 3/6] 172.20.63.1 () {36 vars in 453 bytes} [Tue May 19 10:43:11 2020] GET /login?next=/ => generated 11408 bytes in 25 msecs (HTTP/1.1 200) 9 headers in 472 bytes (1 switches on core 0) /usr/local/lib/python3.5/site-packages/django/db/models/fields/init.py:1369: RuntimeWarning: DateTimeField Test.target_start received a naive datetime (2020-05-19 00:00:00) while time zone support is active. RuntimeWarning) /usr/local/lib/python3.5/site-packages/django/db/models/fields/init.py:1369: RuntimeWarning: DateTimeField Test.target_end received a naive datetime (2020-05-19 00:00:00) while time zone support is active. RuntimeWarning) [pid: 74|app: 0|req: 5/7] 172.20.63.1 () {34 vars in 388 bytes} [Tue May 19 10:43:19 2020] GET / => generated 0 bytes in 0 msecs (HTTP/1.1 302) 7 headers in 215 bytes (1 switches on core 0) applying rules [pid: 74|app: 0|req: 6/8] 172.20.63.1 () {36 vars in 453 bytes} [Tue May 19 10:43:19 2020] GET /login?next=/ => generated 11408 bytes in 36 msecs (HTTP/1.1 200) 9 headers in 472 bytes (1 switches on core 0) applying rules Internal Server Error: /api/v2/import-scan/ Traceback (most recent call last): File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 50, in _reraise_errors yield File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 221, in dumps payload = encoder(data) File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 350, in pickle_dumps return dumper(obj, protocol=pickle_protocol) File "/usr/local/lib/python3.5/tempfile.py", line 483, in func_wrapper return func(*args, **kwargs) TypeError: cannot serialize '_io.BufferedRandom' object

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "./dojo/models.py", line 1830, in save async_dedupe.delay(self, args, kwargs) File "/usr/local/lib/python3.5/site-packages/celery/app/task.py", line 425, in delay return self.apply_async(args, kwargs) File "/usr/local/lib/python3.5/site-packages/celery/app/task.py", line 568, in apply_async options File "/usr/local/lib/python3.5/site-packages/celery/app/base.py", line 775, in send_task amqp.send_task_message(P, name, message, options) File "/usr/local/lib/python3.5/site-packages/celery/app/amqp.py", line 559, in send_task_message properties File "/usr/local/lib/python3.5/site-packages/kombu/messaging.py", line 169, in publish compression, headers) File "/usr/local/lib/python3.5/site-packages/kombu/messaging.py", line 252, in _prepare body) = dumps(body, serializer=serializer) File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 221, in dumps payload = encoder(data) File "/usr/local/lib/python3.5/contextlib.py", line 77, in exit self.gen.throw(type, value, traceback) File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 54, in _reraise_errors reraise(wrapper, wrapper(exc), sys.exc_info()[2]) File "/usr/local/lib/python3.5/site-packages/vine/five.py", line 194, in reraise raise value.with_traceback(tb) File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 50, in _reraise_errors yield File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 221, in dumps payload = encoder(data) File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 350, in pickle_dumps return dumper(obj, protocol=pickle_protocol) File "/usr/local/lib/python3.5/tempfile.py", line 483, in func_wrapper return func(args, **kwargs) kombu.exceptions.EncodeError: cannot serialize '_io.BufferedRandom' object

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 50, in _reraise_errors yield File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 221, in dumps payload = encoder(data) File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 350, in pickle_dumps return dumper(obj, protocol=pickle_protocol) File "/usr/local/lib/python3.5/tempfile.py", line 483, in func_wrapper return func(*args, **kwargs) TypeError: cannot serialize '_io.BufferedRandom' object

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 34, in inner response = get_response(request) File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 115, in _get_response response = self.process_exception_by_middleware(e, request) File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 113, in _get_response response = wrapped_callback(request, *callback_args, callback_kwargs) File "/usr/local/lib/python3.5/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view return view_func(*args, *kwargs) File "/usr/local/lib/python3.5/site-packages/rest_framework/viewsets.py", line 114, in view return self.dispatch(request, args, kwargs) File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 505, in dispatch response = self.handle_exception(exc) File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 465, in handle_exception self.raise_uncaught_exception(exc) File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 476, in raise_uncaught_exception raise exc File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 502, in dispatch response = handler(request, *args, *kwargs) File "/usr/local/lib/python3.5/site-packages/rest_framework/mixins.py", line 19, in create self.perform_create(serializer) File "./dojo/api_v2/views.py", line 666, in perform_create serializer.save(push_to_jira=push_to_jira) File "./dojo/api_v2/serializers.py", line 691, in save item.save(push_to_jira=push_to_jira) File "./dojo/models.py", line 1832, in save async_dedupe.delay(self, args, kwargs) File "/usr/local/lib/python3.5/site-packages/celery/app/task.py", line 425, in delay return self.apply_async(args, kwargs) File "/usr/local/lib/python3.5/site-packages/celery/app/task.py", line 568, in apply_async options File "/usr/local/lib/python3.5/site-packages/celery/app/base.py", line 775, in send_task amqp.send_task_message(P, name, message, **options) File "/usr/local/lib/python3.5/site-packages/celery/app/amqp.py", line 559, in send_task_message

*properties File "/usr/local/lib/python3.5/site-packages/kombu/messaging.py", line 169, in publish compression, headers) File "/usr/local/lib/python3.5/site-packages/kombu/messaging.py", line 252, in _prepare body) = dumps(body, serializer=serializer) File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 221, in dumps payload = encoder(data) File "/usr/local/lib/python3.5/contextlib.py", line 77, in exit self.gen.throw(type, value, traceback) File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 54, in _reraise_errors reraise(wrapper, wrapper(exc), sys.exc_info()[2]) File "/usr/local/lib/python3.5/site-packages/vine/five.py", line 194, in reraise raise value.with_traceback(tb) File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 50, in _reraise_errors yield File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 221, in dumps payload = encoder(data) File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 350, in pickle_dumps return dumper(obj, protocol=pickle_protocol) File "/usr/local/lib/python3.5/tempfile.py", line 483, in func_wrapper return func(args, **kwargs) kombu.exceptions.EncodeError: cannot serialize '_io.BufferedRandom' object

[pid: 1|app: 0|req: 5/16] 172.20.63.1 () {36 vars in 453 bytes} [Tue May 19 10:43:31 2020] GET /login?next=/ => generated 11408 bytes in 26 msecs (HTTP/1.1 200) 9 headers in 472 bytes (1 switches on core 0) applying rules applying rules Internal Server Error: /api/v2/import-scan/ Traceback (most recent call last): File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 50, in _reraise_errors yield File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 221, in dumps payload = encoder(data) File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 350, in pickle_dumps return dumper(obj, protocol=pickle_protocol) File "/usr/local/lib/python3.5/tempfile.py", line 483, in func_wrapper return func(*args, **kwargs) TypeError: cannot serialize '_io.BufferedRandom' object

`

valentijnscholten commented 4 years ago

doesn't seem to ring a bell with anyone. Could you retry with a clean 1.6.2, 1.6.5 or dev branch?

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.