search

rafalp / Misago

Misago is fully featured modern forum application that is fast, scalable and responsive.
http://misago-project.org
GNU General Public License v2.0
2.52k stars 524 forks source link

Django ORM sometimes fails to deserialize `user.profile_fields` from `str` to `dict` #1352

Open bgreatfit opened 3 years ago

bgreatfit commented 3 years ago

The errors I get when I try to log in as an admin or user Internal Server Error: /admincp/ misago_1 | Traceback (most recent call last): misago_1 | File "/usr/local/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner misago_1 | response = get_response(request) misago_1 | File "/usr/local/lib/python3.7/site-packages/django/core/handlers/base.py", line 106, in _get_response misago_1 | response = middleware_method(request, callback, callback_args, callback_kwargs) misago_1 | File "/srv/misago/misago/admin/middleware.py", line 17, in process_view misago_1 | return self.check_admin_authorization(request) misago_1 | File "/srv/misago/misago/admin/middleware.py", line 23, in check_admin_authorization misago_1 | return login(request) misago_1 | File "/usr/local/lib/python3.7/site-packages/django/views/decorators/debug.py", line 76, in sensitive_post_parameters_wrapper misago_1 | return view(request, *args, kwargs) misago_1 | File "/usr/local/lib/python3.7/site-packages/django/utils/decorators.py", line 142, in _wrapped_view misago_1 | response = view_func(request, *args, *kwargs) misago_1 | File "/usr/local/lib/python3.7/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func misago_1 | response = view_func(request, args, kwargs) misago_1 | File "/srv/misago/misago/admin/views/auth.py", line 31, in login misago_1 | return render(request, "misago/admin/login.html", {"form": form, "target": target}) misago_1 | File "/usr/local/lib/python3.7/site-packages/django/shortcuts.py", line 36, in render misago_1 | content = loader.render_to_string(template_name, context, request, using=using) misago_1 | File "/usr/local/lib/python3.7/site-packages/django/template/loader.py", line 62, in render_to_string misago_1 | return template.render(context, request) misago_1 | File "/usr/local/lib/python3.7/site-packages/django/template/backends/django.py", line 61, in render misago_1 | return self.template.render(context) misago_1 | File "/usr/local/lib/python3.7/site-packages/django/template/base.py", line 169, in render misago_1 | with context.bind_template(self): misago_1 | File "/usr/local/lib/python3.7/contextlib.py", line 112, in enter misago_1 | return next(self.gen) misago_1 | File "/usr/local/lib/python3.7/site-packages/debug_toolbar/panels/templates/panel.py", line 50, in _request_context_bind_template misago_1 | context = processor(self.request) misago_1 | File "/srv/misago/misago/users/context_processors.py", line 45, in preload_user_json misago_1 | serialized_user = serializer(request.user, context={"acl": request.user_acl}).data misago_1 | File "/usr/local/lib/python3.7/site-packages/rest_framework/serializers.py", line 563, in data misago_1 | ret = super(Serializer, self).data misago_1 | File "/usr/local/lib/python3.7/site-packages/rest_framework/serializers.py", line 262, in data misago_1 | self._data = self.to_representation(self.instance) misago_1 | File "/usr/local/lib/python3.7/site-packages/rest_framework/serializers.py", line 530, in to_representation misago_1 | ret[field.field_name] = field.to_representation(attribute) misago_1 | File "/usr/local/lib/python3.7/site-packages/rest_framework/fields.py", line 1889, in to_representation misago_1 | return method(value) misago_1 | File "/srv/misago/misago/users/serializers/user.py", line 92, in get_real_name misago_1 | return obj.get_real_name() misago_1 | File "/srv/misago/misago/users/models/user.py", line 331, in get_real_name misago_1 | return self.profile_fields.get("real_name") misago_1 | AttributeError: 'str' object has no attribute 'get'

bgreatfit commented 3 years ago

@rafalp @chimeworld @l0ud pls can you help with this error

soloviola commented 3 years ago

@bgreatfit

What I found is in image

Change profile_fields.get("real_name") to profile_fields

rafalp commented 3 years ago

This seems very wrong. profile_fields is supposed to be hstore with custom profile fields and represented as dict in python.

How are you experiencing this bug?

bgreatfit commented 3 years ago

This seems very wrong. profile_fields is supposed to be hstore with custom profile fields and represented as dict in python.

How are you experiencing this bug?

Just launched the app on my local server after following the Readme instructions

jeroenbakker-atmind commented 1 year ago

I reproduced the same/a similar issue.

Environment:

Request Method: GET
Request URL: http://127.0.0.1:8000/t/only-one-spacecraft-has-visited-distant-uranus/2/

Django Version: 3.2.15
Python Version: 3.11.1
Installed Applications:
['misago',
 'misago.users',
 'django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.postgres',
 'django.contrib.humanize',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'ariadne_django',
 'celery',
 'debug_toolbar',
 'mptt',
 'rest_framework',
 'social_django',
 'misago.admin',
 'misago.acl',
 'misago.analytics',
 'misago.cache',
 'misago.core',
 'misago.conf',
 'misago.icons',
 'misago.themes',
 'misago.markup',
 'misago.legal',
 'misago.categories',
 'misago.threads',
 'misago.readtracker',
 'misago.search',
 'misago.oauth2',
 'misago.socialauth',
 'misago.graphql',
 'misago.faker',
 'misago.menus',
 'misago.plugins']
Installed Middleware:
['debug_toolbar.middleware.DebugToolbarMiddleware',
 'misago.users.middleware.RealIPMiddleware',
 'misago.core.middleware.FrontendContextMiddleware',
 'django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'misago.cache.middleware.cache_versions_middleware',
 'misago.conf.middleware.dynamic_settings_middleware',
 'misago.socialauth.middleware.socialauth_providers_middleware',
 'misago.users.middleware.UserMiddleware',
 'misago.acl.middleware.user_acl_middleware',
 'misago.core.middleware.ExceptionHandlerMiddleware',
 'misago.users.middleware.OnlineTrackerMiddleware',
 'misago.admin.middleware.AdminAuthMiddleware',
 'misago.threads.middleware.UnreadThreadsCountMiddleware']

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
  File "/srv/misago/misago/threads/views/thread.py", line 18, in get
    frontend_context = self.get_frontend_context(request, thread, posts)
  File "/srv/misago/misago/threads/views/thread.py", line 47, in get_frontend_context
    "POSTS": posts.get_frontend_context(),
  File "/srv/misago/misago/threads/viewmodels/posts.py", line 107, in get_frontend_context
    ).data
  File "/usr/local/lib/python3.11/site-packages/rest_framework/serializers.py", line 768, in data
    ret = super().data
  File "/usr/local/lib/python3.11/site-packages/rest_framework/serializers.py", line 253, in data
    self._data = self.to_representation(self.instance)
  File "/usr/local/lib/python3.11/site-packages/rest_framework/serializers.py", line 686, in to_representation
    return [
  File "/usr/local/lib/python3.11/site-packages/rest_framework/serializers.py", line 687, in <listcomp>
    self.child.to_representation(item) for item in iterable
  File "/usr/local/lib/python3.11/site-packages/rest_framework/serializers.py", line 522, in to_representation
    ret[field.field_name] = field.to_representation(attribute)
  File "/usr/local/lib/python3.11/site-packages/rest_framework/serializers.py", line 522, in to_representation
    ret[field.field_name] = field.to_representation(attribute)
  File "/usr/local/lib/python3.11/site-packages/rest_framework/fields.py", line 1886, in to_representation
    return method(value)
  File "/srv/misago/misago/users/serializers/user.py", line 92, in get_real_name
    return obj.get_real_name()
  File "/srv/misago/misago/users/models/user.py", line 329, in get_real_name
    return self.profile_fields.get("real_name")

Exception Type: AttributeError at /t/only-one-spacecraft-has-visited-distant-uranus/2/
Exception Value: 'str' object has no attribute 'get'

Steps to reproduce are:

Will check if this also happens with by restoring/migrating a production database and by running these steps on a clean machine.

EDIT: after a second try from a clean environment this issue doesn't happen, might be related to a change I did before via admincp or that the second time I didn't broke the fakedata process after a shorter time.

rafalp commented 1 year ago

I guess the issue is profile_fields on user model being set to string when it should be a dict?

rafalp commented 1 year ago

@jeroenbakker-atmind You mention doing a change through admin panel. Can you mention what change was that?

rafalp commented 1 year ago

Good news: this doesn't seem to be a bug with Misago!

Bad news: this error occurring means that database driver didn't retrieve valid hstore extension ID from pg_type table when initializing connection. Because it has wrong ID, it doesn't understand that profile_fields value is from hstore field. So it returns it as string, which results in an error.

rafalp commented 1 year ago

Only solution that comes to mind is moving profile_fields from HStore to JSONB, which is standard and has constant OID in PostgreSQL (and it's clients). But then what guarantee there is that bug will not occur when data migrator is running, and profile fields are migrated to JSON as strings?

rafalp commented 1 year ago

@jeroenbakker-atmind I've found a new lead for what can cause this issue. Can you please tell me what memory limits are set in your docker configuration?

rafalp commented 1 year ago

I've couldn't reproduce this, but there are two possibilities now I would like to check:

rafalp commented 1 year ago

I'm unable to reproduce this, but I've added logic that will catch when this bug happens and will print error message with debug data that will help me investigate this further or let me bring this issue to Django dev team, also have a bug for this, but requested more information about it.

This error will ONLY DISPLAY IN DEBUG so production deployments and deployments with misago-docker are safe and will not display the error to the users:

Zrzut ekranu 2023-01-26 o 21 59 49
raydeal commented 1 year ago

I can't reproduce that in Misago 0.30. Moving from HStore to JSONB is good idea. HStoreField is only used for profile_fields so it is only one place to change. I think that Postgres will migrate and check type correctly if migration will be done in db, not through Python libs - newer version of Postgres use hstore_to_jsonb when value is casted to jsonb. When I dived deeply I found that profile_fields is not null and django put there empty string ('').

misago=# SELECT COUNT(*) FROM misago_users_user WHERE profile_fields = '';
 count 
-------
  3141
(1 row)

After migration to JSONB, empty string was replaced with dict.

misago=# SELECT COUNT(*) FROM misago_users_user WHERE profile_fields = '{}'::jsonb;
 count 
-------
  3141
(1 row)
misago=# SELECT profile_fields FROM misago_users_user LIMIT 1;
 profile_fields 
----------------
 {}
(1 row)

It will be easier to find a reason with JSONField in case of any error.

I have created PR - draft, because some tests are failing

ro1and commented 9 months ago

RuntimeError at /

'profile_fields' has wrong type! Please post this WHOLE message on https://github.com/rafalp/Misago/issues/1352 OID: '((), ())' (valid: '((17280,), (17285,))') Receivers: '<function register_type_handlers at 0x7fe98e1adc60>' (has dead: FALSE) Repr: ''
Request Method: | GET -- | -- http://0.0.0.0:8000/ 3.2.23 RuntimeError 'profile_fields' has wrong type! Please post this WHOLE message on https://github.com/rafalp/Misago/issues/1352 OID: '((), ())' (valid: '((17280,), (17285,))') Receivers: '' (has dead: FALSE) Repr: '' /srv/misago/misago/users/models/user.py, line 412, in get_real_name /usr/local/bin/python 3.12.0 ['/srv/misago', '/srv/misago', '/usr/local/lib/python312.zip', '/usr/local/lib/python3.12', '/usr/local/lib/python3.12/lib-dynload', '/usr/local/lib/python3.12/site-packages'] Sat, 11 Nov 2023 19:00:39 +0000
rafalp commented 9 months ago

Thank you for reporting this! With data I've gathered so far, it seems there's a bug in Django where on rare occasions it fails to register HStore extension for database connection.

This bug resolves on itself upon docker restart and seems to only affect dev setups on localhost. Still, its annoying and I plan to move Misago away from HStore to JSON fields in one of future versions.

rafalp commented 2 months ago

Related to: #1755