Open stumbaumr opened 7 months ago
I suspect that there is some corrupt data in postgres. Can you make sure the superuser flag is set in the auth_user
table?
Hi @hubertdeng123 ,
just checked, yes, I have 4 users which have is_superuser set to true.
root@sentry01(dc1.prd):~# docker exec -it sentry-self-hosted-postgres-1 psql -U postgres
psql (14.5 (Debian 14.5-2.pgdg110+2))
Type "help" for help.
postgres=# select count(*) from auth_user where is_superuser = 't';
count
-------
4
(1 row)
postgres=#
When I log on with SAML disabled as a superuser, I get this:
Clicking "If you need to make changes to your user account, please click here." takes me to the organization.
There creating a new "local" user sends out the eMail, when clicking on the link it gives
Is sentry_organizationmapping supposed to be empty?
postgres=# select * from sentry_organizationmapping;
id | organization_id | slug | name | date_created | customer_id | verified | idempotency_key | region_name | status | require_2fa | early_adopter | allow_joinleave | enhanced_privacy | disable_shared_issues | disable_new_visibility_features | require_email_verification | codecov
_access
----+-----------------+------+------+--------------+-------------+----------+-----------------+-------------+--------+-------------+---------------+-----------------+------------------+-----------------------+---------------------------------+----------------------------+--------
--------
(0 rows)
The organization invite link is no longer valid.
Did you make sure to logout or clear cookies before clicking on the link?
The sentry_organizationmapping
is supposed to map org slugs to a specific org and region, which should not be applicable here.
After creating a new entry in the empty sentry_organizationmapping PostGres table we were able to login again and got routed correctly to the organization again and were successfully able to activate SAML and the integrations.
Could you search your web
container logs for the following regex:
service.py", line \d+, in serialize_many
I'm curious if there is an async function that is causing trouble here.
After creating a new entry in the empty sentry_organizationmapping PostGres table we were able to login again and got routed correctly to the organization again and were successfully able to activate SAML and the integrations.
Looks like I was wrong, great find here!
service.py", line \d+, in serialize_many
Tried that, but nothing in the logs...
root@sentry01(dc1.prd):/var/lib/docker/containers# grep -R "service.py\", line \d+, in serialize_many" *
root@sentry01(dc1.prd):/var/lib/docker/containers#
After creating a new entry in the empty sentry_organizationmapping PostGres table we were able to login again and got routed correctly to the organization again and were successfully able to activate SAML and the integrations.
Looks like I was wrong, great find here!
@hubertdeng123, I believe more people will come across that missing entry - is there any way to find out how it got deleted in the first place?
I wonder if we can just put something in the next release to rebuild the organization_mapping
? Seems like we could just wipe the table and rebuild it based on existing entries in the orgnization
table directly.
After creating a new entry in the empty sentry_organizationmapping PostGres table we were able to login again and got routed correctly to the organization again and were successfully able to activate SAML and the integrations.
@stumbaumr Can you give a bit more detail on how you add those data ? We are affected by the same issue when trying to integration Authentik SAML in our current Sentry setup
Start a psql session in your postgres container and insert an entry into that table using a INSERT statement...
root@sentry01(dc1.prd):~# docker exec -it sentry-self-hosted-postgres-1 psql -U postgres
psql (14.5 (Debian 14.5-2.pgdg110+2))
Type "help" for help.
postgres=# select * from sentry_organizationmapping;
id | organization_id | slug | name | date_created | customer_id | verified | idempotency_key | region_name | status | require_2fa | early_adopter | allow_joinleave | enhanced_privacy | disable_shared_issues | disable_new_visibility_features | require_email_veri
fication | codecov_access
----+-----------------+--------+-----------+------------------------+-------------+----------+-----------------+--------------+--------+-------------+---------------+-----------------+------------------+-----------------------+---------------------------------+-------------------
---------+----------------
3 | 1 | XXX | XXX | 2024-01-01 00:00:00+00 | XXX | t | XXX | --monolith-- | 0 | f | f | t | f | f | f | f
| f
(1 row)
postgres=#
@azaslavsky The more users upgrade the more support is required - maybe really best to just provide a delete and recreate procedure...
This happend to our self-hosted instance aswell. There was a lot of repeating stacktrace.
web-1 | During handling of the above exception, another exception occurred:
web-1 |
web-1 | Traceback (most recent call last):
web-1 | File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
web-1 | response = get_response(request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/../sentry_sdk/integrations/django/middleware.py", line 175, in __call__
web-1 | return f(*args, **kwargs)
web-1 | ^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/deprecation.py", line 134, in __call__
web-1 | response = response or self.get_response(request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 57, in inner
web-1 | response = response_for_exception(request, exc)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 140, in response_for_exception
web-1 | response = handle_uncaught_exception(
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 185, in handle_uncaught_exception
web-1 | return callback(request)
web-1 | ^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 104, in view
web-1 | return self.dispatch(request, *args, **kwargs)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/web/frontend/error_500.py", line 37, in dispatch
web-1 | return render_to_response("sentry/500.html", status=500, context=context, request=request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/web/helpers.py", line 43, in render_to_response
web-1 | response = HttpResponse(render_to_string(template, context, request))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/web/helpers.py", line 30, in render_to_string
web-1 | rendered = loader.render_to_string(template, context=context, request=request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/loader.py", line 62, in render_to_string
web-1 | return template.render(context, request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/backends/django.py", line 61, in render
web-1 | return self.template.render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 171, in render
web-1 | return self._render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 163, in _render
web-1 | return self.nodelist.render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 1000, in render
web-1 | return SafeString("".join([node.render_annotated(context) for node in self]))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 1000, in <listcomp>
web-1 | return SafeString("".join([node.render_annotated(context) for node in self]))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 961, in render_annotated
web-1 | return self.render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/loader_tags.py", line 159, in render
web-1 | return compiled_parent._render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 163, in _render
web-1 | return self.nodelist.render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 1000, in render
web-1 | return SafeString("".join([node.render_annotated(context) for node in self]))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 1000, in <listcomp>
web-1 | return SafeString("".join([node.render_annotated(context) for node in self]))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 961, in render_annotated
web-1 | return self.render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/loader_tags.py", line 159, in render
web-1 | return compiled_parent._render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 163, in _render
web-1 | return self.nodelist.render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 1000, in render
web-1 | return SafeString("".join([node.render_annotated(context) for node in self]))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 1000, in <listcomp>
web-1 | return SafeString("".join([node.render_annotated(context) for node in self]))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 961, in render_annotated
web-1 | return self.render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/loader_tags.py", line 65, in render
web-1 | result = block.nodelist.render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 1000, in render
web-1 | return SafeString("".join([node.render_annotated(context) for node in self]))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 1000, in <listcomp>
web-1 | return SafeString("".join([node.render_annotated(context) for node in self]))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 961, in render_annotated
web-1 | return self.render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/templatetags/sentry_assets.py", line 90, in render
web-1 | content = self.nodelist.render(context).strip()
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 1000, in render
web-1 | return SafeString("".join([node.render_annotated(context) for node in self]))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 1000, in <listcomp>
web-1 | return SafeString("".join([node.render_annotated(context) for node in self]))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 961, in render_annotated
web-1 | return self.render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/library.py", line 237, in render
web-1 | output = self.func(*resolved_args, **resolved_kwargs)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/templatetags/sentry_react.py", line 11, in get_react_config
web-1 | context = get_client_config(context.get("request", None), context.get("org_context"))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/web/client_config.py", line 412, in get_client_config
web-1 | return config.get_context()
web-1 | ^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/web/client_config.py", line 397, in get_context
web-1 | "user": self.user_details,
web-1 | ^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__
web-1 | val = self.func(instance)
web-1 | ^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/web/client_config.py", line 316, in user_details
web-1 | user_details["isSuperuser"] = self.user.is_superuser
web-1 | ~~~~~~~~~~~~^^^^^^^^^^^^^^^
web-1 | TypeError: 'NoneType' object does not support item assignment
Just added this stacktrace because it doesn't look all the same from the issue creator. (i could be wrong tho)
We're still thinking about what the best approach here is. Automatically adjusting users organizationmapping
tables requires making a lot of assumptions that might hurt more users than it helps.
Maybe just a proper sample INSERT-SELECT statement in this thread so people know what to tweak. This is not enduser level stuff we are talking about here...
I wonder if your issue popped up after hybrid cloud changes were implemented. At what point did you realize SAML2 was broken? Was it working for your 23.6.2 upgrade?
This flow may very well be broken for everyone so it would be nice to have some sort of long term fix, even though an INSERT-SELECT may do the job for now.
+1 I'm trying to upgrade sentry from 23.3.1 to 24.2.0 and have the same issue. In 23.3.1 the table sentry_organizationmapping is empty.
Same issue here. A way to populate the sentry_organizationmapping
table would be appreciated.
@stephanepechard look at the message from @stumbaumr above. https://github.com/getsentry/self-hosted/issues/2750#issuecomment-1954094951
Impact also Okta !
Everybody with self hosting and external authentication will be impacted after the upgrade. As long as Devs don't know at which version step the problem occurs they can't provide a fix...
After redoing all the step from a snapshot, it's working without issues. Probably I missed a step or some migration failed. but now I don't have any issues with Okta.
22.10 -> 23.6.2(hard stop) -> 24.2
Self-Hosted Version
24.1.1
CPU Architecture
x86_64
Docker Version
24.0.1
Docker Compose Version
v2.24.3
Steps to Reproduce
After the upgrade (see https://github.com/getsentry/self-hosted/issues/2742) I tried the SSO and failed. I removed the configuration
and then recreated it using the Web interface with the Auth link and SAML2 as described here https://docs.sentry.io/product/accounts/sso/azure-sso/ . I tested the configuration in Chrome from the Microsoft Entra Admin side - successfully.
But the sign in fails with Nginx reporting a 502 bad gateway error.
Below the log from the web container.
No idea how to continue from here.
Expected Result
Single Sign On using SAML2 provider should work like it used to...
Actual Result
Event ID
No response