Closed pjv closed 3 years ago
this is the nginx config from the host for proxying to zammad:
server {
listen 80;
server_name helpdesk.example.com;
return 301 https://helpdesk.example.com$request_uri;
}
server {
listen 443 ssl;
server_name helpdesk.example.com;
# SSL configs
ssl_certificate /var/www/certs/bt.com.cert.pem;
ssl_certificate_key /var/www/certs/bt.com.key.pem;
ssl_dhparam /etc/nginx/dhparams.pem;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:20m;
ssl_session_timeout 180m;
access_log /var/log/nginx/helpdesk.example.com.access.log;
error_log /var/log/nginx/helpdesk.example.com.error.log;
location / {
#proxy_pass http://localhost:5000/;
proxy_pass http://127.0.0.1:5000/;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forward-Proto https;
proxy_set_header X-Nginx-Proxy true;
proxy_redirect off;
}
}
And the docker-compose-override.yml looks like this:
version: '2'
services:
zammad-nginx:
ports:
- "5000:80"
volumes:
- ./custom/zammad.conf:/opt/zammad/contrib/nginx/zammad.conf
We had this topic quite a lot already: https://github.com/zammad/zammad-docker-compose/search?q=csrf&type=Issues
You can fix this by adjusting your configuration.
You can configure this now via env variable. See: https://github.com/zammad/zammad-docker-compose/blob/master/containers/zammad/docker-entrypoint.sh#L14
Thanks @MrGeneration, I have read all of those issues and tried to apply all of the configuration changes suggested in them. As a result of that I had a WORKING Zammad installation of v. 3.3.x. I upgraded to 3.4.0-4 without making any changes to my previously working configuration and now I am again having this CSRF token validation issue.
I would not have posted a new issue if any of the issues in the search you listed had helped me. I have already tried all of those fixes and am left with a not-log-in-able server.
Thank you @monotek. Sorry to be dense but I cannot understand from reading that source how to configure whatever I should configure to get past the CSRF token validation error.
Can you suggest which variable I should define in the .env
file to get back to a working config?
In case anyone else runs into the same issue that I did, my issue had nothing to do with #166, nothing to do with configuring environment variables, and nothing to do with any of the nginx configs.
Upgrading Zammad changed the HTTP type
setting for my instance which I had configured to http
for running behind an SSL terminating nginx proxy to https
. I don't know if there is an init default somewhere that gets overwritten or what, but I definitely had it intentionally set to http
and the upgrade changed it to https
.
Since I could no longer log in to fix this via the web UI, I had to do the following to solve this issue:
docker exec -it zammad-docker-compose_zammad-railsserver_1 bash
rails c
Setting.get('http_type')
=> "https"
Setting.set('http_type','http')
quit
exit
docker-compose down
docker-compose up -d
Zammad does detect the to be used http type during running the installation wizard. If at this point this is not yet correct, this has to be adjusted - e.g. like you noted above.
Well, I ran into this on an upgrade of a working Zammad install (from 3.3.x to 3.4.x) not on a new install and the upgrade for whatever reason over-wrote my already-configured HTTP type
setting. I did not go through the installation wizard after the upgrade, so either some part of the install wizard is happening invisibly behind the scenes or that detection must also be happening outside of the installation wizard.
There's no migration that touches your http_type
Setting.
This shouldn't happen automatically (after once set via installation way back in the past) at all.
I agree it shouldn't. You can either believe me or not that it did. Not trying to blame anyone, but in case it seems worth looking for how that could have happened, that's really what happened on my server.
[edit: if I was a rails dev, I'd take a look for it, but ruby code is greek to me - sorry.]
With the new Version the value for "proxy_set_header X-Forwarded-Proto" is configurable.
It sets "$scheme;" as its the default, maybe overwriting changes you did before. The default is taken from here: https://github.com/zammad/zammad/blob/develop/contrib/nginx/zammad.conf#L39
So its likely you want to get an "https" there.
Therefore start everything via:
docker-compose run -e NGINX_SERVER_SCHEME=https
@monotek thank you for getting back to me on that. I did actually try setting exactly that variable and value but it did not solve my issue which I detailed above.
I'll have a look if I'll automate setting http
or https
to be set as setting as well automatically.
That proberbly fixes the issue.
ETA unknown.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
I can confirm that this is an issue, even with new installs.
I was perfectly able to install Zammad behind our Traefik (still v1) and configure everything, after i logged out for the first time, i was not able to log back in again. The csrf token failed every time. I tried various things i found via search engine (eg https://github.com/zammad/zammad-docker-compose/issues/51, https://community.zammad.org/t/problem-accessing-zammad-behind-traefik/5044) but nothing helped.
docker exec -it zammad-docker-compose_zammad-railsserver_1 bash
rails c
Setting.get('http_type')
=> "https"
Setting.set('http_type','http')
quit
exit
docker-compose down
docker-compose up -d
Really did the trick for being able to login in again.
Requesting login via configured IdP (SAML) still fails (template doesnt event get rendered correctly) tho.
Logs read:
zammad-railsserver_1 | I, [2020-12-04T11:05:06.494610 #1-47350277365700] INFO -- : Started POST "/auth/saml" for 172.26.0.4 at 2020-12-04 11:05:06 +0000
zammad-railsserver_1 | I, [2020-12-04T11:05:06.505845 #1-47350277365700] INFO -- : (saml) Request phase initiated.
zammad-railsserver_1 | F, [2020-12-04T11:05:06.508011 #1-47350277365700] FATAL -- :
zammad-railsserver_1 | F, [2020-12-04T11:05:06.508110 #1-47350277365700] FATAL -- : ActionController::InvalidAuthenticityToken (ActionController::InvalidAuthenticityToken):
zammad-railsserver_1 | F, [2020-12-04T11:05:06.508192 #1-47350277365700] FATAL -- :
zammad-railsserver_1 | F, [2020-12-04T11:05:06.508280 #1-47350277365700] FATAL -- : omniauth-rails_csrf_protection (0.1.2) lib/omniauth/rails_csrf_protection/token_verifier.rb:34:in `call'
zammad-railsserver_1 | omniauth (1.9.1) lib/omniauth/strategy.rb:209:in `request_call'
zammad-railsserver_1 | omniauth (1.9.1) lib/omniauth/strategy.rb:188:in `call!'
zammad-railsserver_1 | omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
zammad-railsserver_1 | omniauth (1.9.1) lib/omniauth/strategy.rb:192:in `call!'
zammad-railsserver_1 | omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
zammad-railsserver_1 | omniauth (1.9.1) lib/omniauth/strategy.rb:192:in `call!'
zammad-railsserver_1 | omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
zammad-railsserver_1 | omniauth (1.9.1) lib/omniauth/strategy.rb:192:in `call!'
zammad-railsserver_1 | omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
zammad-railsserver_1 | omniauth (1.9.1) lib/omniauth/strategy.rb:192:in `call!'
zammad-railsserver_1 | omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
zammad-railsserver_1 | omniauth (1.9.1) lib/omniauth/strategy.rb:192:in `call!'
zammad-railsserver_1 | omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
zammad-railsserver_1 | omniauth (1.9.1) lib/omniauth/strategy.rb:192:in `call!'
zammad-railsserver_1 | omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
zammad-railsserver_1 | omniauth (1.9.1) lib/omniauth/strategy.rb:192:in `call!'
zammad-nginx | 172.26.0.4 - - [04/Dec/2020:11:05:06 +0000] "POST /auth/saml HTTP/1.1" 422 727 "https://helpdesk.example.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0"
zammad-railsserver_1 | omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
zammad-railsserver_1 | omniauth (1.9.1) lib/omniauth/strategy.rb:192:in `call!'
zammad-railsserver_1 | omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
zammad-railsserver_1 | omniauth (1.9.1) lib/omniauth/strategy.rb:192:in `call!'
zammad-railsserver_1 | omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
zammad-railsserver_1 | omniauth (1.9.1) lib/omniauth/builder.rb:45:in `call'
zammad-railsserver_1 | rack (2.2.3) lib/rack/tempfile_reaper.rb:15:in `call'
zammad-railsserver_1 | rack (2.2.3) lib/rack/etag.rb:27:in `call'
zammad-railsserver_1 | rack (2.2.3) lib/rack/conditional_get.rb:40:in `call'
zammad-railsserver_1 | rack (2.2.3) lib/rack/head.rb:12:in `call'
zammad-railsserver_1 | actionpack (5.2.4.4) lib/action_dispatch/http/content_security_policy.rb:18:in `call'
zammad-railsserver_1 | rack (2.2.3) lib/rack/session/abstract/id.rb:266:in `context'
zammad-railsserver_1 | rack (2.2.3) lib/rack/session/abstract/id.rb:260:in `call'
zammad-railsserver_1 | actionpack (5.2.4.4) lib/action_dispatch/middleware/cookies.rb:670:in `call'
zammad-railsserver_1 | actionpack (5.2.4.4) lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
zammad-railsserver_1 | activesupport (5.2.4.4) lib/active_support/callbacks.rb:98:in `run_callbacks'
zammad-railsserver_1 | actionpack (5.2.4.4) lib/action_dispatch/middleware/callbacks.rb:26:in `call'
zammad-railsserver_1 | actionpack (5.2.4.4) lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
zammad-railsserver_1 | actionpack (5.2.4.4) lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
zammad-railsserver_1 | railties (5.2.4.4) lib/rails/rack/logger.rb:38:in `call_app'
zammad-railsserver_1 | railties (5.2.4.4) lib/rails/rack/logger.rb:28:in `call'
zammad-railsserver_1 | actionpack (5.2.4.4) lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
zammad-railsserver_1 | actionpack (5.2.4.4) lib/action_dispatch/middleware/request_id.rb:27:in `call'
zammad-railsserver_1 | rack (2.2.3) lib/rack/method_override.rb:24:in `call'
zammad-railsserver_1 | rack (2.2.3) lib/rack/runtime.rb:22:in `call'
zammad-railsserver_1 | activesupport (5.2.4.4) lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
zammad-railsserver_1 | actionpack (5.2.4.4) lib/action_dispatch/middleware/executor.rb:14:in `call'
zammad-railsserver_1 | rack (2.2.3) lib/rack/sendfile.rb:110:in `call'
zammad-railsserver_1 | railties (5.2.4.4) lib/rails/engine.rb:524:in `call'
zammad-railsserver_1 | puma (3.12.6) lib/puma/configuration.rb:227:in `call'
zammad-railsserver_1 | puma (3.12.6) lib/puma/server.rb:706:in `handle_request'
zammad-railsserver_1 | puma (3.12.6) lib/puma/server.rb:476:in `process_client'
zammad-railsserver_1 | puma (3.12.6) lib/puma/server.rb:334:in `block in run'
zammad-railsserver_1 | puma (3.12.6) lib/puma/thread_pool.rb:135:in `block in spawn_thread'
zammad-railsserver_1 | logging (2.2.2) lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
zammad-nginx | 172.26.0.4 - - [04/Dec/2020:11:05:06 +0000] "GET /favicon.ico HTTP/1.1" 200 32988 "https://helpdesk.example.net/auth/saml" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0"
The temporary solution lifes in the history of this issue. In the mean time, please apply these and don't recycle / push this issue more than needed.
My primary focus was to add that the SAML login seems to be hit as well and that for the SAML login the provided temp fix is not working.
@eth3rgeist then your comment doesn't fit in this issue.
I was under the impression that my CSRF problem would fit this issue titled "New CSRF validation issue on upgrade to 3.4.0-4" since it would extend this to not only hit installments on upgrade but on fresh installations as well and not only the basic login but also extend ones.
But okay, sorry for the trouble caused.
I don't see a valid reason why, after patching, one login method works and the other doesn't. In my opinion that's a technical question that needs deeper dive. This is out of scope of github and especially this issue. Sorry.
I get it that you do not want to give me any technical support (your stance on TLS proxies other than the nginx coming with the zammad package is perfectly clear communicated in multiple posts).
But in this case the login fails nearly instant with the internal CSRF check inside the zammad stack, we are never getting to a point were a possible error in the SAML configuration comes into play (No request ever reached our IdP). That's why i attached the log, first 6 lines say it all, i just attached the rest of the stack to help you see something. (Also that error template is garbled, i'm pretty sure it should not look that way.)
In case anyone else runs into the same issue that I did, my issue had nothing to do with #166, nothing to do with configuring environment variables, and nothing to do with any of the nginx configs.
Upgrading Zammad changed the
HTTP type
setting for my instance which I had configured tohttp
for running behind an SSL terminating nginx proxy tohttps
. I don't know if there is an init default somewhere that gets overwritten or what, but I definitely had it intentionally set tohttp
and the upgrade changed it tohttps
.Since I could no longer log in to fix this via the web UI, I had to do the following to solve this issue:
docker exec -it zammad-docker-compose_zammad-railsserver_1 bash rails c Setting.get('http_type') => "https" Setting.set('http_type','http') quit exit docker-compose down docker-compose up -d
This one resolved my issue. It was not even an upgrade but restart of the system caused http_type to be changed to https
@MrGeneration @thorsteneckel Is somewhere in the code tried to set the value automatically by some detection mechanism or how can a change be explained? If not, whats the default value? Maybe we can also add an env var to the container to set it? Is a restart of zammad needed after setting it? Asking because the container is restarted in the comment above.
So I double checked the code base and found no place where the value of the http_type
Setting is changed automatically. The Setting default value is http
for 2 years now. Its value is set when running the initial getting started setup wizard or manually via the admin interface.
There is no restart of the whole application required after changing the HTTP type. A browser refresh is sufficient.
Very strange 🤔
Very strange
Yup. I’ve now done two additional upgrades (to 3.5x and 3.6x) using the exact same steps as I used when I did the original problematic upgrade to 3.4x that prompted me to open this ticket and have not had a repeat of the issue occur.
Thanks for the heads up @pjv ! Would be interesting on what version (from -> to) change others faced this issue!
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Apologies for necrobumping this issue, but I am unaware of any other feasible way to get in contact with users that previously commented here. I do understand that what follows is likely a technical question and therefore not going to be resolved by the Zammad developers – my only goal is it to establish contact to a community member who might be able to help me. Having said this:
@eth3rgeist Have you found a solution to the SAML problem? I have a pretty similar setup (Zammad 4.0 behind a Traefik v2 reverse proxy). Normal login works (only when the HTTP type is set to http
), but SAML login attempts always directly result in a 422. In the logs, I also get an ActionController::InvalidAuthenticityToken
error.
I would really love to hear from you :)
If you want to get in touch with community users, the best way would be the Zammad community https://community.zammad.org . No guaranteed help, but still, probably the better place.
I'm sorry, but I'm limiting the conversation of this issue to contributors only.
Infos:
Expected behavior:
Able to log in
Actual behavior:
Not able to log in with CSRF token validation error
Steps to reproduce the behavior:
Try to log in after upgrading from Zammad 3.3X to 3.4.0-4.
I had a working docker install of Zammad v 3.3.x running behind an nginx proxy on the host which handles SSL (this host itself is behind cloudflare proxy which provides the SSL certs). I solved prior CSRF token validation issues by using the
zammad.conf
provided here.I upgraded this morning by pulling the latest version of this repo, docker-compose pulling the latest images, and then docker-compose up. Now I have the CSRF token validation error again on trying to log in. I have tried all the prior fixes that I could find from issues here at github and from the zammad forum but nothing works.
I have a bunch of customer service agents that are going to arrive at their desks in an hour or two who are not going to be able to log in to Zammad. Any help appreciated.
I'm guessing that this issue is somehow related to #166 (by @monotek) but I don't know enough about what those commits are doing to figure out what to try.