zammad / zammad-docker-compose

Zammad Docker images for docker-compose
https://hub.docker.com/r/zammad/zammad-docker-compose/
GNU Affero General Public License v3.0
274 stars 223 forks source link

New CSRF validation issue on upgrade to 3.4.0-4 #168

Closed pjv closed 3 years ago

pjv commented 4 years ago

Infos:

Expected behavior:

Able to log in

Actual behavior:

Not able to log in with CSRF token validation error

Steps to reproduce the behavior:

Try to log in after upgrading from Zammad 3.3X to 3.4.0-4.

I had a working docker install of Zammad v 3.3.x running behind an nginx proxy on the host which handles SSL (this host itself is behind cloudflare proxy which provides the SSL certs). I solved prior CSRF token validation issues by using the zammad.conf provided here.

I upgraded this morning by pulling the latest version of this repo, docker-compose pulling the latest images, and then docker-compose up. Now I have the CSRF token validation error again on trying to log in. I have tried all the prior fixes that I could find from issues here at github and from the zammad forum but nothing works.

I have a bunch of customer service agents that are going to arrive at their desks in an hour or two who are not going to be able to log in to Zammad. Any help appreciated.

I'm guessing that this issue is somehow related to #166 (by @monotek) but I don't know enough about what those commits are doing to figure out what to try.

pjv commented 4 years ago

this is the nginx config from the host for proxying to zammad:

server {
  listen 80;
  server_name helpdesk.example.com;
  return 301 https://helpdesk.example.com$request_uri;
}

server {
  listen 443 ssl;
  server_name helpdesk.example.com;

  # SSL configs
  ssl_certificate /var/www/certs/bt.com.cert.pem;
  ssl_certificate_key /var/www/certs/bt.com.key.pem;
  ssl_dhparam /etc/nginx/dhparams.pem;
  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:20m;
  ssl_session_timeout 180m;

  access_log   /var/log/nginx/helpdesk.example.com.access.log;
  error_log    /var/log/nginx/helpdesk.example.com.error.log;

  location / {
    #proxy_pass http://localhost:5000/;
    proxy_pass http://127.0.0.1:5000/;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forward-Proto https;
    proxy_set_header X-Nginx-Proxy true;
    proxy_redirect off;
  }
}

And the docker-compose-override.yml looks like this:

version: '2'
services:

  zammad-nginx:
    ports:
      - "5000:80"
    volumes:
      - ./custom/zammad.conf:/opt/zammad/contrib/nginx/zammad.conf
MrGeneration commented 4 years ago

We had this topic quite a lot already: https://github.com/zammad/zammad-docker-compose/search?q=csrf&type=Issues

You can fix this by adjusting your configuration.

monotek commented 4 years ago

You can configure this now via env variable. See: https://github.com/zammad/zammad-docker-compose/blob/master/containers/zammad/docker-entrypoint.sh#L14

pjv commented 4 years ago

Thanks @MrGeneration, I have read all of those issues and tried to apply all of the configuration changes suggested in them. As a result of that I had a WORKING Zammad installation of v. 3.3.x. I upgraded to 3.4.0-4 without making any changes to my previously working configuration and now I am again having this CSRF token validation issue.

I would not have posted a new issue if any of the issues in the search you listed had helped me. I have already tried all of those fixes and am left with a not-log-in-able server.

pjv commented 4 years ago

Thank you @monotek. Sorry to be dense but I cannot understand from reading that source how to configure whatever I should configure to get past the CSRF token validation error.

Can you suggest which variable I should define in the .env file to get back to a working config?

pjv commented 4 years ago

In case anyone else runs into the same issue that I did, my issue had nothing to do with #166, nothing to do with configuring environment variables, and nothing to do with any of the nginx configs.

Upgrading Zammad changed the HTTP type setting for my instance which I had configured to http for running behind an SSL terminating nginx proxy to https. I don't know if there is an init default somewhere that gets overwritten or what, but I definitely had it intentionally set to http and the upgrade changed it to https.

2020-07-16 at 9 00 AM

Since I could no longer log in to fix this via the web UI, I had to do the following to solve this issue:

docker exec -it zammad-docker-compose_zammad-railsserver_1 bash
rails c
Setting.get('http_type')
=> "https"
Setting.set('http_type','http')
quit
exit
docker-compose down
docker-compose up -d
MrGeneration commented 4 years ago

Zammad does detect the to be used http type during running the installation wizard. If at this point this is not yet correct, this has to be adjusted - e.g. like you noted above.

pjv commented 4 years ago

Well, I ran into this on an upgrade of a working Zammad install (from 3.3.x to 3.4.x) not on a new install and the upgrade for whatever reason over-wrote my already-configured HTTP type setting. I did not go through the installation wizard after the upgrade, so either some part of the install wizard is happening invisibly behind the scenes or that detection must also be happening outside of the installation wizard.

MrGeneration commented 4 years ago

There's no migration that touches your http_type Setting. This shouldn't happen automatically (after once set via installation way back in the past) at all.

pjv commented 4 years ago

I agree it shouldn't. You can either believe me or not that it did. Not trying to blame anyone, but in case it seems worth looking for how that could have happened, that's really what happened on my server.

[edit: if I was a rails dev, I'd take a look for it, but ruby code is greek to me - sorry.]

monotek commented 4 years ago

With the new Version the value for "proxy_set_header X-Forwarded-Proto" is configurable.

It sets "$scheme;" as its the default, maybe overwriting changes you did before. The default is taken from here: https://github.com/zammad/zammad/blob/develop/contrib/nginx/zammad.conf#L39

So its likely you want to get an "https" there.

Therefore start everything via:

docker-compose run -e NGINX_SERVER_SCHEME=https
pjv commented 4 years ago

@monotek thank you for getting back to me on that. I did actually try setting exactly that variable and value but it did not solve my issue which I detailed above.

MrGeneration commented 4 years ago

I'll have a look if I'll automate setting http or https to be set as setting as well automatically. That proberbly fixes the issue.

ETA unknown.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

ethrgeist commented 3 years ago

I can confirm that this is an issue, even with new installs.

I was perfectly able to install Zammad behind our Traefik (still v1) and configure everything, after i logged out for the first time, i was not able to log back in again. The csrf token failed every time. I tried various things i found via search engine (eg https://github.com/zammad/zammad-docker-compose/issues/51, https://community.zammad.org/t/problem-accessing-zammad-behind-traefik/5044) but nothing helped.

docker exec -it zammad-docker-compose_zammad-railsserver_1 bash
rails c
Setting.get('http_type')
=> "https"
Setting.set('http_type','http')
quit
exit
docker-compose down
docker-compose up -d

Really did the trick for being able to login in again.

Requesting login via configured IdP (SAML) still fails (template doesnt event get rendered correctly) tho.

zammad-saml

Logs read:

zammad-railsserver_1    | I, [2020-12-04T11:05:06.494610 #1-47350277365700]  INFO -- : Started POST "/auth/saml" for 172.26.0.4 at 2020-12-04 11:05:06 +0000
zammad-railsserver_1    | I, [2020-12-04T11:05:06.505845 #1-47350277365700]  INFO -- : (saml) Request phase initiated.
zammad-railsserver_1    | F, [2020-12-04T11:05:06.508011 #1-47350277365700] FATAL -- :   
zammad-railsserver_1    | F, [2020-12-04T11:05:06.508110 #1-47350277365700] FATAL -- : ActionController::InvalidAuthenticityToken (ActionController::InvalidAuthenticityToken):
zammad-railsserver_1    | F, [2020-12-04T11:05:06.508192 #1-47350277365700] FATAL -- :   
zammad-railsserver_1    | F, [2020-12-04T11:05:06.508280 #1-47350277365700] FATAL -- : omniauth-rails_csrf_protection (0.1.2) lib/omniauth/rails_csrf_protection/token_verifier.rb:34:in `call'
zammad-railsserver_1    | omniauth (1.9.1) lib/omniauth/strategy.rb:209:in `request_call'
zammad-railsserver_1    | omniauth (1.9.1) lib/omniauth/strategy.rb:188:in `call!'
zammad-railsserver_1    | omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
zammad-railsserver_1    | omniauth (1.9.1) lib/omniauth/strategy.rb:192:in `call!'
zammad-railsserver_1    | omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
zammad-railsserver_1    | omniauth (1.9.1) lib/omniauth/strategy.rb:192:in `call!'
zammad-railsserver_1    | omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
zammad-railsserver_1    | omniauth (1.9.1) lib/omniauth/strategy.rb:192:in `call!'
zammad-railsserver_1    | omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
zammad-railsserver_1    | omniauth (1.9.1) lib/omniauth/strategy.rb:192:in `call!'
zammad-railsserver_1    | omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
zammad-railsserver_1    | omniauth (1.9.1) lib/omniauth/strategy.rb:192:in `call!'
zammad-railsserver_1    | omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
zammad-railsserver_1    | omniauth (1.9.1) lib/omniauth/strategy.rb:192:in `call!'
zammad-railsserver_1    | omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
zammad-railsserver_1    | omniauth (1.9.1) lib/omniauth/strategy.rb:192:in `call!'
zammad-nginx            | 172.26.0.4 - - [04/Dec/2020:11:05:06 +0000] "POST /auth/saml HTTP/1.1" 422 727 "https://helpdesk.example.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0"
zammad-railsserver_1    | omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
zammad-railsserver_1    | omniauth (1.9.1) lib/omniauth/strategy.rb:192:in `call!'
zammad-railsserver_1    | omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
zammad-railsserver_1    | omniauth (1.9.1) lib/omniauth/strategy.rb:192:in `call!'
zammad-railsserver_1    | omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
zammad-railsserver_1    | omniauth (1.9.1) lib/omniauth/builder.rb:45:in `call'
zammad-railsserver_1    | rack (2.2.3) lib/rack/tempfile_reaper.rb:15:in `call'
zammad-railsserver_1    | rack (2.2.3) lib/rack/etag.rb:27:in `call'
zammad-railsserver_1    | rack (2.2.3) lib/rack/conditional_get.rb:40:in `call'
zammad-railsserver_1    | rack (2.2.3) lib/rack/head.rb:12:in `call'
zammad-railsserver_1    | actionpack (5.2.4.4) lib/action_dispatch/http/content_security_policy.rb:18:in `call'
zammad-railsserver_1    | rack (2.2.3) lib/rack/session/abstract/id.rb:266:in `context'
zammad-railsserver_1    | rack (2.2.3) lib/rack/session/abstract/id.rb:260:in `call'
zammad-railsserver_1    | actionpack (5.2.4.4) lib/action_dispatch/middleware/cookies.rb:670:in `call'
zammad-railsserver_1    | actionpack (5.2.4.4) lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
zammad-railsserver_1    | activesupport (5.2.4.4) lib/active_support/callbacks.rb:98:in `run_callbacks'
zammad-railsserver_1    | actionpack (5.2.4.4) lib/action_dispatch/middleware/callbacks.rb:26:in `call'
zammad-railsserver_1    | actionpack (5.2.4.4) lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
zammad-railsserver_1    | actionpack (5.2.4.4) lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
zammad-railsserver_1    | railties (5.2.4.4) lib/rails/rack/logger.rb:38:in `call_app'
zammad-railsserver_1    | railties (5.2.4.4) lib/rails/rack/logger.rb:28:in `call'
zammad-railsserver_1    | actionpack (5.2.4.4) lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
zammad-railsserver_1    | actionpack (5.2.4.4) lib/action_dispatch/middleware/request_id.rb:27:in `call'
zammad-railsserver_1    | rack (2.2.3) lib/rack/method_override.rb:24:in `call'
zammad-railsserver_1    | rack (2.2.3) lib/rack/runtime.rb:22:in `call'
zammad-railsserver_1    | activesupport (5.2.4.4) lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
zammad-railsserver_1    | actionpack (5.2.4.4) lib/action_dispatch/middleware/executor.rb:14:in `call'
zammad-railsserver_1    | rack (2.2.3) lib/rack/sendfile.rb:110:in `call'
zammad-railsserver_1    | railties (5.2.4.4) lib/rails/engine.rb:524:in `call'
zammad-railsserver_1    | puma (3.12.6) lib/puma/configuration.rb:227:in `call'
zammad-railsserver_1    | puma (3.12.6) lib/puma/server.rb:706:in `handle_request'
zammad-railsserver_1    | puma (3.12.6) lib/puma/server.rb:476:in `process_client'
zammad-railsserver_1    | puma (3.12.6) lib/puma/server.rb:334:in `block in run'
zammad-railsserver_1    | puma (3.12.6) lib/puma/thread_pool.rb:135:in `block in spawn_thread'
zammad-railsserver_1    | logging (2.2.2) lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
zammad-nginx            | 172.26.0.4 - - [04/Dec/2020:11:05:06 +0000] "GET /favicon.ico HTTP/1.1" 200 32988 "https://helpdesk.example.net/auth/saml" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0"
MrGeneration commented 3 years ago

The temporary solution lifes in the history of this issue. In the mean time, please apply these and don't recycle / push this issue more than needed.

ethrgeist commented 3 years ago

My primary focus was to add that the SAML login seems to be hit as well and that for the SAML login the provided temp fix is not working.

MrGeneration commented 3 years ago

@eth3rgeist then your comment doesn't fit in this issue.

ethrgeist commented 3 years ago

I was under the impression that my CSRF problem would fit this issue titled "New CSRF validation issue on upgrade to 3.4.0-4" since it would extend this to not only hit installments on upgrade but on fresh installations as well and not only the basic login but also extend ones.

But okay, sorry for the trouble caused.

MrGeneration commented 3 years ago

I don't see a valid reason why, after patching, one login method works and the other doesn't. In my opinion that's a technical question that needs deeper dive. This is out of scope of github and especially this issue. Sorry.

ethrgeist commented 3 years ago

I get it that you do not want to give me any technical support (your stance on TLS proxies other than the nginx coming with the zammad package is perfectly clear communicated in multiple posts).

But in this case the login fails nearly instant with the internal CSRF check inside the zammad stack, we are never getting to a point were a possible error in the SAML configuration comes into play (No request ever reached our IdP). That's why i attached the log, first 6 lines say it all, i just attached the rest of the stack to help you see something. (Also that error template is garbled, i'm pretty sure it should not look that way.)

stopak commented 3 years ago

In case anyone else runs into the same issue that I did, my issue had nothing to do with #166, nothing to do with configuring environment variables, and nothing to do with any of the nginx configs.

Upgrading Zammad changed the HTTP type setting for my instance which I had configured to http for running behind an SSL terminating nginx proxy to https. I don't know if there is an init default somewhere that gets overwritten or what, but I definitely had it intentionally set to http and the upgrade changed it to https.

2020-07-16 at 9 00 AM

Since I could no longer log in to fix this via the web UI, I had to do the following to solve this issue:

docker exec -it zammad-docker-compose_zammad-railsserver_1 bash
rails c
Setting.get('http_type')
=> "https"
Setting.set('http_type','http')
quit
exit
docker-compose down
docker-compose up -d

This one resolved my issue. It was not even an upgrade but restart of the system caused http_type to be changed to https

monotek commented 3 years ago

@MrGeneration @thorsteneckel Is somewhere in the code tried to set the value automatically by some detection mechanism or how can a change be explained? If not, whats the default value? Maybe we can also add an env var to the container to set it? Is a restart of zammad needed after setting it? Asking because the container is restarted in the comment above.

thorsteneckel commented 3 years ago

So I double checked the code base and found no place where the value of the http_type Setting is changed automatically. The Setting default value is http for 2 years now. Its value is set when running the initial getting started setup wizard or manually via the admin interface. There is no restart of the whole application required after changing the HTTP type. A browser refresh is sufficient. Very strange 🤔

pjv commented 3 years ago

Very strange

Yup. I’ve now done two additional upgrades (to 3.5x and 3.6x) using the exact same steps as I used when I did the original problematic upgrade to 3.4x that prompted me to open this ticket and have not had a repeat of the issue occur.

thorsteneckel commented 3 years ago

Thanks for the heads up @pjv ! Would be interesting on what version (from -> to) change others faced this issue!

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

TuringTux commented 3 years ago

Apologies for necrobumping this issue, but I am unaware of any other feasible way to get in contact with users that previously commented here. I do understand that what follows is likely a technical question and therefore not going to be resolved by the Zammad developers – my only goal is it to establish contact to a community member who might be able to help me. Having said this:

@eth3rgeist Have you found a solution to the SAML problem? I have a pretty similar setup (Zammad 4.0 behind a Traefik v2 reverse proxy). Normal login works (only when the HTTP type is set to http), but SAML login attempts always directly result in a 422. In the logs, I also get an ActionController::InvalidAuthenticityToken error.

I would really love to hear from you :)

MrGeneration commented 3 years ago

If you want to get in touch with community users, the best way would be the Zammad community https://community.zammad.org . No guaranteed help, but still, probably the better place.

I'm sorry, but I'm limiting the conversation of this issue to contributors only.