search

thomseddon / traefik-forward-auth

Minimal forward authentication service that provides Google/OpenID oauth based login and authentication for the traefik reverse proxy
MIT License
2.11k stars 405 forks source link

Not Authorized error #31

Closed osotechie closed 4 years ago

osotechie commented 5 years ago

Hi,

I've just setup traefik-forward-auth docker container for the first time, and after some playing in both overlay and auth-host modes I get the same "Not Authroized" error displayed.

I ran the container up with debug mode and am getting the following error: Error validating csrf cookie: CSRF cookie does not match state

This is my config (with secrets removed of course):

traefik-forward-auth container: Traefik-Forward-Auth: container_name: Traefik-Forward-Auth hostname: Traefik-Forward-Auth image: thomseddon/traefik-forward-auth networks: IoT: ipv4_address: 10.1.11.253 environment:

whoami container which I'm trying to forward auth as a test: Whoami: container_name: Whoami hostname: whoami image: containous/whoami networks: IoT: ipv4_address: 10.1.11.251 labels:

Traefik - General

- "traefik.enable=true"
- "traefik.frontend.entryPoints=http,https"
- "traefik.default.protocol=http"
- "traefik.default.port=80"
- "traefik.frontend.rule=Host:whoami.<domain>"
# Traefik - SSL
- "traefik.frontend.headers.SSLRedirect=true"
- "traefik.frontend.headers.STSSeconds=315360000"
- "traefik.frontend.headers.browserXSSFilter=true"
- "traefik.frontend.headers.contentTypeNosniff=true"
- "traefik.frontend.headers.forceSTSHeader=true"
- "traefik.frontend.headers.SSLHost=<domain>"
- "traefik.frontend.headers.STSIncludeSubdomains=true"
- "traefik.frontend.headers.STSPreload=true"
- "traefik.frontend.headers.frameDeny=true"
# Traefik - Whitelisting
- "traefik.frontend.auth.forward.address=http://10.1.11.253:4181"
- traefik.frontend.auth.forward.authResponseHeaders = ["X-Forwarded-User"]

As I am using labels for everything instead of making changes to the treafik.toml file I added the auth.forward.address and auth.forward.authResponseHeaders to the treafik-forward-auth container as I saw something around this in another post when using auth-host mode. Which solved my endless login loop when it wasnt in there.

I have gone through as setup the Google side as per the instructions, and added https://auth./_oauth and https://whoami./_oauth (for when I was testing in overlay mode.

Any ideas, I have tried multiple options around the DOMAIN and WHITELIST options, and multiple google accounts. And all give me the same problem. The error probably suggests its not something with the accounts Im using somwhere else Im guessing?

Thanks in advance. O

Originally posted by @owendemooy in https://github.com/thomseddon/traefik-forward-auth/issues/20#issuecomment-468239480

MortenRickiRasmussen commented 5 years ago

I Received that too and i seems to be some cookie/cache related issue. Either wait a couple of hours or try a different browser/ Incognito

osotechie commented 5 years ago

I Received that too and i seems to be some cookie/cache related issue. Either wait a couple of hours or try a different browser/ Incognito

OMG,so in all my testing I had forgot to try Incognito mode... I had tried both Chrome and Edge... But that was prob whilst the config was not right... so retested with Incognito mode and hey presto its working...

Thanks this can be closed 👍

ghost commented 5 years ago

I don't think this should be closed. I run into this problem frequently. It may be because we need a traefik header

ghost commented 5 years ago

Related https://github.com/thomseddon/traefik-forward-auth/issues/20

thomseddon commented 5 years ago

Hmm, I have a similar issue where I've clicked the wrong google account, my only option is to clear cookies and try again - sounds like this could be the same root cause - I wonder if there is a better UX for this?

ghost commented 5 years ago

It does happen if you do choose the wrong account or it happens once the authorization expires with the right account.

I believe it may be due to a missing header on traefik and/or its like it can't refresh the token on it's own.

thomseddon commented 5 years ago

@robertbaker what do you mean it's missing a header?

ghost commented 5 years ago

In the #20 I believe it's mentioned. The Forwarded URI or auth trust heaters.

Although I think it's more likely a bug where its not refreshing the expired token

On Tue, Mar 19, 2019, 9:07 AM Thom Seddon notifications@github.com wrote:

@robertbaker https://github.com/robertbaker what do you mean it's missing a header?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/thomseddon/traefik-forward-auth/issues/31#issuecomment-474446981, or mute the thread https://github.com/notifications/unsubscribe-auth/AA3YJQccmkPKzNmrrTAL6W3Ncj8AumK7ks5vYQtcgaJpZM4bWnaR .

726a67 commented 5 years ago

I experience the same issue intermittently in auth host mode, via both desktop and mobile browsers.

time="2019-03-22T13:48:49Z" level=warning msg="Error validating csrf cookie: CSRF cookie does not match state" RemoteAddr="x.x.x.x:xxxx" csrf=<xyz> state="<zyx>:https://<domain>/"

As @robertbaker stated, this seems to happen most frequently when re-selecting an account which has been authorized (in my case, via WHITELIST) after the session lifetime has expired. Perhaps a token refresh issue?

sharkoz commented 5 years ago

I can confirm I have the same error even after deleting my cookies...

whi-tw commented 5 years ago

I don’t suppose a “logout” endpoint is in the works?

zetas commented 5 years ago

FWIW I've been struggling with this issue for months with no good workaround except to logout of all google accounts, delete all google related cookies and cache, and close and re-open the browser. It happens every month or so and it's incredibly annoying.

The incognito mode fix does work for me though so I think it's safe to say it's something to do with caching and tokens. When I get the "not authorized" message I also get the CSRF error log mentioned by @726a67. I hope we come up with a fix for this soon.

ghost commented 5 years ago

@zetas simply completely closing chrome (including tray icon) should be the only thing you need to do to fix it.

Queuecumber commented 5 years ago

Getting the same error intermittently, completely closing chrome didnt fix it

Queuecumber commented 5 years ago

The weird thing is that it lets me access some of my endpoints but not others when this happens. E.g. some work fine some give the "not authorized" text

thomseddon commented 5 years ago

So I think I might have found the cause of this, after a cookie expires the existing behavior was just to throw an error.

I have just pushed a change which means that the users will be redirected to Google login instead.

If someone would be willing to test the latest tag on docker, with error logging enabled, that would be hugely helpful. I know this can take some time to replicate, so if anyone has error logs enabled then there should be more info in that log already, so if you could post existing logs that would also help!

Queuecumber commented 5 years ago

Just tried it, still getting the error. My log level is set to debug, let me know what to look for

thomseddon commented 5 years ago

Can you double check you're running the image with the tag: 3e6ccc8f

Do you get any logs with the 401?

For extra info, 401 can returned from 4 places:

There's a log message in each location, so if tfa is returning an error then it should be showing more info in the logs

Queuecumber commented 5 years ago

Can you double check you're running the image with the tag: 3e6ccc8

I wasn't able to come up with a good way to check this. The only docker tag on that is latest and there apparently isn't a shell in the container I could use to check the source. I restarted a couple of times, so it should have pulled the image.

Do you get any logs with the 401?

I actually do not have any 401 errors in the log

I have 

logger.Warn("Missing csrf cookie")

I was grepping on "401" before, I didn't realize the logs dont actually include the http status code.

Queuecumber commented 5 years ago

One thing that seems to work consistently is getting to the "not authorized" page, then visiting a different endpoint that for whatever reason works, then going back to my original endpoint lets me continue to the page I wanted.

whi-tw commented 5 years ago

I've done a thing in branch add-logout-endpoint on my fork that, in theory, adds a /_tfa-logout endpoint which will clear the local _forward_auth cookie.

I have written tests, which pass, but I want to do some functional testing before I actually submit a PR. However, I flubbed an unrelated IP on my home setup, and so can't actually access my own environment to deploy the updated version, so I'll do it tomorrow. Thanks, sslh.

I have built the image manually on https://hub.docker.com/r/tnwhitwell/traefik-forward-auth/tags, tag: testing, but obviously I'd recommend building and deploying your own version if you do want to test.

Queuecumber commented 5 years ago

I actually added a logout endpoint to my landing page which does the same thing, I've also been manually deleting the cookie and disabling caching before running these tests so I don't know if that will solve the problem

thomseddon commented 5 years ago

Could you do a docker inspect on the container, mine is showing: Image": "sha256:97d5ed375d0c1fa946b74d4381ab912199056da56d5c2af7a78c87ee074d240f"

I actually think my changes in https://github.com/thomseddon/traefik-forward-auth/commit/3e6ccc8f45c64b40d4f273fcd6603456cefa2d6c are a red herring (which I will partially revert), the browser will expire the cookie after that time so it is actually an error if such a cookie is received.

Missing csrf cookie suggests you're hitting the callback endpoint, you should only hit this endpoint after you've done the login on Google?

Queuecumber commented 5 years ago

Could you do a docker inspect on the container, mine is showing: Image": "sha256:97d5ed375d0c1fa946b74d4381ab912199056da56d5c2af7a78c87ee074d240f"

I'll double check the image hash

Missing csrf cookie suggests you're hitting the callback endpoint, you should only hit this endpoint after you've done the login on Google?

Indeed it's directly after I log into google that I get this screen, which is part of why it makes no sense since I've just logged in.

Do you rely on the X-Forwarded-User header to check the username at the callback endpoint? I don't think I configured traefik to send it for that endpoint

thomseddon commented 5 years ago

Interesting, could you check what cookies are sent before you're redirected to Google (you can inspect the headers on the 307 redirect in your browser)? Are you using the auth host?

Queuecumber commented 5 years ago

Sorry for the late reply, I'm in the process of rebuilding that cluster right now, when it's settled down I'll look into the cookies for you

Queuecumber commented 5 years ago

In the request to my endpoint, I see 

cookie: _forward_auth_csrf=0adc70a3cbde78d0b09e37e7c9602c83

The 307 response has 

set-cookie: _forward_auth_csrf=cabca8a0d088bddc863932ec8704c953; Path=/; Domain=<redacted correct domain>; Expires=Wed, 19 Jun 2019 00:10:51 GMT; HttpOnly; Secure
thomseddon commented 5 years ago

Hmm, so that's why it doesn't work - but why doesn't the cookie match? My first thought would be a domain missmatch?

Queuecumber commented 5 years ago

Domain is correct and if it was that I would expect it to fail for every domain. The only obvious difference is that the working domains set x-forwarded-user and the failing ones don't

westfood commented 5 years ago

I am experiencing exactly same issue as Queuecumber. "Not authorized" after google authentication. Docker Image: 97d5ed375d0c. Traefik services handle traffic on port 8080.

Command is: --cookie-domain=somewhere.com --cookie-domain=test.somewhere.com --cookie-domain=auth.test.somewhere.com In my env: AUTH_HOST: auth.test.somewhere.com:8080

services I am authenticating are path based m1-eu1.test.somewhere.com:8080/traefik/dashboard or m1-eu1.test.somewhere.com:8080/cadvisor

frankforpresident commented 4 years ago

Same issue here, After google login I get "Not authorized".. No clue yet why msg="Missing csrf cookie"

thomseddon commented 4 years ago

@frankforpresident (or anyone else in the thread) could you post:

I'm sure we can get to the bottom of this :)

burntoc commented 4 years ago

I can confirm I'm seeing the same behavior here. Initiating an incognito browser allows me to access the service properly though.

frankforpresident commented 4 years ago

@thomseddon, I did my best to gather some information for you. My setup was based on: https://www.smarthomebeginner.com/google-oauth-with-traefik-docker/

Description: Open chrome in incognite mode (or normal). Visit the configured url and login with configured email address.

Expected: Proper redirection after the authentication. Content is visible.

Actual: After login the webpage redirects and returns with 401 Not authorized.

Note: I did changed some personal data and revoked the 0auth cert just after this run.

Logs:

time="2020-01-20T22:20:42Z" level=debug msg="Starting with options: {\"LogLevel\":\"debug\",\"LogFormat\":\"text\",\"AuthHost\":\"oauth.fakedomainforissue31.com\",\"CookieDomains\":[{\"Domain\":\"fakedomainforissue31.com\",\"DomainLen\":16,\"SubDomain\":\".fakedomainforissue31.com\",\"SubDomainLen\":17}],\"InsecureCookie\":true,\"CookieName\":\"_forward_auth\",\"CSRFCookieName\":\"_forward_auth_csrf\",\"DefaultAction\":\"auth\",\"Domains\":null,\"LifetimeString\":0,\"Path\":\"/_oauth\",\"Whitelist\":[\"fakemailforissue31@gmail.com\"],\"Providers\":{\"Google\":{\"ClientId\":\"xxxxxxxxxxxxx.apps.googleusercontent.com\",\"Scope\":\"https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/userinfo.email\",\"Prompt\":\"\",\"LoginURL\":{\"Scheme\":\"https\",\"Opaque\":\"\",\"User\":null,\"Host\":\"accounts.google.com\",\"Path\":\"/o/oauth2/auth\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"TokenURL\":{\"Scheme\":\"https\",\"Opaque\":\"\",\"User\":null,\"Host\":\"www.googleapis.com\",\"Path\":\"/oauth2/v3/token\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"UserURL\":{\"Scheme\":\"https\",\"Opaque\":\"\",\"User\":null,\"Host\":\"www.googleapis.com\",\"Path\":\"/oauth2/v2/userinfo\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"}}},\"Rules\":{},\"Lifetime\":0,\"CookieDomainsLegacy\":null,\"CookieSecureLegacy\":\"\",\"ClientIdLegacy\":\"\",\"PromptLegacy\":\"\"}"
time="2020-01-20T22:20:42Z" level=info msg="Listening on :4181"
time="2020-01-20T22:21:10Z" level=debug msg="Authenticating request" headers="map[Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9] Accept-Encoding:[gzip, deflate, br] Accept-Language:[en-US,en;q=0.9,nl;q=0.8,af;q=0.7] Sec-Fetch-Mode:[navigate] Sec-Fetch-Site:[none] Sec-Fetch-User:[?1] Upgrade-Insecure-Requests:[1] User-Agent:[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36] X-Forwarded-For:[192.168.0.1] X-Forwarded-Host:[debug.fakedomainforissue31.com] X-Forwarded-Method:[GET] X-Forwarded-Proto:[https] X-Forwarded-Uri:[/]]" rule=default source_ip=192.168.0.1
time="2020-01-20T22:21:10Z" level=debug msg="Set CSRF cookie and redirecting to google login" source_ip=192.168.0.1
time="2020-01-20T22:21:10Z" level=debug msg=Done source_ip=192.168.0.1
time="2020-01-20T22:21:13Z" level=debug msg="Handling callback" headers="map[Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9] Accept-Encoding:[gzip, deflate, br] Accept-Language:[en-US,en;q=0.9,nl;q=0.8,af;q=0.7] Referer:[https://accounts.google.com/signin/oauth/oauthchooseaccount?client_id=xxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com&as=siqf3xKp8E6XuK8hiWhYSw&destination=https%3A%2F%2Foauth.fakedomainforissue31.com&approval_state=!ChRjUnBHenZ0aWlkMlJOWjB1UmZBZRIfODFnTTFFUjNYR1VkY0t0V0xtY192ZHBvNWJsUV9CWQ%E2%88%99AJDr988AAAAAXid5VqkCKO41SE29VN3yjfxvWPdXqezJ&oauthgdpr=1&xsrfsig=ChkAeAh8TzAGW13wZT6mNVUkgf1VI3Un__LKEg5hcHByb3ZhbF9zdGF0ZRILZGVzdGluYXRpb24SBXNvYWN1Eg9vYXV0aHJpc2t5c2NvcGU&flowName=GeneralOAuthFlow] Sec-Fetch-Mode:[navigate] Sec-Fetch-Site:[cross-site] Sec-Fetch-User:[?1] Upgrade-Insecure-Requests:[1] User-Agent:[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36] X-Forwarded-For:[192.168.0.1] X-Forwarded-Host:[oauth.fakedomainforissue31.com] X-Forwarded-Method:[GET] X-Forwarded-Proto:[https] X-Forwarded-Uri:[/_oauth?state=f480a65f3c0c40bc1ec395dd5884d862%3Ahttps%3A%2F%2Fdebug.fakedomainforissue31.com%2F&code=4%2FvgFS7t4mz4NGZPiFu_WqGI4IM-rwRbZpSLkhC8lsjakaQ3wZcRe1VJIuXxL_8bzruh7dP5LfKUGJraM3UyPUQc4&scope=email+profile+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.profile+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+openid&authuser=0&prompt=consent&session_state=efcdb271432c17ec7c1ac1b246b908fbc2dd43eb..413a]]" rule=default source_ip=192.168.0.1
time="2020-01-20T22:21:13Z" level=warning msg="Missing csrf cookie" source_ip=192.168.0.1
time="2020-01-20T22:21:13Z" level=debug msg="Authenticating request" headers="map[Accept:[image/webp,image/apng,image/*,*/*;q=0.8] Accept-Encoding:[gzip, deflate, br] Accept-Language:[en-US,en;q=0.9,nl;q=0.8,af;q=0.7] Referer:[https://oauth.fakedomainforissue31.com/_oauth?state=f480a65f3c0c40bc1ec395dd5884d862%3Ahttps%3A%2F%2Fdebug.fakedomainforissue31.com%2F&code=4%2FvgFS7t4mz4NGZPiFu_WqGI4IM-rwRbZpSLkhC8lsjakaQ3wZcRe1VJIuXxL_8bzruh7dP5LfKUGJraM3UyPUQc4&scope=email+profile+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.profile+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+openid&authuser=0&prompt=consent&session_state=efcdb271432c17ec7c1ac1b246b908fbc2dd43eb..413a] Sec-Fetch-Mode:[no-cors] Sec-Fetch-Site:[same-origin] User-Agent:[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36] X-Forwarded-For:[192.168.0.1] X-Forwarded-Host:[oauth.fakedomainforissue31.com] X-Forwarded-Method:[GET] X-Forwarded-Proto:[https] X-Forwarded-Uri:[/favicon.ico]]" rule=default source_ip=192.168.0.1
time="2020-01-20T22:21:13Z" level=debug msg="Set CSRF cookie and redirecting to google login" source_ip=192.168.0.1
time="2020-01-20T22:21:13Z" level=debug msg=Done source_ip=192.168.0.1

docker-compose.yaml

whoami:
    container_name: debug
    hostname: debug
    image: containous/whoami
    labels:
      - traefik.frontend.rule=Host:debug.fakedomainforissue31.com
      - traefik.port=80
      - traefik.backend=whoami
      - traefik-docker.network=internal
      - traefik.frontend.auth.forward.address=http://oauth:4181
      - traefik.frontend.auth.forward.authResponseHeaders=X-Forwarded-User
      - traefik.frontend.auth.forward.trustForwardHeader=true
      - traefik.redirectorservice.frontend.entryPoints=http
      - traefik.redirectorservice.frontend.redirect.entryPoint=https
      - traefik.webservice.frontend.entryPoints=https

oauth:
    image: thomseddon/traefik-forward-auth:latest
    container_name: oauth
    hostname: oauth
    restart: always
    env_file:
      - oauth.env
    labels:
      - traefik.enable=true
      - traefik.port=4181
      - traefik.backend=oauth
      - traefik.docker.network=internal
      - traefik.frontend.rule=Host:oauth.fakedomainforissue31.com
      - traefik.frontend.headers.SSLHost=oauth.fakedomainforissue31.com
      - traefik.frontend.auth.forward.address=http://oauth:4181
      - traefik.frontend.auth.forward.authResponseHeaders=X-Forwarded-User
      - traefik.frontend.auth.forward.trustForwardHeader=true
      - traefik.frontend.headers.SSLRedirect=true
      - traefik.frontend.headers.browserXSSFilter=true
      - traefik.frontend.headers.contentTypeNosniff=true
      - traefik.frontend.headers.forceSTSHeader=true
      - traefik.frontend.headers.STSSeconds=315360000
      - traefik.frontend.headers.STSIncludeSubdomains=true
      - traefik.frontend.headers.STSPreload=true
      - traefik.frontend.headers.frameDeny=true
      - traefik.frontend.passHostHeader=true
      - traefik.frontend.headers.SSLForceHost=true
      - traefik.frontend.headers.customResponseHeaders=X-Robots-Tag:noindex,nofollow,nosnippet,noarchive,notranslate,noimageindex
      - traefik.redirectorservice.frontend.entryPoints=http
      - traefik.redirectorservice.frontend.redirect.entryPoint=https
      - traefik.webservice.frontend.entryPoints=https

oauth.env

PROVIDERS_GOOGLE_CLIENT_ID=XXXXXXXXXXXXXXXXXX.apps.googleusercontent.com
PROVIDERS_GOOGLE_CLIENT_SECRET=XXXXXXXXXXXXXX
SECRET=XXXXXXXXXXXXXXX
COOKIE_DOMAIN=fakedomainforissue31.com
INSECURE_COOKIE=true
AUTH_HOST=oauth.fakedomainforissue31.com
URL_PATH=/_oauth
WHITELIST=fakemailforissue31@gmail.com
LOG_LEVEL=debug
LIFETIME=2592000 # 30 days
OIDC_ISSUER=https://accounts.google.com
CSRF_COOKIE_NAME=_forward_auth_csrf

Note: I also tried INSECURE_COOKIE=false, changing _forward_auth_csrf to a different key and a different email address

How do you want me to collect the request logs? or is this enough to get started?

I did saw this message in the console after landing on the 401 page.

/_oauth?state=014308…49a7a48f6c..576a#:1 A cookie associated with a cross-site resource at https://accounts.google.com/ was set without the `SameSite` attribute. A future release of Chrome will only deliver cookies with cross-site requests if they are set with `SameSite=None` and `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.
/_oauth?state=014308…49a7a48f6c..576a#:1 A cookie associated with a cross-site resource at http://google.com/ was set without the `SameSite` attribute. A future release of Chrome will only deliver cookies with cross-site requests if they are set with `SameSite=None` and `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.
/_oauth?state=014308…49a7a48f6c..576a#:1 A cookie associated with a cross-site resource at https://google.com/ was set without the `SameSite` attribute. A future release of Chrome will only deliver cookies with cross-site requests if they are set with `SameSite=None` and `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.
Cross-Origin Read Blocking (CORB) blocked cross-origin response https://accounts.google.com/o/oauth2/auth?client_id=110929115210-iebt17j9me…b0168a5…%3Ahttps%3A%2F%2Foauth.computerheld.net%2Ffavicon.ico with MIME type text/html. See https://www.chromestatus.com/feature/5629709824032768 for more details.

and I can't seem to find the_forward_auth_csrf cookie after the redirect under Application>Storage>Cookies. Not sure if this is the cause of my problem.

frankforpresident commented 4 years ago

@thomseddon, (or anyone else) Any update on this issue? Is there something I could help you with? If more logs are required don't hesitate to contact me.

I would love to implement this in my setup.

thomseddon commented 4 years ago

Sorry I missed your previous comment, thanks for the detailed info. Could you post your entire compose config (including traefik) - the absolute jackpot would be if you can give me a docker-compose.yaml + associated config so I can exactly replicate it?

With regards to capturing the requests, if you export them from the chrome network tab, that would be great (although if I can replicate that won't be needed)

0xknon commented 4 years ago

I am having the same issue here. I guess my config are more naive.

Error: time="2020-02-27T08:25:03Z" level=warning msg="Missing csrf cookie" source_ip=192.168.9.195

Traefik (installed with helm - stable/traefik): values.yaml:

ssl:
  enabled: true
  enforced: true
  insecureSkipVerify: true
dashboard:
  enabled: true
  domain: traefik.stg.example.com
  ingress:
    annotations:
      kubernetes.io/ingress.class: traefik
      ingress.kubernetes.io/auth-type: forward
      ingress.kubernetes.io/auth-url: http://traefik-forward-auth.kube-system.svc.cluster.local
      ingress.kubernetes.io/auth-response-headers: X-Forwarded-User
rbac:
  enabled: true
deployment:
  hostPort:
    httpsEnabled: true

traefik-forward-auth (reference to Example for k8s):

##
# Secrets to store Google's client secret and the app's secret
##
kind: Secret
apiVersion: v1
metadata:
  name: traefik-forward-auth-secrets
  namespace: kube-system
  labels:
    name: traefik
type: Opaque
data:
  CLIENT_SECRET: <- CLIENT_SECRET -> 
  SECRET: <- SECRET ->
---

##
# Main deployment
## 
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: traefik-forward-auth
  name: traefik-forward-auth
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik-forward-auth
  template:
    metadata:
      labels:
        app: traefik-forward-auth
    spec:
      containers:
        - name: traefik-forward-auth
          image: thomseddon/traefik-forward-auth
          ports:
            - containerPort: 4181
              protocol: TCP
          env:
            - name: CLIENT_ID
              value: <- CLIENT_ID ->
            - name: CLIENT_SECRET
              valueFrom:
                secretKeyRef:
                  name: traefik-forward-auth-secrets
                  key: CLIENT_SECRET
            - name: SECRET
              valueFrom:
                secretKeyRef:
                  name: traefik-forward-auth-secrets
                  key: SECRET
            - name: DOMAIN
              value: example.com
          livenessProbe:
            tcpSocket:
              port: 4181
            initialDelaySeconds: 20
            failureThreshold: 3
            successThreshold: 1
            periodSeconds: 10
            timeoutSeconds: 2
---

kind: Service
apiVersion: v1
metadata:
  name: traefik-forward-auth
  namespace: kube-system
spec:
  selector:
    app: traefik-forward-auth
  ports:
    - port: 80
      targetPort: 4181
      protocol: TCP
kk17 commented 4 years ago

I found out the clear the browser data can solve this problem.

ghhv commented 4 years ago

@thomseddon, I did my best to gather some information for you. My setup was based on: https://www.smarthomebeginner.com/google-oauth-with-traefik-docker/

Description: Open chrome in incognite mode (or normal). Visit the configured url and login with configured email address.

Expected: Proper redirection after the authentication. Content is visible.

Actual: After login the webpage redirects and returns with 401 Not authorized.

Note: I did changed some personal data and revoked the 0auth cert just after this run.

Logs:

time="2020-01-20T22:20:42Z" level=debug msg="Starting with options: {\"LogLevel\":\"debug\",\"LogFormat\":\"text\",\"AuthHost\":\"oauth.fakedomainforissue31.com\",\"CookieDomains\":[{\"Domain\":\"fakedomainforissue31.com\",\"DomainLen\":16,\"SubDomain\":\".fakedomainforissue31.com\",\"SubDomainLen\":17}],\"InsecureCookie\":true,\"CookieName\":\"_forward_auth\",\"CSRFCookieName\":\"_forward_auth_csrf\",\"DefaultAction\":\"auth\",\"Domains\":null,\"LifetimeString\":0,\"Path\":\"/_oauth\",\"Whitelist\":[\"fakemailforissue31@gmail.com\"],\"Providers\":{\"Google\":{\"ClientId\":\"xxxxxxxxxxxxx.apps.googleusercontent.com\",\"Scope\":\"https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/userinfo.email\",\"Prompt\":\"\",\"LoginURL\":{\"Scheme\":\"https\",\"Opaque\":\"\",\"User\":null,\"Host\":\"accounts.google.com\",\"Path\":\"/o/oauth2/auth\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"TokenURL\":{\"Scheme\":\"https\",\"Opaque\":\"\",\"User\":null,\"Host\":\"www.googleapis.com\",\"Path\":\"/oauth2/v3/token\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"UserURL\":{\"Scheme\":\"https\",\"Opaque\":\"\",\"User\":null,\"Host\":\"www.googleapis.com\",\"Path\":\"/oauth2/v2/userinfo\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"}}},\"Rules\":{},\"Lifetime\":0,\"CookieDomainsLegacy\":null,\"CookieSecureLegacy\":\"\",\"ClientIdLegacy\":\"\",\"PromptLegacy\":\"\"}"
time="2020-01-20T22:20:42Z" level=info msg="Listening on :4181"
time="2020-01-20T22:21:10Z" level=debug msg="Authenticating request" headers="map[Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9] Accept-Encoding:[gzip, deflate, br] Accept-Language:[en-US,en;q=0.9,nl;q=0.8,af;q=0.7] Sec-Fetch-Mode:[navigate] Sec-Fetch-Site:[none] Sec-Fetch-User:[?1] Upgrade-Insecure-Requests:[1] User-Agent:[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36] X-Forwarded-For:[192.168.0.1] X-Forwarded-Host:[debug.fakedomainforissue31.com] X-Forwarded-Method:[GET] X-Forwarded-Proto:[https] X-Forwarded-Uri:[/]]" rule=default source_ip=192.168.0.1
time="2020-01-20T22:21:10Z" level=debug msg="Set CSRF cookie and redirecting to google login" source_ip=192.168.0.1
time="2020-01-20T22:21:10Z" level=debug msg=Done source_ip=192.168.0.1
time="2020-01-20T22:21:13Z" level=debug msg="Handling callback" headers="map[Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9] Accept-Encoding:[gzip, deflate, br] Accept-Language:[en-US,en;q=0.9,nl;q=0.8,af;q=0.7] Referer:[https://accounts.google.com/signin/oauth/oauthchooseaccount?client_id=xxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com&as=siqf3xKp8E6XuK8hiWhYSw&destination=https%3A%2F%2Foauth.fakedomainforissue31.com&approval_state=!ChRjUnBHenZ0aWlkMlJOWjB1UmZBZRIfODFnTTFFUjNYR1VkY0t0V0xtY192ZHBvNWJsUV9CWQ%E2%88%99AJDr988AAAAAXid5VqkCKO41SE29VN3yjfxvWPdXqezJ&oauthgdpr=1&xsrfsig=ChkAeAh8TzAGW13wZT6mNVUkgf1VI3Un__LKEg5hcHByb3ZhbF9zdGF0ZRILZGVzdGluYXRpb24SBXNvYWN1Eg9vYXV0aHJpc2t5c2NvcGU&flowName=GeneralOAuthFlow] Sec-Fetch-Mode:[navigate] Sec-Fetch-Site:[cross-site] Sec-Fetch-User:[?1] Upgrade-Insecure-Requests:[1] User-Agent:[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36] X-Forwarded-For:[192.168.0.1] X-Forwarded-Host:[oauth.fakedomainforissue31.com] X-Forwarded-Method:[GET] X-Forwarded-Proto:[https] X-Forwarded-Uri:[/_oauth?state=f480a65f3c0c40bc1ec395dd5884d862%3Ahttps%3A%2F%2Fdebug.fakedomainforissue31.com%2F&code=4%2FvgFS7t4mz4NGZPiFu_WqGI4IM-rwRbZpSLkhC8lsjakaQ3wZcRe1VJIuXxL_8bzruh7dP5LfKUGJraM3UyPUQc4&scope=email+profile+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.profile+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+openid&authuser=0&prompt=consent&session_state=efcdb271432c17ec7c1ac1b246b908fbc2dd43eb..413a]]" rule=default source_ip=192.168.0.1
time="2020-01-20T22:21:13Z" level=warning msg="Missing csrf cookie" source_ip=192.168.0.1
time="2020-01-20T22:21:13Z" level=debug msg="Authenticating request" headers="map[Accept:[image/webp,image/apng,image/*,*/*;q=0.8] Accept-Encoding:[gzip, deflate, br] Accept-Language:[en-US,en;q=0.9,nl;q=0.8,af;q=0.7] Referer:[https://oauth.fakedomainforissue31.com/_oauth?state=f480a65f3c0c40bc1ec395dd5884d862%3Ahttps%3A%2F%2Fdebug.fakedomainforissue31.com%2F&code=4%2FvgFS7t4mz4NGZPiFu_WqGI4IM-rwRbZpSLkhC8lsjakaQ3wZcRe1VJIuXxL_8bzruh7dP5LfKUGJraM3UyPUQc4&scope=email+profile+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.profile+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+openid&authuser=0&prompt=consent&session_state=efcdb271432c17ec7c1ac1b246b908fbc2dd43eb..413a] Sec-Fetch-Mode:[no-cors] Sec-Fetch-Site:[same-origin] User-Agent:[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36] X-Forwarded-For:[192.168.0.1] X-Forwarded-Host:[oauth.fakedomainforissue31.com] X-Forwarded-Method:[GET] X-Forwarded-Proto:[https] X-Forwarded-Uri:[/favicon.ico]]" rule=default source_ip=192.168.0.1
time="2020-01-20T22:21:13Z" level=debug msg="Set CSRF cookie and redirecting to google login" source_ip=192.168.0.1
time="2020-01-20T22:21:13Z" level=debug msg=Done source_ip=192.168.0.1

docker-compose.yaml

whoami:
    container_name: debug
    hostname: debug
    image: containous/whoami
    labels:
      - traefik.frontend.rule=Host:debug.fakedomainforissue31.com
      - traefik.port=80
      - traefik.backend=whoami
      - traefik-docker.network=internal
      - traefik.frontend.auth.forward.address=http://oauth:4181
      - traefik.frontend.auth.forward.authResponseHeaders=X-Forwarded-User
      - traefik.frontend.auth.forward.trustForwardHeader=true
      - traefik.redirectorservice.frontend.entryPoints=http
      - traefik.redirectorservice.frontend.redirect.entryPoint=https
      - traefik.webservice.frontend.entryPoints=https

oauth:
    image: thomseddon/traefik-forward-auth:latest
    container_name: oauth
    hostname: oauth
    restart: always
    env_file:
      - oauth.env
    labels:
      - traefik.enable=true
      - traefik.port=4181
      - traefik.backend=oauth
      - traefik.docker.network=internal
      - traefik.frontend.rule=Host:oauth.fakedomainforissue31.com
      - traefik.frontend.headers.SSLHost=oauth.fakedomainforissue31.com
      - traefik.frontend.auth.forward.address=http://oauth:4181
      - traefik.frontend.auth.forward.authResponseHeaders=X-Forwarded-User
      - traefik.frontend.auth.forward.trustForwardHeader=true
      - traefik.frontend.headers.SSLRedirect=true
      - traefik.frontend.headers.browserXSSFilter=true
      - traefik.frontend.headers.contentTypeNosniff=true
      - traefik.frontend.headers.forceSTSHeader=true
      - traefik.frontend.headers.STSSeconds=315360000
      - traefik.frontend.headers.STSIncludeSubdomains=true
      - traefik.frontend.headers.STSPreload=true
      - traefik.frontend.headers.frameDeny=true
      - traefik.frontend.passHostHeader=true
      - traefik.frontend.headers.SSLForceHost=true
      - traefik.frontend.headers.customResponseHeaders=X-Robots-Tag:noindex,nofollow,nosnippet,noarchive,notranslate,noimageindex
      - traefik.redirectorservice.frontend.entryPoints=http
      - traefik.redirectorservice.frontend.redirect.entryPoint=https
      - traefik.webservice.frontend.entryPoints=https

oauth.env

PROVIDERS_GOOGLE_CLIENT_ID=XXXXXXXXXXXXXXXXXX.apps.googleusercontent.com
PROVIDERS_GOOGLE_CLIENT_SECRET=XXXXXXXXXXXXXX
SECRET=XXXXXXXXXXXXXXX
COOKIE_DOMAIN=fakedomainforissue31.com
INSECURE_COOKIE=true
AUTH_HOST=oauth.fakedomainforissue31.com
URL_PATH=/_oauth
WHITELIST=fakemailforissue31@gmail.com
LOG_LEVEL=debug
LIFETIME=2592000 # 30 days
OIDC_ISSUER=https://accounts.google.com
CSRF_COOKIE_NAME=_forward_auth_csrf

Note: I also tried INSECURE_COOKIE=false, changing _forward_auth_csrf to a different key and a different email address

How do you want me to collect the request logs? or is this enough to get started?

I did saw this message in the console after landing on the 401 page.

/_oauth?state=014308…49a7a48f6c..576a#:1 A cookie associated with a cross-site resource at https://accounts.google.com/ was set without the `SameSite` attribute. A future release of Chrome will only deliver cookies with cross-site requests if they are set with `SameSite=None` and `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.
/_oauth?state=014308…49a7a48f6c..576a#:1 A cookie associated with a cross-site resource at http://google.com/ was set without the `SameSite` attribute. A future release of Chrome will only deliver cookies with cross-site requests if they are set with `SameSite=None` and `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.
/_oauth?state=014308…49a7a48f6c..576a#:1 A cookie associated with a cross-site resource at https://google.com/ was set without the `SameSite` attribute. A future release of Chrome will only deliver cookies with cross-site requests if they are set with `SameSite=None` and `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.
Cross-Origin Read Blocking (CORB) blocked cross-origin response https://accounts.google.com/o/oauth2/auth?client_id=110929115210-iebt17j9me…b0168a5…%3Ahttps%3A%2F%2Foauth.computerheld.net%2Ffavicon.ico with MIME type text/html. See https://www.chromestatus.com/feature/5629709824032768 for more details.

and I can't seem to find the_forward_auth_csrf cookie after the redirect under Application>Storage>Cookies. Not sure if this is the cause of my problem.

Hey Frank, I also used the SmartHomeBeginner sample. I was also getting 'unauthorized' because the Google email address I was logging in with, was not in the WHITELIST environment variable passed to the forward-auth! Adding this fixed it.. Clear cookies to ensure it's re-authenticating.

This error clue is also apparent from the service log for the container which will say invalid email. time="2020-04-04T12:13:37Z" level=error msg="Invalid email" email=me@sample.com

Hope that helps someone!

frankforpresident commented 4 years ago

Hey Frank, I also used the SmartHomeBeginner sample. I was also getting 'unauthorized' because the Google email address I was logging in with, was not in the WHITELIST environment variable passed to the forward-auth! Adding this fixed it.. Clear cookies to ensure it's re-authenticating.

This error clue is also apparent from the service log for the container which will say invalid email. time="2020-04-04T12:13:37Z" level=error msg="Invalid email" email=me@sample.com

Hope that helps someone!

I will try it again tonight but if i remember correctly it was already set to the correct email address.

frankforpresident commented 4 years ago

My email address is ok and I don't see this message in the logs.

I have a question, could it be related to my ip range? I've noticed in the logs a 192.168.0.1 ip address but that would be the range of my modem while my internal ip range of my router is 10.0.0.X

Anyway. on every attempt I see the same result. missing cookie, setting cookie, done

time="2020-04-09T10:53:09Z" level=warning msg="Missing csrf cookie" source_ip=192.168.0.1 time="2020-04-09T10:53:10Z" level=debug msg="Set CSRF cookie and redirecting to google login" source_ip=192.168.0.1 time="2020-04-09T10:53:10Z" level=debug msg=Done source_ip=192.168.0.1

ghhv commented 4 years ago

Ahh, OK.. that should be an easy fix.. Did you generate and pass a cookie? e.g. environment:

frankforpresident commented 4 years ago

Ahh, OK.. that should be an easy fix.. Did you generate and pass a cookie? e.g. environment:

  • CLIENT_ID=$OAUTH2_PROXY_CLIENT_ID
  • CLIENT_SECRET=$OAUTH2_PROXY_CLIENT_SECRET - SECRET=$OAUTH2_PROXY_COOKIE_SECRET

Yes, generated with: openssl rand -hex 16

You can check my config here, it didn't changed that much. https://github.com/thomseddon/traefik-forward-auth/issues/31#issuecomment-576454102

ghhv commented 4 years ago

OK, comparing yours to mine, I have 2 differences:

And I do not have the "CSRF_COOKIE_NAME=_forward_auth_csrf"

So that could be why you are getting your error.

Note also that SmartHomeBeginner has drastically changed his sample code in the past week..

bekoeppel commented 4 years ago

I got the same errors and was able to reproduce it as follows:

Closing Tab A, so that it would not make requests and issue new _forward_auth_csrf cookies then allowed me to log in to the second application in tab B.

thomseddon commented 4 years ago

@bekoeppel I have created an issue to track that specific pattern as I think there could be a better way of handling it.

I believe a lot of people here will have run into #103 whereby Google will not offer up the account selection by default on subsequent requests, so if you select the wrong account you have to wait for that selection to expire on Google's side before having another go (which is why some people have said you just need to wait and clearing cookies is not enough).

Beyond the two instances above, I think the most common cause will be a misconfiguration. I have opened #114 to improve debug logging to help track these down

@frankforpresident I will confirm when #114 is merged, if you could send your debug logs again following that, it would be greatly appreciated.

frankforpresident commented 4 years ago

@thomseddon, Will do

0xknon commented 4 years ago

I am having the same issue here. I guess my config are more naive.

Error: time="2020-02-27T08:25:03Z" level=warning msg="Missing csrf cookie" source_ip=192.168.9.195

Traefik (installed with helm - stable/traefik): values.yaml:

ssl:
  enabled: true
  enforced: true
  insecureSkipVerify: true
dashboard:
  enabled: true
  domain: traefik.stg.example.com
  ingress:
    annotations:
      kubernetes.io/ingress.class: traefik
      ingress.kubernetes.io/auth-type: forward
      ingress.kubernetes.io/auth-url: http://traefik-forward-auth.kube-system.svc.cluster.local
      ingress.kubernetes.io/auth-response-headers: X-Forwarded-User
rbac:
  enabled: true
deployment:
  hostPort:
    httpsEnabled: true

traefik-forward-auth (reference to Example for k8s):

##
# Secrets to store Google's client secret and the app's secret
##
kind: Secret
apiVersion: v1
metadata:
  name: traefik-forward-auth-secrets
  namespace: kube-system
  labels:
    name: traefik
type: Opaque
data:
  CLIENT_SECRET: <- CLIENT_SECRET -> 
  SECRET: <- SECRET ->
---

##
# Main deployment
## 
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: traefik-forward-auth
  name: traefik-forward-auth
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik-forward-auth
  template:
    metadata:
      labels:
        app: traefik-forward-auth
    spec:
      containers:
        - name: traefik-forward-auth
          image: thomseddon/traefik-forward-auth
          ports:
            - containerPort: 4181
              protocol: TCP
          env:
            - name: CLIENT_ID
              value: <- CLIENT_ID ->
            - name: CLIENT_SECRET
              valueFrom:
                secretKeyRef:
                  name: traefik-forward-auth-secrets
                  key: CLIENT_SECRET
            - name: SECRET
              valueFrom:
                secretKeyRef:
                  name: traefik-forward-auth-secrets
                  key: SECRET
            - name: DOMAIN
              value: example.com
          livenessProbe:
            tcpSocket:
              port: 4181
            initialDelaySeconds: 20
            failureThreshold: 3
            successThreshold: 1
            periodSeconds: 10
            timeoutSeconds: 2
---

kind: Service
apiVersion: v1
metadata:
  name: traefik-forward-auth
  namespace: kube-system
spec:
  selector:
    app: traefik-forward-auth
  ports:
    - port: 80
      targetPort: 4181
      protocol: TCP

Just want to share my Experience on debugging my issue.

My architecture is like this:

Route53 -> CloudFront -> EKS -> traefik -> traefik-forward-auth -> The application

The biggest challenge to me is the Cloudfront since i am not familiar with it. Here are the errors I experienced

First Error - Missing csrf cookie

The reason is that I didn't allow the Cloudfront to Forward Cookies

Second Error - Error validating csrf cookie: Invalid CSRF state value

The reason of getting this is because the Allowed HTTP Methods are GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE. After I change it to just GET, HEAD, The error changed.

Third Error - Invalid email

It seems like the config in the kubeconfig.yaml is incorrect. I am still investigating it.

thomseddon commented 4 years ago

Hello everyone - the good news is that v2.2 was released yesterday, this includes a significant improvement in logging as well as a warning about some invalid configuration states.

This also includes a tweak to the Google provider to ensure the user is always prompted to select which account they would like to use, this fixes the issue where a user selects the wrong account and is then unable to reselect the right account as their previous selection is "cached" (which may result in a Not Authorized error).

I've also released a large amount of new examples for both k8s and swarm, using both traefik v1.7 and 2+ - this shows the advised way to deploy the service and should help many people ending up in a bad/conflicting configuration state.

As mentioned before, I think this issue may represent a number of different issues that people have had, most of which I believe will have been caused by me not providing enough configuration examples - which I hope is now fixed. Due to this, and the other fixes outlined above, I'm going to close this issue.

If anyone is still having a problem when using v2.2, following the new examples, please please please open a new issue with your configuration and logs and I will dig into it right away.

Thank you everyone for your contributions to this, it's very much appreciated.

hollie commented 3 years ago

Hello all,

for reference to other users: I too was experiencing the error:

msg="Error validating csrf cookie" 
error="CSRF cookie does not match state"

with the in the documentation recommended docker v2 image from docker hub.

The error happened after the initial cookie expired (which was 30 days after initial authentication).

I could work around the issue by using a private browser session.

After updating to the latest docker image the issue seems to be resolved.

Best regards, Lieven.