search

TheThingsNetwork / lorawan-stack

The Things Stack, an Open Source LoRaWAN Network Server
https://www.thethingsindustries.com/stack/
Apache License 2.0
975 stars 306 forks source link

Document the troubleshooting guide for 'Token exchange refused' and other common OAuth problems #3273

Closed TheIronNinja closed 3 years ago

TheIronNinja commented 4 years ago

Summary

Missing Authentication reference in the Troubleshooting section of the documentation. https://thethingsstack.io/getting-started/troubleshooting/

...

Why do we need this ?

The text is there but it links to the same page so it's supposed to exist already.

...

What is already there? What do you see now?

Same as above

...

What is missing? What do you want to see?

I've been having trouble with logging in to the Stack console with the message "Forbidden: Token exchange refused". I've tried everything explained in #2521, #2838, #1752, and this has also been talked about in #2353 and #2714. Since this has been a returning problem, a nice troubleshooting guide would be very useful so there would be no need to read a bunch of threads trying to fix it.

...

How do you propose to document this?

Really just a thorough summary of everything that could fix it in an understandable and easy to follow way.

...

Can you do this yourself and submit a Pull Request?

Not really since this should go in the https://thethingsstack.io/getting-started/troubleshooting page

...

TheIronNinja commented 4 years ago

If anyone stumbles upon this issue looking for answers, go here https://www.thethingsnetwork.org/forum/t/thethingsstack-v3-8-7-with-self-signed-certificate-forbidden-token-exchange-refused/38902/15

johanstokking commented 4 years ago

cc @nejraselimovic

nejraselimovic commented 4 years ago

cc @nejraselimovic

Thanks for the insight @johanstokking, and good job @TheIronNinja!

I can confirm that on Ubuntu 18.04 I got rid of Token exchange refused error that I had with setup that uses localhost as a hostname and self-signed certificates by doing the following:

  1. adding TTN_LW_OAUTH_SERVER_ADDRESS: https://localhost:8885/oauth to environment in docker-compose.yml
  2. changing all https://thethings.example.com/ occurrences with https://localhost:8885/ in default ttn-lw-stack-docker.yml file
  3. using --redirect-uri "https://localhost:8885/console/oauth/callback" and --logout-redirect-uri "https://localhost:8885/console" as @TheIronNinja described in TTN forum thread

Having this setup, I only needed to import ca.pem I created according to this section under Authorities in my browser to resolve all security warnings.

johanstokking commented 4 years ago

OK. Do we need to change or clarify the documentation on these points?

nejraselimovic commented 4 years ago

OK. Do we need to change or clarify the documentation on these points?

Maybe just a note in troubleshooting section in this case when localhost and self-signed certs are being used?

TheIronNinja commented 4 years ago

I'd add a note saying that if you are not installing the stack on your local machine you need to change those things. In my case I'm not using self-signed certificates so this always applies.

nejraselimovic commented 4 years ago

I'd add a note saying that if you are not installing the stack on your local machine you need to change those things. In my case I'm not using self-signed certificates so this always applies.

Oh, I see. But also, I had this problem on my local machine and solved it as I mentioned above. So what we first need here is a good explanation of why this was necessary to make things work in both of our cases, then we can document it properly.

htdvisser commented 4 years ago

With #3352 the error information for these errors will be returned in window.__ttn_config__.PAGE_DATA.error and could be shown to the user. @kschiffer already confirmed that this is possible, but he's concerned that this would be "disclosing development information to the user".

benolayinka commented 4 years ago

I'd add a note saying that if you are not installing the stack on your local machine you need to change those things. In my case I'm not using self-signed certificates so this always applies.

@TheIronNinja if you're not installing the stack on your local machine, you shouldn't need to add these environment variables to your docker-compose.yml. The configuration options you posted are in our example ttn-lw-config-docker.yml, and if you follow along in the installation instructions they should be set to your server's domain, not localhost. If you configured them as localhost on a remote server, that wouldn't work. It also shouldn't be necessary to specify the port in those parameters, since all TLS requests are redirected in docker-compose.yml. 8885 is also passed through, so it will work the way you have it, but it isn't necessary

ports:
      # If deploying on a public server:
      - "80:1885"
      - "443:8885"
      - "1881:1881"
      - "8881:8881"
      - "1882:1882"
      - "8882:8882"
      - "1883:1883"
      - "8883:8883"
      - "1884:1884"
      - "8884:8884"
      - "1885:1885"
      - "8885:8885"
      - "1887:1887"
      - "8887:8887"
      - "1700:1700/udp"

TTN_LW_OAUTH_SERVER_ADDRESS is not a config parameter.

TTN_LW_CONSOLE_UI_AS_BASE_URL: https://mydomain:8885/api/v3
TTN_LW_CONSOLE_UI_IS_BASE_URL: https://mydomain:8885/api/v3
TTN_LW_OAUTH_SERVER_ADDRESS: https://mydomain:8885/oauth # this one is not a real config
TTN_LW_IS_OAUTH_UI_CANONICAL_URL: https://mydomain:8885/oauth
TTN_LW_IS_OAUTH_UI_IS_BASE_URL: https://mydomain:8885/api/v3
TTN_LW_APPLICATION_SERVER_GRPC_ADDRESS: mydomain:8884
TTN_LW_DEVICE_CLAIMING_SERVER_GRPC_ADDRESS: mydomain:8884

@nejraselimovic has a slightly different problem - she's running docker on localhost, and token exchange is failing, i think because if you don't specify the ports, requests WITHIN docker to https://localhost/oauth become localhost/oauth:443 by default, which is not where the stack is listening.

For @nejraselimovic , changing e.g localhost/oauth to localhost/oauth:8885 fixed internal redirects inside Docker.

TheIronNinja commented 4 years ago

Hi @benolayinka, thanks for the detailed info. I guess I saw the TTN_LW_OAUTH_SERVER_ADDRESS in some other post and assumed it was correct, but if it doesn't exist then I'll delete it from my config. As for some of the other parameters I know some of them are defined in the ttn-lw-config-docker.yml, but for some reason they are ignored and I only see the changes reflected if I put them in the docker-compose.yml. In fact, I ended up introducing most of the config parameters there just to make sure they have the value I want them to have.

benolayinka commented 4 years ago

Hi @benolayinka, thanks for the detailed info. I guess I saw the TTN_LW_OAUTH_SERVER_ADDRESS in some other post and assumed it was correct, but if it doesn't exist then I'll delete it from my config. As for some of the other parameters I know some of them are defined in the ttn-lw-config-docker.yml, but for some reason they are ignored and I only see the changes reflected if I put them in the docker-compose.yml. In fact, I ended up introducing most of the config parameters there just to make sure they have the value I want them to have.

If the parameters in ttn-lw-config-docker.yml are ignored we need to figure out why! Can you share your docker-compose.yml and ttn-lw-config-docker.yml so that we can try to reproduce the issue?

TheIronNinja commented 4 years ago

Sure. This are the contents of the docker-compose.yml:

version: '3.7'
services:

  # If using CockroachDB:
  cockroach:
    image: cockroachdb/cockroach:v20.1.5
    command: start --http-port 26256 --insecure
    restart: unless-stopped
    volumes:
      - .env/data/cockroach:/cockroach/cockroach-data
    ports:
      - 127.0.0.1:26257:26257 # Cockroach
      - 127.0.0.1:26256:26256 # WebUI

  redis:
    image: redis:6.0.8
    command: redis-server --appendonly yes
    restart: unless-stopped
    volumes:
      - .env/data/redis:/data
    ports:
      - 127.0.0.1:6379:6379

  stack:
    image: thethingsnetwork/lorawan-stack:3.9.4
    entrypoint: ttn-lw-stack
    command: start
    restart: unless-stopped
    depends_on:
      - redis
      - cockroach
    volumes:
      - ./blob:/srv/ttn-lorawan/public/blob
      - ./config/stack:/stack:ro
      # If using Let's Encrypt:
      - ./acme:/var/lib/acme
    environment:
      TTN_LW_BLOB_LOCAL_DIRECTORY: /srv/ttn-lorawan/public/blob
      TTN_LW_REDIS_ADDRESS: redis:6379
      TTN_LW_IS_DATABASE_URI: postgres://root@cockroach:26257/ttn_lorawan?sslmode=disable
#      TTN_LW_LOG_LEVEL: DEBUG
      TTN_LW_TLS_SOURCE: acme
      TTN_LW_TLS_ACME_DIR: /var/lib/acme
      TTN_LW_TLS_ACME_EMAIL: REDACTED
      TTN_LW_TLS_ACME_HOSTS: MY_DOMAIN
      TTN_LW_TLS_ACME_DEFAULT_HOST: MY_DOMAIN
      TTN_LW_CONSOLE_OAUTH_AUTHORIZE_URL: https://MY_DOMAIN:8885/oauth/authorize
      TTN_LW_CONSOLE_OAUTH_TOKEN_URL: https://MY_DOMAIN:8885/oauth/token
      TTN_LW_CONSOLE_UI_CANONICAL_URL: https://MY_DOMAIN:8885/console
      TTN_LW_CONSOLE_UI_AS_BASE_URL: https://MY_DOMAIN:8885/api/v3
      TTN_LW_CONSOLE_UI_IS_BASE_URL: https://MY_DOMAIN:8885/api/v3
      TTN_LW_OAUTH_SERVER_ADDRESS: https://MY_DOMAIN:8885/oauth
      TTN_LW_IS_OAUTH_UI_CANONICAL_URL: https://MY_DOMAIN:8885/oauth
      TTN_LW_IS_OAUTH_UI_IS_BASE_URL: https://MY_DOMAIN:8885/api/v3

      TTN_LW_APPLICATION_SERVER_GRPC_ADDRESS: MY_DOMAIN:8884
      TTN_LW_DEVICE_CLAIMING_SERVER_GRPC_ADDRESS: MY_DOMAIN:8884
      TTN_LW_DEVICE_TEMPLATE_CONVERTER_GRPC_ADDRESS: MY_DOMAIN:8884
      TTN_LW_DEVICE_TEMPLATE_CONVERTER_SERVER_GRPC_ADDRESS: MY_DOMAIN:8884
      TTN_LW_GATEWAY_SERVER_GRPC_ADDRESS: MY_DOMAIN:8884
      TTN_LW_IDENTITY_SERVER_GRPC_ADDRESS: MY_DOMAIN:8884
      TTN_LW_JOIN_SERVER_GRPC_ADDRESS: MY_DOMAIN:8884
      TTN_LW_NETWORK_SERVER_GRPC_ADDRESS: MY_DOMAIN:8884
      TTN_LW_QR_CODE_GENERATOR_GRPC_ADDRESS: MY_DOMAIN:8884

      TTN_LW_AS_MQTT_PUBLIC_ADDRESS: "MY_DOMAIN:1883"
      TTN_LW_AS_MQTT_PUBLIC_TLS_ADDRESS: "MY_DOMAIN:8883"
      TTN_LW_AS_WEBHOOKS_DOWNLINK_PUBLIC_ADDRESS: "https://MY_DOMAIN:8885/api/v3"
      TTN_LW_CONSOLE_OAUTH_LOGOUT_URL: "https://MY_DOMAIN:8885/oauth/logout"
      TTN_LW_CONSOLE_UI_EDTC_BASE_URL: "https://MY_DOMAIN:8885/api/v3"
      TTN_LW_CONSOLE_UI_GS_BASE_URL: "https://MY_DOMAIN:8885/api/v3"
      TTN_LW_CONSOLE_UI_JS_BASE_URL: "https://MY_DOMAIN:8885/api/v3"
      TTN_LW_CONSOLE_UI_NS_BASE_URL: "https://MY_DOMAIN:8885/api/v3"
      TTN_LW_CONSOLE_UI_QRG_BASE_URL: "https://MY_DOMAIN:8885/api/v3"
      TTN_LW_GCS_BASIC_STATION_DEFAULT_LNS_URI: "wss://MY_DOMAIN:8887"
      TTN_LW_GCS_THE_THINGS_GATEWAY_DEFAULT_MQTT_SERVER: "mqtts://MY_DOMAIN:8881"
      TTN_LW_GS_MQTT_V2_PUBLIC_ADDRESS: "MY_DOMAIN:1881"
      TTN_LW_GS_MQTT_V2_PUBLIC_TLS_ADDRESS: "MY_DOMAIN:8881"
      TTN_LW_GS_MQTT_PUBLIC_ADDRESS: "MY_DOMAIN:1882"
      TTN_LW_GS_MQTT_PUBLIC_TLS_ADDRESS: "MY_DOMAIN:8882"
      TTN_LW_IS_EMAIL_NETWORK_CONSOLE_URL: "https://MY_DOMAIN:8885/console"
      TTN_LW_IS_EMAIL_NETWORK_IDENTITY_SERVER_URL: "https://MY_DOMAIN:8885/oauth"

    ports:
      # If deploying on a public server:
      - 80:1885
      - 443:8885
      - 1881:1881
      - 8881:8881
      - 1882:1882
      - 8882:8882
      - 1883:1883
      - 8883:8883
      - 1884:1884
      - 8884:8884
      - 1885:1885
      - 8885:8885
      - 1887:1887
      - 8887:8887
      - 1700:1700/udp

    # If using custom certificates:
#    secrets:
#      - ca.pem
#      - cert.pem
#      - key.pem

# If using custom certificates:
#secrets:
#  ca.pem:
#    file: ./ca.pem
#  cert.pem:
#    file: ./cert.pem
#  key.pem:
#    file: ./key.pem

And here the contents of the ttn-lw-stack-docker.yml:

# Identity Server configuration
# Email configuration for MY_DOMAIN
is:
  email:
    sender-name: 'Test name'
    sender-address: 'noreply@MY_DOMAIN'
    network:
      name: 'Test name'
      console-url: 'https://MY_DOMAIN/console'
      identity-server-url: 'https://MY_DOMAIN/oauth'

  # Web UI configuration for MY_DOMAIN:
  oauth:
    ui:
      canonical-url: 'https://MY_DOMAIN/oauth'
      is:
        base-url: 'https://MY_DOMAIN/api/v3'

# HTTP server configuration
http:
  cookie:
    block-key: 'REDACTED'                # generate 32 bytes (openssl rand -hex 32)
    hash-key: 'REDACTED'                 # generate 64 bytes (openssl rand -hex 64)
  metrics:
    password: 'REDACTED'               # choose a password
  pprof:
    password: 'REDACTED'                 # choose a password
  redirect-to-tls: 'true'

# Let's encrypt for MY_DOMAIN
#tls:
#  source: 'acme'
#  acme:
#    dir: '/var/lib/acme'
#    email: 'REDACTED'
#    hosts: ['MY_DOMAIN']
#    default-host: 'MY_DOMAIN'

# If Gateway Server enabled, defaults for MY_DOMAIN:
gs:
  mqtt:
    public-address: 'MY_DOMAIN:1882'
    public-tls-address: 'MY_DOMAIN:8882'
  mqtt-v2:
    public-address: 'MY_DOMAIN:1881'
    public-tls-address: 'MY_DOMAIN:8881'

# If Gateway Configuration Server enabled, defaults for MY_DOMAIN:
gcs:
  basic-station:
    default:
      lns-uri: 'wss://MY_DOMAIN:8887'
  the-things-gateway:
    default:
      mqtt-server: 'mqtts://MY_DOMAIN:8881'

# Web UI configuration for HTTPS
console:
  ui:
    canonical-url: 'https://MY_DOMAIN/console'
    is:
      base-url: 'https://MY_DOMAIN/api/v3'
    gs:
      base-url: 'https://MY_DOMAIN/api/v3'
    ns:
      base-url: 'https://MY_DOMAIN/api/v3'
    as:
      base-url: 'https://MY_DOMAIN/api/v3'
    js:
      base-url: 'https://MY_DOMAIN/api/v3'
    qrg:
      base-url: 'https://MY_DOMAIN/api/v3'
    edtc:
      base-url: 'https://MY_DOMAIN/api/v3'

  oauth:
    authorize-url: 'https://MY_DOMAIN/oauth/authorize'
    token-url: 'https://MY_DOMAIN/oauth/token'
    logout-url: 'https://MY_DOMAIN/oauth/logout'
    client-id: 'console'
    client-secret: 'console'          # choose or generate a secret

# If Application Server enabled, defaults for MY_DOMAIN:
as:
  mqtt:
    public-address: 'http://MY_DOMAIN:1883'
    public-tls-address: 'https://MY_DOMAIN:8883'
  webhooks:
    downlink:
      public-address: 'MY_DOMAIN:1885/api/v3'
benolayinka commented 4 years ago

Sure. This are the contents of the docker-compose.yml:

version: '3.7'
services:

  stack:
    image: thethingsnetwork/lorawan-stack:3.9.4
    entrypoint: ttn-lw-stack
    command: start
    restart: unless-stopped
    depends_on:
      - redis
      - cockroach
    volumes:
      - ./blob:/srv/ttn-lorawan/public/blob
      - ./config/stack:/stack:ro
      # If using Let's Encrypt:
      - ./acme:/var/lib/acme

ah, you're missing

services:
  stack:
    entrypoint: ttn-lw-stack -c /config/ttn-lw-config-docker.yml

See here for an example file, and here for an explanation

TheIronNinja commented 4 years ago

If I add that option I get the following log when trying to start the dockers and the service doesn't start:

stack_1 | open /config/stack/ttn-lw-stack-docker.yml: no such file or directory

I've changed the path and the filename to match the ones that I have, which are the ones explained in the Getting Started guide that you linked. Using /config/ttn-lw-config-docker.yml has the same result though, as well as config/stack/... and ./config/stack/...

TheIronNinja commented 4 years ago

TTN_LW_OAUTH_SERVER_ADDRESS is not a config parameter.

TTN_LW_OAUTH_SERVER_ADDRESS: https://mydomain:8885/oauth # this one is not a real config

I was testing right now and this seems to not be true by the way. This parameter is, at the very least, used when I try to login to the server via the CLI. If I comment the parameter and do a login it tries to send me to localhost which causes an error since my computer is not running a server, but with the parameter active it sends me to my domain and everything works as explained above.

johanstokking commented 3 years ago

What is the status here?

@benolayinka is this being addressed with self-signed certs on localhost?

@kschiffer can we show at least the full error for users to see what's going on? I.e. wrong issuer, expired cert, wrong host name?

kschiffer commented 3 years ago

@kschiffer can we show at least the full error for users to see what's going on? I.e. wrong issuer, expired cert, wrong host name?

Iirc the exact error gets swallowed by the OAuth package, so that has to be addressed first. I'll take another look to see how we can improve the communication there.

htdvisser commented 3 years ago

@kschiffer that was already addressed by https://github.com/TheThingsNetwork/lorawan-stack/pull/3352

johanstokking commented 3 years ago

OK, so what is the action item here? @benolayinka @nejraselimovic what is the user experience now?

kschiffer commented 3 years ago

Even with #3352, the errors are still mostly unhelpful. E.g. I tried authorizing the console using an incorrect client secret and the error I got was client is not authorized to request a token using this method, which is misleading to say the least. The same error is thrown if the client has not set redirect URIs. 

I think these unspecific error messages are a part of the security concept of the specification. See also openshift/osin#169

So I think our best bet here is to improve troubleshooting guides in the doc.

johanstokking commented 3 years ago

So I think our best bet here is to improve troubleshooting guides in the doc.

OK!

benolayinka commented 3 years ago

This issue and some others are addressed in https://thethingsstack.io/getting-started/installation/troubleshooting/. Please create issues for specific unaddressed errors