Error fetching openid configuration: 502 Bad Gateway

pheyvaer commented 3 years ago

Hi,

I get an error when running the server behind NGINX. NSS and NGINX are running on two different machines.

I do solid start with the following config:

{
  "root": "/users/pheyvaer/nss/data",
  "port": "8443",
  "serverUri": "https://dev.inclusie-in-4d.ilabt.imec.be",
  "webid": true,
  "mount": "",
  "configPath": "./config",
  "configFile": "./config.json",
  "dbPath": "",
  "sslKey": "privkey.pem",
  "sslCert": "fullchain.pem",
  "multiuser": true,
  "enforceToc": false,
  "disablePasswordChecks": false,
  "supportEmail": "pieter.heyvaert@ugent.be",
  "server": {
    "name": "Inclusie in 4D",
    "description": "",
    "logo": ""
  }
}

I get this error

Error registering a new client:  Error: Error fetching openid configuration: 502 Bad Gateway
    at /users/pheyvaer/.nvm/versions/node/v12.19.0/lib/node_modules/solid-server/node_modules/@solid/oidc-rp/src/onHttpError.js:32:17
    at processTicksAndRejections (internal/process/task_queues.js:97:5) {
  response: Response {
    size: 0,
    timeout: 0,
    [Symbol(Body internals)]: { body: [PassThrough], disturbed: false, error: null },
    [Symbol(Response internals)]: {
      url: 'https://dev.inclusie-in-4d.ilabt.imec.be/.well-known/openid-configuration',
      status: 502,
      statusText: 'Bad Gateway',
      headers: [Headers],
      counter: 0
    }
  },
  statusCode: 502
}

A wildcard certificate is implemented for the *.dev.inclusie-in-4d.ilabt.imec.be. I'm also wondering where the .well-known/openid-configuration is coming from. Is it something that is provided by NSS?

bourgeoa commented 3 years ago

Can you try to start with DEBUG="solid:*"

michielbdejong commented 3 years ago

what do you see when you visit https://dev.inclusie-in-4d.ilabt.imec.be/.well-known/openid-configuration from the box that runs NSS?

I see a 502 when I visit it from the public internet.

The cert is valid though, look: https://dev.inclusie-in-4d.ilabt.imec.be/.well-known/openid-configuration

This looks like an error that's coming from Nginx. Have a look at what Nginx would see when it proxies this request to https://nss-box-local-ip-addr:8443/.well-known/openid-configuration

Yes, this file is provided by NSS, it should look something like https://solidcommunity.net/.well-known/openid-configuration And it's also consumed by NSS (which admittedly is a bit weird, it should not go via the public internet when dereferencing files it hosts itself, but that's due to separation between IDP code and LDP storage code)

pheyvaer commented 3 years ago

Using curl I get when asking for https://nss-box-local-ip-addr:8443/.well-known/openid-configuration

curl: (51) SSL: no alternative certificate subject name matches target host name '192.168.49.1'

Trying with localhost instead of the IP gives the same result.

Edit:

Running solid with the same config on the same machine as NGINX resolves the issue.

michielbdejong commented 3 years ago

Try curl -k to ignore the SSL error. Also make sure to run node-solid-server with the NODE_TLS_REJECT_UNAUTHORIZED=0 to work around your self-signed cert. An easy way to do this is to use ./bin/solid-test start instead of ./bin/solid start.

pheyvaer commented 3 years ago

But it's not a self-signed cert though. We use Let's Encrypt to have a wildcard cert for the whole domain.

vemonet commented 3 years ago

We are facing similar issues with deploying Solid behind a Nginx proxy with Letsencrypt companion (cf. https://github.com/nginx-proxy)

The exact same setup was perfectly working a few days/weeks ago.

Now restarting the docker-compose, I am getting the same issue as @pheyvaer :

solid-server | 2020-11-10T15:04:30.763Z solid:authentication Provider keys loaded from config
solid-server | 2020-11-10T15:04:30.781Z solid:authentication Provider keychain initialized
solid-server | 2020-11-10T15:04:30.782Z solid:authentication Not sleeping before client registration...
solid-server | 2020-11-10T15:04:30.783Z solid:authentication Client not present for issuer https://solid.137.120.31.102.nip.io, initializing new client
solid-server | 2020-11-10T15:04:30.783Z solid:authentication Registering new client for issuer  https://solid.137.120.31.102.nip.io
solid-server | 2020-11-10T15:04:30.817Z solid:authentication Error registering a new client:  { FetchError: request to https://solid.137.120.31.102.nip.io/.well-known/openid-configuration failed, reason: self signed certificate
solid-server |     at ClientRequest.<anonymous> (/usr/local/lib/node_modules/solid-server/node_modules/node-fetch/lib/index.js:1455:11)
solid-server |     at ClientRequest.emit (events.js:198:13)
solid-server |     at TLSSocket.socketErrorListener (_http_client.js:401:9)
solid-server |     at TLSSocket.emit (events.js:198:13)
solid-server |     at emitErrorNT (internal/streams/destroy.js:91:8)
solid-server |     at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)
solid-server |     at process._tickCallback (internal/process/next_tick.js:63:19)
solid-server |   message:
solid-server |    'request to https://solid.137.120.31.102.nip.io/.well-known/openid-configuration failed, reason: self signed certificate',
solid-server |   type: 'system',
solid-server |   errno: 'DEPTH_ZERO_SELF_SIGNED_CERT',
solid-server |   code: 'DEPTH_ZERO_SELF_SIGNED_CERT' }
solid-server | 2020-11-10T15:04:30.819Z solid:authentication Error initializing local RP client:  { FetchError: request to https://solid.137.120.31.102.nip.io/.well-known/openid-configuration failed, reason: self signed certificate
solid-server |     at ClientRequest.<anonymous> (/usr/local/lib/node_modules/solid-server/node_modules/node-fetch/lib/index.js:1455:11)
solid-server |     at ClientRequest.emit (events.js:198:13)
solid-server |     at TLSSocket.socketErrorListener (_http_client.js:401:9)
solid-server |     at TLSSocket.emit (events.js:198:13)
solid-server |     at emitErrorNT (internal/streams/destroy.js:91:8)
solid-server |     at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)
solid-server |     at process._tickCallback (internal/process/next_tick.js:63:19)
solid-server |   message:
solid-server |    'request to https://solid.137.120.31.102.nip.io/.well-known/openid-configuration failed, reason: self signed certificate',
solid-server |   type: 'system',
solid-server |   errno: 'DEPTH_ZERO_SELF_SIGNED_CERT',
solid-server |   code: 'DEPTH_ZERO_SELF_SIGNED_CERT' }

Those are not self-signed certificates. Those are certificates that work for all of others applications run on our server (a few dozen), and who were working a few weeks ago

Here is the docker-compose.yml we use:

version: "3"
services:
  solid:
    image: nodesolidserver/node-solid-server:latest
    container_name: solid-server
    restart: unless-stopped
    ports:
      - 8443:8443
    environment: 
      - "SOLID_SERVER_URI=https://solid.137.120.31.102.nip.io"
      - "SOLID_SSL_KEY=/opt/solid/certs/solid.137.120.31.102.nip.io/key.pem"
      - "SOLID_SSL_CERT=/opt/solid/certs/solid.137.120.31.102.nip.io/fullchain.pem"
      - "SOLID_MULTIUSER=true"
      - "SOLID_SERVER_NAME=Solid server at IDS"
      - "SOLID_SERVER_DESCRIPTION=Solid server hosted at the Institute of Data Science at Maastricht University"
      - "VIRTUAL_HOST=solid.137.120.31.102.nip.io,*.solid.137.120.31.102.nip.io"
      - "LETSENCRYPT_HOST=solid.137.120.31.102.nip.io,vemonet.solid.137.120.31.102.nip.io"
      - "VIRTUAL_PORT=8443"
      - "VIRTUAL_PROTO=https"
    volumes: 
      - /data/certs:/opt/solid/certs
      - /data/solid/data:/opt/solid/data
      - /data/solid/.db:/opt/solid/.db
      - /data/solid/config:/opt/solid/config

But it's not a self-signed cert though. We use Let's Encrypt to have a wildcard cert for the whole domain.

@pheyvaer Last time I checked LetsEncrypt could not handle wildcard (according to discussion in issues, only VIRTUAL_HOST would handle wildcarded subdomains). Did it work for you to provide something like *.solid.137.120.31.102.nip.io for the LETSENCRYPT_HOST ?

vemonet commented 3 years ago

@pheyvaer I just found out the issue: even VIRTUAL_HOST does not seems to handle wildcards subdomains!

So I need to manually define each single subdomain one by one for each new user!

vemonet commented 3 years ago

The mentioned issue: https://github.com/nginx-proxy/docker-letsencrypt-nginx-proxy-companion/issues/319

xpat commented 1 year ago

I'm racking my brains here unsuccessfully trying to get a node-solid-server fully running again.

Latest error message:

nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
root@teampoutineinstance:/var/www/mypod.dev/node-solid-server# systemctl restart nginx
root@teampoutineinstance:/var/www/mypod.dev/node-solid-server# solid start --server-uri https://mypod.dev
  solid:settings Server URI: https://mypod.dev +0ms
  solid:settings Auth method: oidc +8ms
  solid:settings Strict origins: true +3ms
  solid:settings Allowed origins:  +0ms
  solid:settings Db path: ./.db +0ms
  solid:settings Config path: ./config +0ms
  solid:settings Suffix Acl: undefined +4ms
  solid:settings Suffix Meta: undefined +0ms
  solid:settings Allow WebID authentication: true +0ms
  solid:settings Live-updates: true +0ms
  solid:settings Multi-user: true +0ms
  solid:settings Suppress default data browser app: undefined +0ms
  solid:settings Default data browser app file path: default +0ms
  solid:settings Base URL (--mount): / +69ms
  solid:settings SSL Private Key path: /etc/letsencrypt/live/mypod.dev/privkey.pem +1ms
  solid:settings SSL Certificate path: /etc/letsencrypt/live/mypod.dev/fullchain.pem +0ms
Solid server (5.7.4) running on https://localhost:8443/
Press <ctrl>+c to stop
  solid:authentication Provider keys loaded from config +0ms
  solid:authentication Provider keychain initialized +74ms
  solid:authentication Not sleeping before client registration... +1ms
  solid:authentication Client not present for issuer https://mypod.dev, initializing new client +7ms
  solid:authentication Registering new client for issuer  https://mypod.dev +0ms
  solid:authentication Error registering a new client:  Error: Error fetching openid configuration: 502 Bad Gateway
    at /usr/lib/node_modules/solid-server/node_modules/@solid/oidc-rp/src/onHttpError.js:32:17
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  response: Response {
    size: 0,
    timeout: 0,
    [Symbol(Body internals)]: { body: [PassThrough], disturbed: false, error: null },
    [Symbol(Response internals)]: {
      url: 'https://mypod.dev/.well-known/openid-configuration',
      status: 502,
      statusText: 'Bad Gateway',
      headers: [Headers],
      counter: 0
    }
  },
  statusCode: 502
} +135ms
  solid:authentication Error initializing local RP client:  Error: Error fetching openid configuration: 502 Bad Gateway
    at /usr/lib/node_modules/solid-server/node_modules/@solid/oidc-rp/src/onHttpError.js:32:17
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  response: Response {
    size: 0,
    timeout: 0,
    [Symbol(Body internals)]: { body: [PassThrough], disturbed: false, error: null },
    [Symbol(Response internals)]: {
      url: 'https://mypod.dev/.well-known/openid-configuration',
      status: 502,
      statusText: 'Bad Gateway',
      headers: [Headers],
      counter: 0
    }
  },
  statusCode: 502

So many basic questions about how to get this working again - it was about two years ago that I set up several NSS successfully on Digital Ocean. A lot has changed since then, right? I did manage to get a community-solid-server running the other day, but that was just frustrating because I jumped into it without reading that it's not for multi-users. (Perhaps a name change to personal-solid-server?). Q1. Should I run the command solid start using sudo? Q2. How to persist the instance (I tried using PM2, which worked, though nothing is working now -- see latest error messages -- also I got to where I had a 403 forbidden, which I thought was an improvement over 502 Gateway. Now I'm not sure.)

I deleted that CSS server and installed NSS, according to this and this and following my notes from way back in 2020 and using information I gleaned from comparing and contrasting the instructions from NSS and some from CSS (especially the Nginx configs).

I've managed to make mince-meat out of my brain by trying to mix and match information from all the different sources out there (and my notes probably only add to my confusion).

Essential variables:

Host VM
DNS
NGINX
SSL/Letsencrypt
Permissions
Server configuration

Host VM.
$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 22.04.1 LTS Release: 22.04 Codename: jammy

DNS

mypod.dev   A   1 hour  35.193.8.13
mypod.dev   CAA 1 minute    0 issue "letsencrypt.org" 0 issuewild "letsencrypt.org"
mypod.dev   TXT 1 hour  "v=spf1 include:mailgun.org ~all"
*.mypod.dev CNAME   1 hour  mypod.dev.
_acme-challenge.mypod.dev   TXT 1 hour  "V20ksy-F0d-I52Gjm3IZqYsII...."
email.mypod.dev CNAME   1 hour  mailgun.org.
pic._domainkey.mypod.dev    TXT 1 hour  "k=rsa; p=MIGfMA0G..."
www.mypod.dev   CNAME   1 hour  mypod.dev.

I got a new NSS server partially working. For troubleshooting purposes, I haven't changed my DNS settings which include _acme-challenge TXT string pointing to mypod.dev;

NGINX

(I've tried various pure and mixed configurations based on Solid Projects suggestions, such as this one and this one, and yes, I was able to get this up and running a few times but... there was always an error message when it came to logging in. "Couldn't Fetch", though I could register at one point and navigate to https://antoine.mypod.dev/profile/card#me)

upstream node-solid-server{
  server 127.0.0.1:3050;
  keepalive 64;

}

# Redirect HTTP to HTTPS
server {
  server_name *.mypod.dev mypod.dev;
  listen 80;
  return 301 https://$host$request_uri;
}

# Proxy traffic for https://solid.example/ to http://localhost:3000/
server {
  server_name *.mypod.dev mypod.dev;

  # HTTPS certificate setup (can be autogenerated by tools such as certbot)
  listen 443 ssl http2;
  ssl_certificate         /etc/letsencrypt/live/mypod.dev/fullchain.pem;
  ssl_certificate_key     /etc/letsencrypt/live/mypod.dev/privkey.pem;
  ssl_trusted_certificate /etc/letsencrypt/live/mypod.dev/chain.pem;
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

  access_log  /var/log/nginx/mypod.dev.solid_ssl_access.log;
  error_log   /var/log/nginx/mypod.dev.solid_ssl_error.log;

  # Proxy all other traffic to the Solid server
  location / {
    # Delegate to the Solid server, passing the original host and protocol
    proxy_pass http://node-solid-server$request_uri;
    proxy_set_header X-Forwarded-Host  $host;
    proxy_set_header X-Forwarded-Proto $scheme;

    # Pass these headers from the Solid server back to the client
    proxy_pass_header Server;
    proxy_pass_header Set-Cookie;

    # Enable Websocket support
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";

    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";

    # Prevent ETag modification (https://github.com/solid/community-server/issues/1036)
    gzip off;
  }
}

SSL/Letsencrypt

If I run $solid start I inevitably get an error message that the Hostname Localhost was not in the certificate, or that it can't find the certificate (depending on if I used sudo or not).

PERMISSIONS

I installed via git clone and ran: npm install -g solid server per the instructions (I didn't do this in root the first time, but re-read the instructions and did it later) changing to root by running sudo su - `

I changed the ownership of directories and files according to the instructions:

chown solid config directories and config-file:
$ cd /var/www/your.host.example.org/
$ chown solid:www-data config.json
$ chown solid:www-data -R config/ data/ .db/
start the service: $ systemctl start solid.service

I ran: setfacl -R -m u:solid:rX /etc/letsencrypt

SERVER CONFIGURATION -- what's causing my errors, NSS installation, HOST, DNS, NGINX..... ahhh!!!?

bourgeoa commented 1 year ago

@xpat Hope this can be of help.

NSS
- solidcommunity.net is running on a Digital Ocean. Details on the installation can be found here https://github.com/solid/solidcommunity.net/wiki
- There are 2 running versions : the production one solidcommunity.net and a test one solidcommunity.net:8443
- They both use letsencrypt's certificates and pm2
CSS is a multiuser solid server that can be run as a mono-user.
- It is running at https://solidweb.me with a mashlib recipe
- for mashlib recipe see https://github.com/CommunitySolidServer/Recipes
- You can also look at https://github.com/SolidOS/css-mashlib to run a localhost
Nginx : personally I do not use Nginx to redirect and certs. Both NSS and CSS can be run using directly the certs and a https://ownDomain

nodeSolidServer / node-solid-server

Error fetching openid configuration: 502 Bad Gateway #1504