search

hackmdio / codimd

CodiMD - Realtime collaborative markdown notes on all platforms.
https://hackmd.io/c/codimd-documentation
GNU Affero General Public License v3.0
9.32k stars 1.06k forks source link

GitLab Oauth Internal Server Error #1134

Open mcnesium opened 5 years ago

mcnesium commented 5 years ago

My GitLab authentication stopped working. CodiMD is running in Docker, here is the docker-compose.yml:

version: '3'
services:
    database:
        image: postgres:9.6-alpine
        container_name: hackmd_database
        environment:
        - POSTGRES_USER=hackmd
        - POSTGRES_PASSWORD=*****
        - POSTGRES_DB=hackmd
        volumes:
        - /data/hackmd/database:/var/lib/postgresql/data
        restart: always
    app:
        image: hackmdio/hackmd:alpine
        container_name: hackmd_app
        volumes:
        - /data/hackmd/uploads:/hackmd/public/uploads
        environment:
        - CMD_DB_URL=postgres://hackmd:*****@database:5432/hackmd
        - CMD_USECDN=false
        - CMD_DOMAIN=hackmd.mydomain.org
        - CMD_URL_ADDPORT=false
        - CMD_PROTOCOL_USESSL=true
        - CMD_ALLOW_FREEURL=true
        - CMD_EMAIL=false
        - CMD_ALLOW_ANONYMOUS=false
        - CMD_ALLOW_EMAIL_REGISTER=false
        - CMD_GITLAB_BASEURL=https://gitlab.mydomain.org/
        - CMD_GITLAB_CLIENTID=*****…
        - CMD_GITLAB_CLIENTSECRET=*****…
        - CMD_IMAGE_UPLOAD_TYPE=filesystem
        - CMD_ALLOW_GRAVATAR=false
        ports:
        - "127.0.0.1:44311:3000"
        restart: always
        depends_on:
        - database

The proxying is done by apache2 on the host:

  …
  RewriteEngine On
  RewriteCond %{HTTP:Connection} Upgrade [NC]
  RewriteCond %{HTTP:Upgrade} websocket [NC]
  RewriteRule /(.*) ws://localhost:44311/$1 [P,L]
  ProxyPass           /  http://localhost:44311/
  ProxyPassReverse    /  http://localhost:44311/
  …

In the running hackmd_app container the environment variables are all set:

$ docker-compose exec app sh
/codimd # printenv
CMD_ALLOW_EMAIL_REGISTER=false
CMD_ALLOW_GRAVATAR=false
CMD_ALLOW_PDF_EXPORT=false
CMD_GITLAB_CLIENTID=*****…
CMD_GITLAB_CLIENTSECRET=*****…
NODE_VERSION=8.15.0
HOSTNAME=2548fff412f3
YARN_VERSION=1.12.3
CMD_PROTOCOL_USESSL=true
SHLVL=1
HOME=/root
CMD_IMAGE_UPLOAD_TYPE=filesystem
CMD_ALLOW_FREEURL=true
CMD_GITLAB_BASEURL=https://gitlab.mydomain.org/
TERM=xterm
CMD_DB_URL=postgres://hackmd:*****@database:5432/hackmd
CMD_URL_ADDPORT=false
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
CMD_EMAIL=false
CMD_DOMAIN=hackmd.mydomain.org
DOCKERIZE_VERSION=v0.6.1
GOSU_VERSION=1.11
PWD=/codimd
CMD_USECDN=false
CMD_ALLOW_ANONYMOUS=false
NODE_ENV=production

The application is correctly registered in GitLab:

appcodimd

When visiting the site and hitting Login button, the modal with the GitLab button appears, and when this one is hit, the browser tab keeps waiting for an answer from the hackmd domain for about two minutes and then shows Internal Server Error on a blank page.

Here is the output of docker-compose logs -f starting with the boot process until the failed GitLab authentication:

Creating network "codimd_default" with the default driver
Creating hackmd_database ... done
Creating hackmd_app      ... done
Attaching to hackmd_app, hackmd_database
hackmd_app  | 2019/01/31 14:10:48 Waiting for: tcp://database:5432
hackmd_app  | 2019/01/31 14:10:48 Connected to tcp://database:5432
hackmd_app  |
hackmd_app  | Sequelize [Node: 8.15.0, CLI: 2.8.0, ORM: 3.30.4]
hackmd_app  |
hackmd_database | LOG:  database system was interrupted; last known up at 2019-01-31 14:07:38 UTC
hackmd_database | LOG:  database system was not properly shut down; automatic recovery in progress
hackmd_database | LOG:  redo starts at 0/912E4C8
hackmd_database | LOG:  invalid record length at 0/9132170: wanted 24, got 0
hackmd_database | LOG:  redo done at 0/9132148
hackmd_database | LOG:  last completed transaction was at log time 2019-01-31 14:10:13.807358+00
hackmd_database | LOG:  MultiXact member wraparound protections are now enabled
hackmd_database | LOG:  database system is ready to accept connections
hackmd_database | LOG:  autovacuum launcher started
hackmd_database | LOG:  incomplete startup packet
hackmd_app  | Parsed url postgres://hackmd:*****@database:5432/hackmd
hackmd_app  | (node:25) DeprecationWarning: Using the automatically created return value from client.query as an event emitter is deprecated and will be removed in pg@7.0. Please see the upgrade guide at https://node-postgres.com/guides/upgrading
hackmd_app  | == 20180525153000-user-add-delete-token: migrating =======
hackmd_app  | == 20180525153000-user-add-delete-token: migrated (0.028s)
hackmd_app  |
hackmd_app  |         #################################################################
hackmd_app  |         ###                                                           ###
hackmd_app  |         ###                        !!!WARNING!!!                      ###
hackmd_app  |         ###                                                           ###
hackmd_app  |         ###        Using local uploads without persistence is         ###
hackmd_app  |         ###            dangerous. You'll loose your data on           ###
hackmd_app  |         ###              container removal. Check out:                ###
hackmd_app  |         ###  https://docs.docker.com/engine/tutorials/dockervolumes/  ###
hackmd_app  |         ###                                                           ###
hackmd_app  |         ###                       !!!WARNING!!!                       ###
hackmd_app  |         ###                                                           ###
hackmd_app  |         #################################################################
hackmd_app  |
hackmd_app  | 2019-01-31T14:10:53.558Z - warn: Session secret not set. Using random generated one. Please set `sessionSecret` in your config.js file. All users will be logged out.
hackmd_app  | (node:1) DeprecationWarning: Using the automatically created return value from client.query as an event emitter is deprecated and will be removed in pg@7.0. Please see the upgrade guide at https://node-postgres.com/guides/upgrading
hackmd_app  | >> WARNING: PostgreSQL does not support TEXT with options. Plain `TEXT` will be used instead.
hackmd_app  | >> Check: http://www.postgresql.org/docs/9.4/static/datatype.html
hackmd_app  | 2019-01-31T14:10:54.647Z - info: HTTP Server listening at 0.0.0.0:3000
hackmd_app  | 2019-01-31T14:11:09.542Z - info: 192.168.224.1 - - [31/Jan/2019:14:11:09 +0000] "GET / HTTP/1.1" 304 - "-" "Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0"
hackmd_app  | 2019-01-31T14:11:09.685Z - info: 192.168.224.1 - - [31/Jan/2019:14:11:09 +0000] "GET /config HTTP/1.1" 200 235 "https://hackmd.mydomain.org/" "Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0"
hackmd_app  | 2019-01-31T14:11:09.934Z - info: 192.168.224.1 - - [31/Jan/2019:14:11:09 +0000] "GET /me HTTP/1.1" 304 - "https://hackmd.mydomain.org/" "Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0"
hackmd_app  | 2019-01-31T14:11:10.389Z - info: 192.168.224.1 - - [31/Jan/2019:14:11:10 +0000] "GET /build/bootstrap.min.css.map HTTP/1.1" 302 57 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0"
hackmd_app  | 2019-01-31T14:11:10.394Z - info: 192.168.224.1 - - [31/Jan/2019:14:11:10 +0000] "GET /build HTTP/1.1" 301 169 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0"
hackmd_app  | [repeats about 20 times]
hackmd_app  | 2019-01-31T14:11:10.755Z - info: 192.168.224.1 - - [31/Jan/2019:14:11:10 +0000] "GET /build/ HTTP/1.1" 301 69 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0"
hackmd_app  | 2019-01-31T14:11:12.326Z - info: 192.168.224.1 - - [31/Jan/2019:14:11:12 +0000] "GET /auth/gitlab HTTP/1.1" 302 0 "https://hackmd.mydomain.org/" "Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0"
hackmd_app  | 2019-01-31T14:13:12.513Z - info: 192.168.224.1 - - [31/Jan/2019:14:13:12 +0000] "GET /auth/gitlab/callback?code=127717b5898fbd0a07ea5d56f20d04… HTTP/1.1" - - "-" "Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0"
hackmd_app  | TokenError: The provided authorization grant is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client.
hackmd_app  |     at Strategy.OAuth2Strategy.parseErrorResponse (/codimd/node_modules/passport-oauth2/lib/strategy.js:329:12)
hackmd_app  |     at Strategy.OAuth2Strategy._createOAuthError (/codimd/node_modules/passport-oauth2/lib/strategy.js:376:16)
hackmd_app  |     at /codimd/node_modules/passport-oauth2/lib/strategy.js:166:45
hackmd_app  |     at /codimd/node_modules/oauth/lib/oauth2.js:191:18
hackmd_app  |     at passBackControl (/codimd/node_modules/oauth/lib/oauth2.js:132:9)
hackmd_app  |     at IncomingMessage.<anonymous> (/codimd/node_modules/oauth/lib/oauth2.js:157:7)
hackmd_app  |     at emitNone (events.js:111:20)
hackmd_app  |     at IncomingMessage.emit (events.js:208:7)
hackmd_app  |     at endReadableNT (_stream_readable.js:1064:12)
hackmd_app  |     at _combinedTickCallback (internal/process/next_tick.js:139:11)
hackmd_app  |     at process._tickCallback (internal/process/next_tick.js:181:9)
hackmd_app  | 2019-01-31T14:13:12.668Z - info: 192.168.224.1 - - [31/Jan/2019:14:13:12 +0000] "GET /auth/gitlab/callback?code=127717b5898fbd0a07ea5d56f20d04… HTTP/1.1" 500 148 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0"

The TokenError: The provided authorization grant is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client. is not true, as they have been refreshed right before that session.

The CodiMD setup has been running for quite some time, you can see this has been running under the name HackMD first. The only thing I can think of what I did was upgrading the underlying host system Ubuntu from 1604 to 1804.

There was a database issue that came up in the logs stating

PANIC:  could not locate a valid checkpoint record

which I solved by doing pg_resetxlog on the database:

docker run -ti --user postgres  -v /data/hackmd/database:/var/lib/postgresql/data postgres:9.6-alpine pg_resetxlog /var/lib/postgresql/data/

While I was puzzling about what and why it has happened to the db, I cheeringly saw that the site going back up, but then the stated GitLab authentication fails.

So whats going on with this? Any help?

SISheogorath commented 5 years ago

:warning: First of all, please check that you changed the volume path in your compose file. We now use /codimd/public/uploads. :warning:

Then please double-check the gitlab client secret and client id. Also important question: Do you use GitLab.com or do you use a local gitlab? In case of the latter you are missing CMD_GITLAB_BASEURL.

mcnesium commented 5 years ago

So I tripple-checked the client secret and ID and they do match.

Yes, this is a self-hosted GitLab. But please look again, I am not missing CMD_GITLAB_BASEURL, as it appears in both env var listings above.

Thank you for pointing out the deprecated upload path, though :+1:

ccoenen commented 5 years ago

Can you contact the gitlab server from within the CodiMD container? (a simple curl or wget or even ping would suffice)

mcnesium commented 5 years ago

Yes I can. wget [domain]/api/v4/projects stores a json string of all the public projects in a file called projects.