"Server error. Please retry later."

[sfstube]

What happened?

The website shows the PeerTube front-end, but an Error "Server error. Please retry later." shows up.

What do you expect to happen instead?

I expected to see recently added videos.

Steps to reproduce:

This is happening out of nowhere.

Additional information

• PeerTube version: v1.1.0-alpha.1

• URL: https://freespeechtube.org/

• Browser name/version: Firefox 63.0 (64-bit)

• NodeJS version: v8.12.0

• Link to browser console log if useful:

https://i.imgur.com/LPrPlEe.png https://paste.debian.net/1049608/

• Link to server log if useful (journalctl or /var/www/peertube/storage/logs/):

I don't think this is related, but: https://paste.debian.net/1049609/

As for journalctl. Nobody in #systemd could tell me how to line wrap 'journalctl -u peertube'. In theory it might be 'journalctl -xn --no-pager -u peertube|less' but in practice this only shows a non-scrollable (PgDn/Up), small portion. So this is a bit of a puzzle, but it should have the info: https://paste.debian.net/1049612/

I know I can workaround the problem by restarting redis/peertube, but I'll just keep it broken for now, so the problem can be solved properly. It's in a broken state now, which might help solve it.

[sfstube]

This is an image of the error: https://www.sfstube.nl/misc/Server_error.png

I've tried to find the cause/origin.

client/src/app/shared/rest/rest-extractor.service.ts has:

      } else if (err.status === 500) {
        errorMessage = this.i18n('Server error. Please retry later.')
      }

I've tried to find possibly relevant inclusions of "500" and came up with these:

./server/middlewares/servers.ts has:

    // Problem with the url parsing?
    if (hostWithPort === null) {
      return res.sendStatus(500)
    }

./server/middlewares/validators/follows.ts has:

    // Force https if the administrator wants to make friends
    if (isTestInstance() === false && CONFIG.WEBSERVER.SCHEME === 'http') {
      return res.status(500)
        .json({
          error: 'Cannot follow on a non HTTPS web server.'
        })
        .end()
    }

./server/tests/real-world/real-world.ts has:

const actionInterval = program['intervalAction'] !== undefined ? parseInt(program['intervalAction'], 10) : 500

./server.ts has:

return res.status(err.status || 500).end()

I ran a grep -rl "err.status === 500" . in /usr/share/peertube and /var/lib/peertube but this yielded no results.

MDN Web Docs about code 500:

This error response is a generic "catch-all" response.

It might be useful for PeerTube administrators if this error message would not just show up for end-users, but if administrators would also receive a(n email) notification.

As the MDN Web Docs say, error code 500 is a generic response, which makes it not very informational. Currently, as a PeerTube administrator, I honestly have no idea where I should even begin to look in order to fix the problem that is causing this error message. As a non-developer, to me, anything could be causing this: peertube, redis, firewall, certificate, postgres, apache, configuration, etc. I suggest modification of the PeerTube code to return more informational error messages.

The instance in question is still giving the error message most of the time. Every now and then it's up again for a couple of hours, but then the error shows up again for more than 24hrs. This should make it an ideal environment to test what's causing the problem. Thus, if you have any commands/tests you want me to run while it's showing the error, to help figure out the cause of the problem, let me know and I'll run these commands/tests. Thanks.

[sfstube]

Put a $100 bounty on this at Bountysource.

[rigelk]

In the event of a 500 error, client logs are not useful. Your server log is difficult to read. Try exporting it in a more readable format next time with journalctl -u peertube --output=json > my_file_to_paste.json. YMMV.

As for the ResourceRequest timed out contained in your server log, it is coming from the connection to your PostgreSQL service. See https://github.com/sequelize/sequelize/issues/7884 for instance. Is your postgres server on a reliable connection (local to your machine or remote?). You might want to check that first. PeerTube refuses to start without a postgres connection, so it probably dropped after.

[sfstube]

Hello rigelk.

https://www.sfstube.nl/misc/my_file_to_paste.json

PostgreSQL is local to the instance. It appears to be up.

# ss -lntp|grep postgres
LISTEN     0      128    127.0.0.1:5432                     *:*                   users:(("postgres",pid=28651,fd=5))
LISTEN     0      128    127.0.0.200:5432                     *:*                   users:(("postgres",pid=28651,fd=3))
LISTEN     0      128        ::1:5432                    :::*                   users:(("postgres",pid=28651,fd=4))
# su - postgres
Last login: Sat Nov  3 10:15:02 CET 2018 on pts/0
-bash-4.2$ psql
Password: 
psql (9.2.24)
Type "help" for help.

postgres=#

Hopefully this information is useful.

[rigelk]

From what I could see, there is a misconfiguration of your pg_hba.conf (postgres). You can see where it is from your psql shell: psql -t -P format=unaligned -c 'show hba_file';.

[sfstube]

Hello again rigelk.

/var/lib/pgsql/data/pg_hba.conf contains (not including commented lines):

local samerole all        md5
host samerole all  127.0.0.200   255.255.255.255   pam  pamservice=postgresql_cpses
host samerole all  127.0.0.1   255.255.255.255   md5
host samerole all  ::1/128      md5
local all postgres        md5
host all postgres  127.0.0.1   255.255.255.255   md5

host peertube_prod peertube 127.0.0.1/32 md5

Another change appears to have happened about 4 or so days ago, where /etc/peertube/production.yaml is now an older version without changes I made through the web interface (the instance name is back to "PeerTube", signup is on false again, etc).

[sfstube]

The production.yaml is currently: http://paste.debian.net/1050652/

Attempting to login as root on the web interface results in: "ResourceRequest timed out"

[sfstube]

And, for completeness sake, here are the related Apache files: http://paste.debian.net/1050653/ http://paste.debian.net/1050654/

[sfstube]

This is a "ps auxwww" diff (with meld) of all the processes active on the server. https://www.sfstube.nl/misc/processes.jpg

On the left, when PeerTube works, which is only briefly after I reboot the server. On the right, when the error described in this GitHub issue shows up, which is most of the time.

I don't know if this is useful. At the bottom, when the error is active, it seems there are two extra postgres entries. Perhaps this is relevant, I don't know.

[sfstube]

I think it has been up for 2 full days in a row, without the error showing up; not sure. But, maybe something changed (improved); don't know what...

[sfstube]

(Error's back.)

[Chocobozzz]

~What is your PostgreSQL version?~ Saw it was 9.2. Please upgrade to >= 9.6

[sfstube]

My PeerTube instance is a hobby project that I mostly work on during weekends, when I'm free from work. It's currently Friday, and my plan is to get a newer PostgreSQL up and running this weekend. If a newer version does not fix the issue, I think it makes sense I'd request you re-open this issue. If a PostgreSQL version equal to or greater than 9.6 is required for PeerTube to prevent the error mentioned in this issue from showing up, it would be useful for other (current and future) PeerTube users if administrators would very clearly be warned about incompatible PostgreSQL versions on their systems. Merely making "Server error. Please retry later." visible for end-users is insufficient to inform administrators that their PostgreSQL version is too old to be used with PeerTube. Error code 500 is one of the most generic errors a system can give. I believe that PeerTube has enough privileges to be able to relay a more informative error message.

[sfstube]

My VPS has a cPanel/WHM environment, and unfortunately it only supports PostgreSQL 9.2 officially, with a feature request for newer versions.

To get 9.6, I decided to ignore what's officially supported and followed the "steps taken to upgrade" PostgreSQL from here but for 9.6 because the feature request mentioned above includes a comment that "unfortunately cPanel scripts can't handle version 10".

I started by downloading the postgresql.org package found here; this is what I tried:

# wget https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-centos96-9.6-3.noarch.rpm
# yum install pgdg-centos96-9.6-3.noarch.rpm
# postgres --version
postgres (PostgreSQL) 9.2.24
# systemctl reload postgresql
# postgres --version
postgres (PostgreSQL) 9.2.24
# systemctl stop postgresql
# systemctl start postgresql
# postgres --version
postgres (PostgreSQL) 9.2.24
# service postgresql stop
Redirecting to /bin/systemctl stop postgresql.service
# service postgresql start
Redirecting to /bin/systemctl start postgresql.service
# postgres --version
postgres (PostgreSQL) 9.2.24
# rpm -qa |grep post
postgresql-9.2.24-1.el7_5.x86_64
postgresql-libs-9.2.24-1.el7_5.x86_64
cpanel-postgresql-libs-9.2.23-1.cp1176.x86_64
postgresql-contrib-9.2.24-1.el7_5.x86_64
cpanel-postgresql-9.2.23-1.cp1176.x86_64
postgresql-server-9.2.24-1.el7_5.x86_64
postgresql-devel-9.2.24-1.el7_5.x86_64
# yum install postgresql96 postgresql96-devel postgresql96-libs postgresql96-server
# service postgresql restart
# postgres --version
postgres (PostgreSQL) 9.2.24
# yum remove postgresql postgresql-devel postgresql-libs postgresql-server
# service postgresql restart
Redirecting to /bin/systemctl restart postgresql.service
Failed to restart postgresql.service: Unit not found.
# cat /var/lib/pgsql/data/PG_VERSION
9.2

(From here I tried to just copy+paste the commands mentioned in the post that describes the steps to upgrade.)

# mv /var/lib/pgsql/.bash_profile.rpmsave /var/lib/pgsql/.bash_profile
# ln -sfn /etc/init.d/postgresql-10 /etc/init.d/postgresql
# service postgresql initdb
The service command supports only basic LSB actions (start, stop, restart, try-restart, reload, force-reload, status). For other actions, please try to use systemctl.
# chkconfig postgresql on
error reading information on service postgresql: No such file or directory
# cd /var/lib/pgsql; ln -sfn 10/backups; ln -sfn 10/data; ln -sfn 10/pgstartup.log
ln: ‘./data’: cannot overwrite directory
# cd /usr/bin
# ln -sfn /usr/pgsql-10/bin/postgres
# ln -sfn /usr/pgsql-10/bin/postmaster
# ln -sfn /usr/pgsql-10/bin/pg_config
# ln -sfn /usr/pgsql-10/bin/pg_ctl
# ln -sfn /etc/alternatives/pgsql-pg_dump pg_dump
# ln -sfn /etc/alternatives/pgsql-pg_dumpall pg_dumpall
# ln -sfn /etc/alternatives/pgsql-pg_restore pg_restore
# ln -sfn /etc/alternatives/pgsql-psql psql
# postgres --version
-bash: /usr/bin/postgres: No such file or directory

All in all, it didn't work, and I don't know what do do next.

[rigelk]

@sfstube now it's a PostgreSQL-specific issue and I'm sure you would have better support by their issue tracker or on your distribution forums.

Blindly copying commands is not going to help, though. Uninstalling postgresql and installing postgresql96 left no service unit (systemd service config file) and no binary. Maybe they are installed under a different name (postgresql96?).

[sfstube]

To wrap up this thread and post my solution to the problems I had with PeerTube.

I have created my own video sharing content management system, "FSTube". It is available as free and open-source software: https://www.fstube.org/

An example (demo) website is FreeSpeechTube: https://www.freespeechtube.org/