Plugin Server Fails to Start on Fresh Self-Hosted Docker Install (Ubuntu 24.04)

Environment

PostHog Version: latest
Installation Method: Self-hosted Docker
Operating System: Ubuntu 24.04

Description

After performing a fresh installation of PostHog using Docker on Ubuntu 24.04, the plugin server is failing to start. The error logs indicate that the encryption keys are not set, causing a startup failure.

Steps to Reproduce

Set up a fresh Ubuntu 24.04 server
Install Docker and Docker Compose
Follow the PostHog self-hosted Docker installation instructions
Attempt to start the PostHog services

Error Logs

{"level":"error","time":1727979202583,"pid":73,"hostname":"f31d1dde2c9f","error":"Error: Encryption keys are not set\n    at new EncryptedFields (/code/plugin-server/dist/cdp/encryption-utils.js:10:19)\n    at createHub (/code/plugin-server/dist/utils/db/hub.js:184:26)\n    at async setupHub (/code/plugin-server/dist/main/pluginsServer.js:157:34)\n    at async startPluginsServer (/code/plugin-server/dist/main/pluginsServer.js:166:25)","msg":"[MAIN] 💥 Launchpad failure!"}
{"level":"error","time":1727979202583,"pid":73,"hostname":"f31d1dde2c9f","error":{},"msg":"[MAIN] 💥 Exception while starting server, shutting down!"}

Expected Behavior

The plugin server should start successfully without any errors.

Actual Behavior

The plugin server fails to start, reporting that encryption keys are not set.

Additional Context

This is a fresh installation, so no custom configurations have been applied.
All other services (ClickHouse, Kafka, PostgreSQL, Redis, and object storage) appear to connect successfully.

Questions

Are there any additional configuration steps required for setting encryption keys in a self-hosted Docker environment?
Could this be related to any recent changes in PostHog or specific requirements for Ubuntu 24.04?

Any assistance in resolving this issue would be greatly appreciated. Thank you!

Debug info

No response

samuelnygaard commented 1 month ago

I also have this issue

Kvintus commented 1 month ago

Same

Kvintus commented 1 month ago

@switchp0rt @samuelnygaard

So... I found the issue. It's not documented anywhere but we have to set the ENCRYPTION_SALT_KEYS env variable to the compose services, it seems like, anywhere where the SECRET_KEY is provided as well. Most likely just the plugin-server .

The value 00beef0000beef0000beef0000beef00 should be fine. 😊

TLDR: set this env var: ENCRYPTION_SALT_KEYS=00beef0000beef0000beef0000beef00

And to someone from the PostHog team, probably @benjackwhite :

The issue originates here: https://github.com/PostHog/posthog/pull/24966

And although it's stated here, that default value should be provided for hobby deployments:

https://github.com/PostHog/posthog/blob/914ef395c85f31c662188268a39df8d883cf7166/posthog/settings/access.py#L74-L75

The same does not seem to be the case for the plugin-server: https://github.com/PostHog/posthog/blob/914ef395c85f31c662188268a39df8d883cf7166/plugin-server/src/config/config.ts#L178-L180

pauldambra commented 1 month ago

fix to set value on install or upgrade is here https://github.com/PostHog/posthog/pull/25454

pauldambra commented 1 month ago

Hey I believe this is fixed by #25518 (certainly fixed my self hosted instance)

If you run the upgrade script we should now generate the key you need and things will start

PostHog / posthog