Hobby deployment failing - Clickhouse errors

Jdimango commented 1 month ago

Bug Description

When attempting to run the hobby deployment script after 10 minutes the deployment does not update. Checking the docker containers with ps I see four containers continuously restarting.

clickhouse
plugins
web
worker

I believe I've deduced the issue to the click house not starting due to a dependency issue.

How to reproduce

Run hobby deployment script from docs /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/posthog/posthog/HEAD/bin/deploy-hobby)"
Wait for the 5-10 minutes
Check docker logs for click house

Additional context

I have run the upgrade script a few times over the last few days after viewing some other similar click house issues being resolved. /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/posthog/posthog/HEAD/bin/upgrade-hobby)"

I am running this on a GCP VM. The image was built with the following.

Canonical, Ubuntu, 20.04 LTS, amd64 focal image built on 2024-08-30

I've pasted the error logs from the click house container below.

Debug info

/entrypoint.sh: running /docker-entrypoint-initdb.d/init-db.sh
Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
python3.9 is already the newest version (3.9.5-3ubuntu0~20.04.1).
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 libc-bin : Depends: libc6 (< 2.32) but 2.35-0ubuntu3 is to be installed
 libc6 : Breaks: locales (< 2.35) but 2.31-0ubuntu9.14 is to be installed
         Recommends: libnss-nis but it is not installable
         Recommends: libnss-nisplus but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).

Jdimango commented 1 month ago

Any updates on the dependencies? I've attempted the update script again with no luck.

hosseinhashemi commented 2 weeks ago

I have the same issue on Ubuntu 20.4. Any update?

pauldambra commented 2 weeks ago

@Jdimango @hosseinhashemi i just ran the upgrade script on my ubuntu 24 and it succeeds

you get the logs above from the clickhouse container?

Jdimango commented 2 weeks ago

I said I was using ubuntu 20.4 not 24. I’ve ran the upgrade script whenever I noticed a new release. No luck.

My solution was switching to Debian.

pauldambra commented 2 weeks ago

I said I was using ubuntu 20.4 not 24.

cool cool, i was just being clear on what i was testing with

My solution was switching to Debian.

my guess is the solution there was really the switch to a new install and we weren't recovering from the E: Unmet dependencies. error - but that's only a guess - glad you found a solution!

@hosseinhashemi that would suggest that if you're able to start from scratch things would be ok for you - is that possible? is that true?

pauldambra commented 2 weeks ago

the other thing of course is... what happens if you run apt --fix-broken install as the error message suggests?

hosseinhashemi commented 2 weeks ago

@pauldambra To fix the issue, I installed Docker first and then I installed the PostHog. By this, the problem resolved.

Jdimango commented 2 weeks ago

Running apt --fix-broken install unfortunately didn’t install any packages and left 7 alone. I actually did try installing docker first and ran into the same issue. It seems that a newer version of libc is installed and it can’t downgrade.

PostHog / posthog