Open futureweb opened 1 month ago
azaslavsky
commented
1 month ago Two previous issues, both of which were related to permissions and unexpected machine state:
https://github.com/getsentry/self-hosted/issues/1658 https://github.com/getsentry/self-hosted/issues/1794
Can you try the approaches recommended here and report back?
futureweb
commented
1 month ago @azaslavsky - one approach there is to run "./reset.sh" which is no option as we need the captured Data.
The other one is to chmod 777 everything - doesn't sound like an approach one wants to do on a productive machine?
chmod 777 /opt/sentry
chmod 777 /opt/sentry/sentry/postgres/init_hba.sh
chmod 777 /opt/sentry/sentry/postgres/postgres-entrypoint.sh
chmod 777 /opt/sentry/sentry/sentry/entrypoint.shAlso on both of those 2 linked issues it's failing on postgres which in this case isn't the problem if I don't overlook anything? In my case it's not even saying where it's failing?!?
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't get final child's PID from pipe: EOF: unknownAny other ideas on how to further debug this issue?
azaslavsky
commented
1 month ago Some googling revealed this suggestion, but tat this point I am just prowling the internet for random answers. This error is very likely to do with your system configuration - in the past, this has often been with regards to permissioning, hence why chmod solution was suggested, but it could be a number of other settings as well. Ultimately, I don't have very much information about your specific system's settings, and docker is not great at providing detailed error information for investigation.
futureweb
commented
1 month ago @azaslavsky unfortunately the max_user_namespaces also seems to be already high enough here:
# sysctl user.max_user_namespaces
user.max_user_namespaces = 2147483647I'm running Sentry on clean install of AlmaLinux 8:
# cat /etc/redhat-release
AlmaLinux release 8.9 (Midnight Oncilla)running out of ideas on what else to try to get sentry updated here :-(
azaslavsky
commented
1 month ago I'm not sure what we can do remotely either. We test on ubuntu and alpine linux, and it works in both environments. Ultimately, there are any number of reasons why this could pop up, and the solution is likely not self-hosted specific.
aldy505
commented
1 month ago I'm running Sentry on clean install of AlmaLinux 8:
# cat /etc/redhat-release AlmaLinux release 8.9 (Midnight Oncilla)running out of ideas on what else to try to get sentry updated here :-(
Hey @futureweb can you try to install it on higher version of AlmaLinux? A wild guess from me is that it failed to build on Linux kernel 4.x, but AlmaLinux 9.x and Ubuntu 20.x uses Linux kernel 5.x. If you're okay with moving to some other distro other than RHEL derivatives, I'd highly recommend to install sentry self-hosted on Debian 12. If installing on AlmaLinux 9.x or Debian 11/12 works with no problem, then I can assume that.. Docker does not plays well with old distro running old kernel version.
futureweb
commented
1 month ago @aldy505 - as we operate all our Servers on RHEL based Servers we would like to stay within this universe, if possible. Just tried to install same Sentry Version (23.7.2) on AlmaLinux 9 as we got on our AlmaLinux 8 to migrate current installation to new server. But it won't even install in first place! :-/
VERSION="23.7.2"
git clone https://github.com/getsentry/self-hosted.git
cd self-hosted
git checkout ${VERSION}
SENTRY_IMAGE=getsentry/sentry:${VERSION} ./install.sh --report-self-hosted-issues
...
...
...
File "/usr/local/lib/python3.8/site-packages/django/core/cache/backends/memcached.py", line 234, in __init__
import pymemcache.serde
ModuleNotFoundError: No module named 'pymemcache'
Error in install/set-up-and-migrate-database.sh:12.
'$dcr web upgrade' exited with status 1
-> ./install.sh:main:34
--> install/set-up-and-migrate-database.sh:source:12
install/error-handling.sh: line 82: /bin/docker: Argument list too long# uname -a
Linux hostname 5.14.0 #1 SMP Tue Jan 9 19:45:01 MSK 2024 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/redhat-release
AlmaLinux release 9.4 (Seafoam Ocelot)
# docker info --format '{{.Architecture}}'
x86_64
# docker --version
Docker version 26.1.2, build 211e74b
# docker compose version
Docker Compose version v2.27.0seems like no luck with rhel based servers here?!? :-(
futureweb
commented
1 month ago update: after running /var/www/sentry/self-hosted/scripts/reset.sh and another fresh install the installation went through completely ... but then when I tried to fire it up:
# docker compose up -d
[+] Running 36/39
✔ Container sentry-self-hosted-redis-1 Healthy 50.2s
✔ Container sentry-self-hosted-symbolicator-cleanup-1 Started 4.9s
✔ Container sentry-self-hosted-vroom-cleanup-1 Started 4.4s
✔ Container sentry-self-hosted-smtp-1 Started 5.0s
✔ Container sentry-self-hosted-zookeeper-1 Healthy 35.5s
✔ Container sentry-self-hosted-symbolicator-1 Started 4.5s
✔ Container sentry-self-hosted-clickhouse-1 Healthy 36.6s
✔ Container sentry-self-hosted-geoipupdate-1 Started 5.0s
✔ Container sentry-self-hosted-postgres-1 Healthy 50.2s
✔ Container sentry-self-hosted-memcached-1 Started 4.3s
✔ Container sentry-self-hosted-kafka-1 Healthy 50.2s
✔ Container sentry-self-hosted-snuba-profiling-profiles-consumer-1 Started 49.6s
✔ Container sentry-self-hosted-snuba-replays-consumer-1 Started 49.6s
✔ Container sentry-self-hosted-snuba-outcomes-consumer-1 Started 49.0s
✔ Container sentry-self-hosted-snuba-subscription-consumer-events-1 Started 49.6s
✔ Container sentry-self-hosted-snuba-consumer-1 Started 49.7s
✔ Container sentry-self-hosted-vroom-1 Started 49.6s
✔ Container sentry-self-hosted-snuba-replacer-1 Started 49.3s
✔ Container sentry-self-hosted-snuba-transactions-consumer-1 Started 48.7s
✔ Container sentry-self-hosted-snuba-subscription-consumer-sessions-1 Started 49.6s
✔ Container sentry-self-hosted-snuba-profiling-functions-consumer-1 Started 49.6s
✔ Container sentry-self-hosted-snuba-subscription-consumer-transactions-1 Started 49.7s
✔ Container sentry-self-hosted-snuba-api-1 Started 49.2s
✔ Container sentry-self-hosted-snuba-sessions-consumer-1 Started 49.0s
✔ Container sentry-self-hosted-worker-1 Started 54.6s
✔ Container sentry-self-hosted-post-process-forwarder-errors-1 Started 54.4s
✔ Container sentry-self-hosted-ingest-replay-recordings-1 Started 54.6s
✔ Container sentry-self-hosted-web-1 Started 54.4s
✔ Container sentry-self-hosted-post-process-forwarder-transactions-1 Started 54.6s
✔ Container sentry-self-hosted-cron-1 Started 54.3s
⠇ Container sentry-self-hosted-sentry-cleanup-1 Starting 54.6s
✔ Container sentry-self-hosted-events-consumer-1 Started 54.2s
⠇ Container sentry-self-hosted-subscription-consumer-events-1 Starting 54.6s
✔ Container sentry-self-hosted-attachments-consumer-1 Started 54.6s
✔ Container sentry-self-hosted-transactions-consumer-1 Started 54.4s
⠇ Container sentry-self-hosted-subscription-consumer-transactions-1 Starting 54.6s
✔ Container sentry-self-hosted-ingest-profiles-1 Started 54.0s
✔ Container sentry-self-hosted-relay-1 Created 0.1s
✔ Container sentry-self-hosted-nginx-1 Created 0.1s
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't get final child's PID from pipe: EOF: unknowndamnit ... this is frustrating :-(
futureweb
commented
1 month ago further update:
chmod 777 self-hosted
cd self-hosted
# chmod 777 /var/www/sentry/self-hosted/postgres/postgres-entrypoint.sh
# chmod 777 /var/www/sentry/self-hosted/postgres/init_hba.sh
# chmod 777 /var/www/sentry/self-hosted/sentry/entrypoint.shThen the 'Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't get final child's PID from pipe: EOF: unknown' Error was gone.
Will try to see if I can migrate old Data to new Server + then Update to latest Sentry
aldy505
commented
1 month ago I wonder if you could actually try Debian or Ubuntu just for the sake of it, and see if it suddenly works.....
futureweb
commented
1 month ago @aldy505 never worked with Debian so far - but yea - could create a Debian Machine and try. I guess it should work there when it's tested on Debian ^^ Just tried to avoid it as it would be the only Debian Machine in our Infrastructure with noone really used to work with Debian and also blowing up Virtuozzo Template Cache, Ploop Cache and some other things on the Root Servers ... :-/
notboring
commented
1 week ago The same error happens when starting up the self hosted version on a fresh ubuntu 22.04 (Linux 5.2) using latest docker version. So the error is not neccessarily related to old linux versions. In my case ./scripts/reset.sh or chmod 777 on all files also didn't help with anything.
azaslavsky
commented
1 week ago I don't think this related to your version of self-hosted, but rather to something in your environment, whether it's permissions or something else. Ex: this stackoverflow issue. It doesn't happen in our CI or dogfood instance, so it's a bit hard to remotely figure out what exactly about this particular configuration is causing issues from afar.
Self-Hosted Version
24.4.2
CPU Architecture
x86_64
Docker Version
Docker version 26.1.1, build 4cf5afa
Docker Compose Version
Docker Compose version v2.27.0
Steps to Reproduce
Update or fresh Re-Install Version 24.4.2 (coming from Sentry 23.7.2 adbbcd8) Fire it up (docker compose up -d) Won't start - Error:
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't get final child's PID from pipe: EOF: unknownExpected Result
Sentry should start without Problems
Actual Result
composer.log install.log
Event ID
No response