search

getsentry / self-hosted

Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept
https://develop.sentry.dev/self-hosted/
Other
7.83k stars 1.76k forks source link

SIGERR on line 113 when Upgrading using install.sh #911

Closed pjlamb12 closed 3 years ago

pjlamb12 commented 3 years ago

Version Information

Version: 21.1.0 → 21.2.0

Steps to Reproduce

I started yesterday on v20.10.1 and started upgrading tag by tag to the latest. I hadn't made any changes to any of the files, so all I needed to do was checkout the tag and run ./install.sh. I successfully upgraded to 20.11.0, then 20.11.1, then 20.12.1, and finally 21.1.0. Everything worked perfectly. I then tried upgrading to 21.2.0 using the same method as before. This is where I saw the error.

Expected Result

Expected the upgrade to work.

Actual Result

An error occurs, making the upgrade unsuccessful

Logs

▶ Defining variables and helpers ...

▶ Parsing command line ...

▶ Setting up error handling ...

▶ Checking minimum requirements ... docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"nproc\": executable file not found in $PATH": unknown. time="2021-03-30T15:28:03Z" level=error msg="error waiting for container: context canceled" An error occurred, caught SIGERR on line 113 Cleaning up...

System Information

docker --version: 19.03.13, build 4484c46d9d docker-compose --version: 1.25.4, build 8d51620a python --version: 2.7.16 python3 --version: 3.7.3

BYK commented 3 years ago

I think this is a transitional error on your system. Might be due to a corrupted download or something as 21.2.0 images are confirmed to be working.

Btw you can just skip all the way to latest for now.

pjlamb12 commented 3 years ago

Any ideas on what steps to take to get the error to go away? I know it's impossible for you to know for sure but I'm just hoping you might have some ideas of what has happened in the past.

pjlamb12 commented 3 years ago

We've restarted the server we're running this on, but still to no avail. The latest tag that works is 21.1.0. Everything after that causes the above error.

BYK commented 3 years ago

I think this is due to https://github.com/getsentry/onpremise/pull/834. Are you sure you are getting the right version of the repo which adds the new entrypoint.sh under the sentry directory?

pjlamb12 commented 3 years ago

Yes, the sentry/entrypoint.sh file is present. I'm currently trying this on the latest from master, but it fails in the same way and manner as I mentioned above.

BYK commented 3 years ago

@pjlamb12 could you have some overrides regarding your volume mounts and sentry config directory?

pjlamb12 commented 3 years ago

When I run git status to see if I had changed anything (did this for each of the upgrades that I did yesterday) There are no changes. It's been running for about a year though, so if the changes wouldn't show in git I guess I could have done that and don't remember. Where would I check for that?

pjlamb12 commented 3 years ago

As far as I can tell, the only thing I've changed is in sentry/config.yml and the value I provided is system.url-prefix

BYK commented 3 years ago

What do you get when you run git clean -n -d -x? Also can you share your install logs?

pjlamb12 commented 3 years ago 
Would remove geoip/GeoLite2-City.mmdb
Would remove relay/config.yml
Would remove relay/credentials.json
Would remove sentry/config.yml
Would remove sentry/requirements.txt
Would remove sentry/sentry.conf.py
Would remove sentry_install_log-2021-03-29_21-27-22.txt
Would remove sentry_install_log-2021-03-29_21-42-03.txt
Would remove sentry_install_log-2021-03-29_21-48-46.txt
Would remove sentry_install_log-2021-03-29_21-56-59.txt
Would remove sentry_install_log-2021-03-29_22-05-40.txt
Would remove sentry_install_log-2021-03-29_22-07-28.txt
Would remove sentry_install_log-2021-03-29_22-08-10.txt
Would remove sentry_install_log-2021-03-29_22-08-57.txt
Would remove sentry_install_log-2021-03-29_22-12-23.txt
Would remove sentry_install_log-2021-03-29_22-17-36.txt
Would remove sentry_install_log-2021-03-29_22-52-38.txt
Would remove sentry_install_log-2021-03-29_22-55-07.txt
Would remove sentry_install_log-2021-03-30_15-21-38.txt
Would remove sentry_install_log-2021-03-30_15-27-59.txt
Would remove sentry_install_log-2021-03-30_16-31-19.txt
Would remove sentry_install_log-2021-03-30_17-00-35.txt
Would remove sentry_install_log-2021-03-30_19-31-15.txt
Would remove sentry_install_log-2021-03-30_19-36-04.txt
Would remove sentry_install_log-2021-03-30_19-36-28.txt
Would remove symbolicator/config.yml
pjlamb12 commented 3 years ago

Obviously the logs can be removed. What about the rest of those?

pjlamb12 commented 3 years ago

Logs trying to install latest:

▶ Parsing command line ...

▶ Setting up error handling ...

▶ Checking minimum requirements ...
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"nproc\": executable file not found in $PATH": unknown.
time="2021-03-30T19:36:07Z" level=error msg="error waiting for container: context canceled"
An error occurred, caught SIGERR on line 13
Cleaning up...
pjlamb12 commented 3 years ago

Logs trying to install v21.1.0 (successful):

Checking minimum requirements...
WARN: Recommended minimum RAM available to Docker is 8000 MB, found 7979 MB

Creating volumes for persistent storage...
Created sentry-data.
Created sentry-postgres.
Created sentry-redis.
Created sentry-zookeeper.
Created sentry-kafka.
Created sentry-clickhouse.
Created sentry-symbolicator.

sentry/sentry.conf.py already exists, skipped creation.
sentry/config.yml already exists, skipped creation.
sentry/requirements.txt already exists, skipped creation.
symbolicator/config.yml already exists, skipped creation.
relay/config.yml already exists, skipped creation.

Fetching and updating Docker images...

Some service image(s) must be built from source by running:
    docker-compose build worker subscription-consumer-transactions sentry-cleanup cron snuba-cleanup post-process-forwarder web symbolicator-cleanup subscription-consumer-events ingest-consumer
21.1.0: Pulling from getsentry/sentry
Digest: sha256:130b8887ea5eb2c54bf9661546ecbc1cb410a8d914de82fddfc03374316450ed
Status: Image is up to date for getsentry/sentry:21.1.0
docker.io/getsentry/sentry:21.1.0

Building and tagging Docker images...

Building web
Step 1/5 : ARG SENTRY_IMAGE
Step 2/5 : ARG SENTRY_PYTHON2
Step 3/5 : FROM ${SENTRY_IMAGE}${SENTRY_PYTHON2:+-py2}
 ---> b3428e39dda5
Step 4/5 : COPY . /usr/src/sentry
 ---> Using cache
 ---> f42043f078cf
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> 26a5bba381e8

Successfully built 26a5bba381e8
Successfully tagged sentry-onpremise-local:latest
smtp uses an image, skipping
memcached uses an image, skipping
redis uses an image, skipping
postgres uses an image, skipping
zookeeper uses an image, skipping
kafka uses an image, skipping
clickhouse uses an image, skipping
geoipupdate uses an image, skipping
snuba-api uses an image, skipping
snuba-consumer uses an image, skipping
snuba-outcomes-consumer uses an image, skipping
snuba-sessions-consumer uses an image, skipping
snuba-transactions-consumer uses an image, skipping
snuba-replacer uses an image, skipping
snuba-subscription-consumer-events uses an image, skipping
snuba-subscription-consumer-transactions uses an image, skipping
symbolicator uses an image, skipping
relay uses an image, skipping
nginx uses an image, skipping
Building snuba-cleanup                      ... 
Building symbolicator-cleanup               ... 
Building web                                ... 
Building cron                               ... 
Building worker                             ... 
Building ingest-consumer                    ... 
Building post-process-forwarder             ... 
Building subscription-consumer-events       ... 
Building subscription-consumer-transactions ... 
Building sentry-cleanup                     ... 
Building post-process-forwarder
Building subscription-consumer-transactions
Building cron
Building ingest-consumer
Building worker
Step 1/5 : ARG SENTRY_IMAGE
Step 2/5 : ARG SENTRY_PYTHON2
Step 3/5 : FROM ${SENTRY_IMAGE}${SENTRY_PYTHON2:+-py2}
 ---> b3428e39dda5
Step 4/5 : COPY . /usr/src/sentry
 ---> Using cache
 ---> f42043f078cf
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> 26a5bba381e8

Successfully built 26a5bba381e8
Successfully tagged sentry-onpremise-local:latest
Building subscription-consumer-transactions ... done
Building subscription-consumer-events
Step 1/5 : ARG SENTRY_IMAGE
Step 2/5 : ARG SENTRY_PYTHON2
Step 3/5 : FROM ${SENTRY_IMAGE}${SENTRY_PYTHON2:+-py2}
 ---> b3428e39dda5
Step 4/5 : COPY . /usr/src/sentry
 ---> Using cache
 ---> f42043f078cf
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> 26a5bba381e8

Successfully built 26a5bba381e8
Successfully tagged sentry-onpremise-local:latest
Building post-process-forwarder             ... done
Building symbolicator-cleanup
Step 1/5 : ARG SENTRY_IMAGE
Step 2/5 : ARG SENTRY_PYTHON2
Step 3/5 : FROM ${SENTRY_IMAGE}${SENTRY_PYTHON2:+-py2}
 ---> b3428e39dda5
Step 4/5 : COPY . /usr/src/sentry
 ---> Using cache
 ---> f42043f078cf
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> 26a5bba381e8

Successfully built 26a5bba381e8
Successfully tagged sentry-onpremise-local:latest
Building cron                               ... done
Building sentry-cleanup
Step 1/5 : ARG SENTRY_IMAGE
Step 2/5 : ARG SENTRY_PYTHON2
Step 3/5 : FROM ${SENTRY_IMAGE}${SENTRY_PYTHON2:+-py2}
 ---> b3428e39dda5
Step 4/5 : COPY . /usr/src/sentry
 ---> Using cache
 ---> f42043f078cf
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> 26a5bba381e8

Successfully built 26a5bba381e8
Successfully tagged sentry-onpremise-local:latest
Building worker                             ... done
Building web
Step 1/5 : ARG SENTRY_IMAGE
Step 2/5 : ARG SENTRY_PYTHON2
Step 3/5 : FROM ${SENTRY_IMAGE}${SENTRY_PYTHON2:+-py2}
 ---> b3428e39dda5
Step 4/5 : COPY . /usr/src/sentry
 ---> Using cache
 ---> f42043f078cf
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> 26a5bba381e8

Successfully built 26a5bba381e8
Successfully tagged sentry-onpremise-local:latest
Building ingest-consumer                    ... done
Building snuba-cleanup
Step 1/5 : ARG SENTRY_IMAGE
Step 2/5 : ARG SENTRY_PYTHON2
Step 3/5 : FROM ${SENTRY_IMAGE}${SENTRY_PYTHON2:+-py2}
 ---> b3428e39dda5
Step 4/5 : COPY . /usr/src/sentry
 ---> Using cache
 ---> f42043f078cf
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> 26a5bba381e8

Successfully built 26a5bba381e8
Successfully tagged sentry-onpremise-local:latest
Building subscription-consumer-events       ... done
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
 ---> 0e9e67a40b39
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*
 ---> Using cache
 ---> 2d4f68cde53e
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
 ---> Using cache
 ---> aabde557a0f5
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
 ---> Using cache
 ---> 078ab9199a4f

Successfully built 078ab9199a4f
Successfully tagged symbolicator-cleanup-onpremise-local:latest
Building symbolicator-cleanup               ... done
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
 ---> 26a5bba381e8
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*
 ---> Using cache
 ---> ff2a02d56f0a
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
 ---> Using cache
 ---> fc5b7b163773
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
 ---> Using cache
 ---> a2d137501cb4

Successfully built a2d137501cb4
Successfully tagged sentry-cleanup-onpremise-local:latest
Building sentry-cleanup                     ... done
Step 1/5 : ARG SENTRY_IMAGE
Step 2/5 : ARG SENTRY_PYTHON2
Step 3/5 : FROM ${SENTRY_IMAGE}${SENTRY_PYTHON2:+-py2}
 ---> b3428e39dda5
Step 4/5 : COPY . /usr/src/sentry
 ---> Using cache
 ---> f42043f078cf
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> 26a5bba381e8

Successfully built 26a5bba381e8
Step 1/5 : ARG BASE_IMAGESuccessfully tagged sentry-onpremise-local:latest
Building web                                ... done

Step 2/5 : FROM ${BASE_IMAGE}
 ---> 7cfc0fa844f0
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*
 ---> Using cache
 ---> 37b71e523b2b
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
 ---> Using cache
 ---> fd58d3bb785d
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
 ---> Using cache
 ---> 9795fdca7713

Successfully built 9795fdca7713
Successfully tagged snuba-cleanup-onpremise-local:latest
Building snuba-cleanup                      ... done

Docker images built.
Removing network onpremise_default
Network onpremise_default not found.
Removing network sentry_onpremise_default
Creating network "sentry_onpremise_default" with the default driver
Bootstrapping and migrating Snuba...
Creating sentry_onpremise_redis_1 ... 
Creating sentry_onpremise_zookeeper_1 ... 
Creating sentry_onpremise_clickhouse_1 ... 
Creating sentry_onpremise_zookeeper_1  ... done
Creating sentry_onpremise_kafka_1      ... 
Creating sentry_onpremise_redis_1      ... done
Creating sentry_onpremise_clickhouse_1 ... done
Creating sentry_onpremise_kafka_1      ... done
+ '[' b = - ']'
+ snuba bootstrap --help
+ set -- snuba bootstrap --no-migrate --force
+ set gosu snuba snuba bootstrap --no-migrate --force
+ exec gosu snuba snuba bootstrap --no-migrate --force
%3|1617133031.637|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 1ms in state CONNECT)
2021-03-30 19:37:12,635 Connection to Kafka failed (attempt 0)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
%3|1617133032.643|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 7ms in state CONNECT, 1 identical error(s) suppressed)
%3|1617133033.647|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 2ms in state CONNECT)
%3|1617133034.647|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2021-03-30 19:37:14,655 Connection to Kafka failed (attempt 1)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
%3|1617133035.659|FAIL|rdkafka#producer-3| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1617133036.659|FAIL|rdkafka#producer-3| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2021-03-30 19:37:16,667 Connection to Kafka failed (attempt 2)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
%3|1617133037.671|FAIL|rdkafka#producer-4| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1617133038.676|FAIL|rdkafka#producer-4| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2021-03-30 19:37:18,682 Connection to Kafka failed (attempt 3)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
%3|1617133039.685|FAIL|rdkafka#producer-5| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1617133040.690|FAIL|rdkafka#producer-5| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2021-03-30 19:37:20,692 Connection to Kafka failed (attempt 4)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
%3|1617133041.698|FAIL|rdkafka#producer-6| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1617133042.702|FAIL|rdkafka#producer-6| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2021-03-30 19:37:22,705 Connection to Kafka failed (attempt 5)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
%3|1617133043.712|FAIL|rdkafka#producer-7| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
2021-03-30 19:37:24,718 Connection to Kafka failed (attempt 6)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
%3|1617133044.727|FAIL|rdkafka#producer-7| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 8ms in state CONNECT, 1 identical error(s) suppressed)
%3|1617133045.720|FAIL|rdkafka#producer-8| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1617133046.724|FAIL|rdkafka#producer-8| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2021-03-30 19:37:26,727 Connection to Kafka failed (attempt 7)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
%3|1617133047.733|FAIL|rdkafka#producer-9| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1617133048.737|FAIL|rdkafka#producer-9| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2021-03-30 19:37:28,742 Connection to Kafka failed (attempt 8)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
%3|1617133049.745|FAIL|rdkafka#producer-10| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1617133050.749|FAIL|rdkafka#producer-10| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2021-03-30 19:37:30,752 Connection to Kafka failed (attempt 9)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
%3|1617133051.758|FAIL|rdkafka#producer-11| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1617133052.758|FAIL|rdkafka#producer-11| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2021-03-30 19:37:32,765 Connection to Kafka failed (attempt 10)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
%3|1617133053.768|FAIL|rdkafka#producer-12| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
2021-03-30 19:37:34,774 Connection to Kafka failed (attempt 11)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
2021-03-30 19:37:37,319 Failed to create topic cdc
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 91, in bootstrap
    future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'cdc' already exists."}
2021-03-30 19:37:37,344 Failed to create topic outcomes
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 91, in bootstrap
    future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'outcomes' already exists."}
2021-03-30 19:37:37,346 Failed to create topic ingest-sessions
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 91, in bootstrap
    future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'ingest-sessions' already exists."}
2021-03-30 19:37:37,347 Failed to create topic events
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 91, in bootstrap
    future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'events' already exists."}
2021-03-30 19:37:37,350 Failed to create topic snuba-commit-log
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 91, in bootstrap
    future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'snuba-commit-log' already exists."}
2021-03-30 19:37:37,350 Failed to create topic errors-replacements
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 91, in bootstrap
    future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'errors-replacements' already exists."}
2021-03-30 19:37:37,351 Failed to create topic event-replacements
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 91, in bootstrap
    future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'event-replacements' already exists."}
Starting sentry_onpremise_redis_1 ... 
Starting sentry_onpremise_redis_1 ... done
Starting sentry_onpremise_clickhouse_1 ... 
Starting sentry_onpremise_zookeeper_1  ... 
Starting sentry_onpremise_clickhouse_1 ... done
Starting sentry_onpremise_zookeeper_1  ... done
Starting sentry_onpremise_kafka_1      ... 
Starting sentry_onpremise_kafka_1      ... done
+ '[' m = - ']'
+ snuba migrations --help
+ set -- snuba migrations migrate --force
+ set gosu snuba snuba migrations migrate --force
+ exec gosu snuba snuba migrations migrate --force
Finished running migrations

Setting up database...
Creating sentry_onpremise_memcached_1 ... 
Starting sentry_onpremise_zookeeper_1 ... 
Starting sentry_onpremise_zookeeper_1 ... done
Creating sentry_onpremise_symbolicator_1 ... 
Creating sentry_onpremise_postgres_1     ... 
Starting sentry_onpremise_clickhouse_1   ... 
Starting sentry_onpremise_clickhouse_1   ... done
Starting sentry_onpremise_redis_1        ... 
Creating sentry_onpremise_smtp_1         ... 
Starting sentry_onpremise_redis_1        ... done
Starting sentry_onpremise_kafka_1        ... 
Starting sentry_onpremise_kafka_1        ... done
Creating sentry_onpremise_snuba-transactions-consumer_1 ... 
Creating sentry_onpremise_snuba-sessions-consumer_1     ... 
Creating sentry_onpremise_snuba-outcomes-consumer_1     ... 
Creating sentry_onpremise_snuba-api_1                   ... 
Creating sentry_onpremise_snuba-consumer_1              ... 
Creating sentry_onpremise_snuba-replacer_1              ... 
Creating sentry_onpremise_snuba-transactions-consumer_1 ... done
Creating sentry_onpremise_snuba-consumer_1              ... done
Creating sentry_onpremise_symbolicator_1                ... done
Creating sentry_onpremise_snuba-sessions-consumer_1     ... done
Creating sentry_onpremise_snuba-api_1                   ... done
Creating sentry_onpremise_memcached_1                   ... done
Creating sentry_onpremise_smtp_1                        ... done
Creating sentry_onpremise_snuba-replacer_1              ... done
Creating sentry_onpremise_snuba-outcomes-consumer_1     ... done
Creating sentry_onpremise_postgres_1                    ... done
/usr/local/lib/python3.6/site-packages/sentry/runner/initializer.py:190: DeprecatedSettingWarning: The GITHUB_EXTENDED_PERMISSIONS setting is deprecated. Please use SENTRY_OPTIONS['github-login.extended-permissions'] instead.
  warnings.warn(DeprecatedSettingWarning(options_mapper[k], "SENTRY_OPTIONS['%s']" % k))
19:38:25 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
19:38:54 [INFO] sentry.plugins.github: apps-not-configured
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, jira_ac, nodestore, sentry, sessions, sites, social_auth
Running migrations:
  No migrations to apply.
Creating missing DSNs
Correcting Group.num_comments counter
Setting up IP address geolocation ...
IP address geolocation database already exists.
IP address geolocation is not configured for updates.
See https://develop.sentry.dev/self-hosted/geolocation/ for instructions.
Error setting up IP address geolocation.

----------------
You're all done! Run the following command to get Sentry running:

  docker-compose up -d
BYK commented 3 years ago

Obviously the logs can be removed. What about the rest of those?

Yeah don't remove the others :)

When looking closer and with the install logs, just found the problem, it's this line: https://github.com/getsentry/onpremise/blob/2ab05909248f563147ab1cd4ed7488475debb4d9/install/check-minimum-requirements.sh#L13

For some reason your busybox image does not have nproc in it. Could you have a very old busybox image lingering around? If you don't mind can you run docker rmi busybox:latest and try again?

pjlamb12 commented 3 years ago

I think that fixed it! I got another error, but this one is because only 1 core is available to docker and I need to up that to 2. Thanks so much for your help!

BYK commented 3 years ago

Glad that helped! Will close this for now :)

@chadwhitacre - should we pin the busybox version?

pjlamb12 commented 3 years ago

One more question, although not directly related to this bug. Do you know why "Performance" isn't showing up in the main nav on the updated installation, but is on the one I installed locally for testing? screenshot 104

BYK commented 3 years ago

Probably a new switch we enabled in the sentry.conf.py file. If you are sure your local copy is unchanged, just delete it and it will be regenerated from our default, which should enable the feature for you.