search

benoitc / gunicorn

gunicorn 'Green Unicorn' is a WSGI HTTP Server for UNIX, fast clients and sleepy applications.
http://www.gunicorn.org
Other
9.71k stars 1.74k forks source link

QdrantClient seems to crash Gunicorn #3213

Closed thomasf1 closed 1 month ago

thomasf1 commented 3 months ago

Not sure if this is more a problem with Gunicorn or Qdrant:

This code works with Gunicorn (launched with gunicorn -w 2 -b 0.0.0.0:3000 gunicorn-test:app --enable-stdio-inheritance --log-level debug):

from flask import Flask
from qdrant_client import QdrantClient

app = Flask(__name__)

@app.route("/")
def hello_world():
    print("Hello, World!", flush=True)
    return "<p>Hello, World!</p>"

This code leads to a crash loop:

from flask import Flask
from qdrant_client import QdrantClient

app = Flask(__name__)

client = QdrantClient(
        url="https://(redacted).cloud.qdrant.io:6333", 
        api_key="(redacted)",
    )

@app.route("/")
def hello_world():
    print("Hello, World!", flush=True)
    return "<p>Hello, World!</p>"

Leading to

[2024-05-16 13:21:10 +0200] [60914] [DEBUG] 2 workers
[2024-05-16 13:21:12 +0200] [60914] [ERROR] Worker (pid:60929) was sent SIGSEGV!
[2024-05-16 13:21:12 +0200] [60932] [INFO] Booting worker with pid: 60932
[2024-05-16 13:21:12 +0200] [60914] [ERROR] Worker (pid:60930) was sent SIGSEGV!
[2024-05-16 13:21:12 +0200] [60935] [INFO] Booting worker with pid: 60935
[2024-05-16 13:21:13 +0200] [60914] [ERROR] Worker (pid:60932) was sent SIGSEGV!
[2024-05-16 13:21:13 +0200] [60937] [INFO] Booting worker with pid: 60937
[2024-05-16 13:21:13 +0200] [60914] [ERROR] Worker (pid:60935) was sent SIGSEGV!
[2024-05-16 13:21:13 +0200] [60938] [INFO] Booting worker with pid: 60938
[2024-05-16 13:21:14 +0200] [60914] [ERROR] Worker (pid:60937) was sent SIGSEGV!
[2024-05-16 13:21:14 +0200] [60939] [INFO] Booting worker with pid: 60939
[2024-05-16 13:21:14 +0200] [60914] [ERROR] Worker (pid:60938) was sent SIGSEGV!
[2024-05-16 13:21:14 +0200] [60940] [INFO] Booting worker with pid: 60940
[2024-05-16 13:21:15 +0200] [60914] [ERROR] Worker (pid:60939) was sent SIGSEGV!
[2024-05-16 13:21:15 +0200] [60941] [INFO] Booting worker with pid: 60941
[2024-05-16 13:21:15 +0200] [60914] [ERROR] Worker (pid:60940) was sent SIGSEGV!
[2024-05-16 13:21:15 +0200] [60942] [INFO] Booting worker with pid: 60942
[2024-05-16 13:21:16 +0200] [60914] [ERROR] Worker (pid:60941) was sent SIGSEGV!
[2024-05-16 13:21:16 +0200] [60946] [INFO] Booting worker with pid: 60946
[2024-05-16 13:21:16 +0200] [60914] [ERROR] Worker (pid:60942) was sent SIGSEGV!
[2024-05-16 13:21:16 +0200] [60947] [INFO] Booting worker with pid: 60947
[2024-05-16 13:21:17 +0200] [60914] [ERROR] Worker (pid:60946) was sent SIGSEGV!
[2024-05-16 13:21:17 +0200] [60948] [INFO] Booting worker with pid: 60948
[2024-05-16 13:21:17 +0200] [60914] [ERROR] Worker (pid:60947) was sent SIGSEGV!
[2024-05-16 13:21:17 +0200] [60914] [DEBUG] 1 workers
[2024-05-16 13:21:17 +0200] [60949] [INFO] Booting worker with pid: 60949
[2024-05-16 13:21:17 +0200] [60914] [DEBUG] 2 workers
[2024-05-16 13:21:18 +0200] [60914] [ERROR] Worker (pid:60948) was sent SIGSEGV!
[2024-05-16 13:21:18 +0200] [60950] [INFO] Booting worker with pid: 60950
[2024-05-16 13:21:18 +0200] [60914] [ERROR] Worker (pid:60949) was sent SIGSEGV!
[2024-05-16 13:21:18 +0200] [60914] [DEBUG] 1 workers
[2024-05-16 13:21:18 +0200] [60951] [INFO] Booting worker with pid: 60951
[2024-05-16 13:21:18 +0200] [60914] [DEBUG] 2 workers
[2024-05-16 13:21:19 +0200] [60914] [ERROR] Worker (pid:60950) was sent SIGSEGV!
[2024-05-16 13:21:19 +0200] [60952] [INFO] Booting worker with pid: 60952
[2024-05-16 13:21:19 +0200] [60914] [ERROR] Worker (pid:60951) was sent SIGSEGV!
[2024-05-16 13:21:19 +0200] [60914] [DEBUG] 1 workers
[2024-05-16 13:21:19 +0200] [60953] [INFO] Booting worker with pid: 60953
[2024-05-16 13:21:19 +0200] [60914] [DEBUG] 2 workers
[2024-05-16 13:21:20 +0200] [60914] [ERROR] Worker (pid:60952) was sent SIGSEGV!
[2024-05-16 13:21:20 +0200] [60954] [INFO] Booting worker with pid: 60954
[2024-05-16 13:21:20 +0200] [60914] [ERROR] Worker (pid:60953) was sent SIGSEGV!
[2024-05-16 13:21:20 +0200] [60914] [DEBUG] 1 workers
[2024-05-16 13:21:20 +0200] [60955] [INFO] Booting worker with pid: 60955
[2024-05-16 13:21:20 +0200] [60914] [DEBUG] 2 workers
[2024-05-16 13:21:21 +0200] [60914] [ERROR] Worker (pid:60954) was sent SIGSEGV!
[2024-05-16 13:21:21 +0200] [60956] [INFO] Booting worker with pid: 60956
[2024-05-16 13:21:21 +0200] [60914] [ERROR] Worker (pid:60955) was sent SIGSEGV!
[2024-05-16 13:21:21 +0200] [60957] [INFO] Booting worker with pid: 60957
[2024-05-16 13:21:21 +0200] [60914] [ERROR] Worker (pid:60956) was sent SIGSEGV!
[2024-05-16 13:21:21 +0200] [60961] [INFO] Booting worker with pid: 60961
[2024-05-16 13:21:21 +0200] [60914] [ERROR] Worker (pid:60957) was sent SIGSEGV!
[2024-05-16 13:21:21 +0200] [60914] [DEBUG] 1 workers
[2024-05-16 13:21:21 +0200] [60962] [INFO] Booting worker with pid: 60962
[2024-05-16 13:21:21 +0200] [60914] [DEBUG] 2 workers
[2024-05-16 13:21:22 +0200] [60914] [ERROR] Worker (pid:60961) was sent SIGSEGV!
[2024-05-16 13:21:22 +0200] [60963] [INFO] Booting worker with pid: 60963
[2024-05-16 13:21:22 +0200] [60914] [ERROR] Worker (pid:60962) was sent SIGSEGV!

Using it QdrantClient with Flask directly works, no idea how to debug the problem... Any Ideas?

pajod commented 3 months ago

Unable to reproduce with just importing the qdrant_client module.. did you possibly import gRPC before the worker forks, e.g. by preloading your application, either with the gunicorn feature or importing into gunicorn configuation?

benoitc commented 1 month ago

no activity since awhile. closing feel free to create a new ticket if needed.