Custom client (GCP pubsub) resulting in segfault for 2+ users

[merten2000]

Prerequisites

• [X] I am using the latest version of Locust
• [X] I am reporting a bug, not asking a question

Description

I have created a GCP pubsub user, but when running with 2 or more users locust/python crashed with a segmentation fault error. Tested both on Ubuntu and RHEL resulting the same error. Monkey patched Python to ensure gevent is working properly with grpc. The error does not appear when using the GCP pubsub simulator Not 100% sure if this a locust bug or something else but would be great if we can find a solution and maybe add the GCP pubsub client

I have a coredump available if needed

error message:

Fatal Python error: Segmentation fault

Thread 0x00007f70057fa640 (most recent call first): File "/home/merten/.local/lib/python3.10/site-packages/gevent/_threading.py", line 36 in acquire_with_timeout File "/home/merten/.local/lib/python3.10/site-packages/gevent/_threading.py", line 86 in wait File "/home/merten/.local/lib/python3.10/site-packages/gevent/_threading.py", line 207 in get File "/home/merten/.local/lib/python3.10/site-packages/gevent/threadpool.py", line 195 in run

Thread 0x00007f7005ffb640 (most recent call first): File "/home/merten/.local/lib/python3.10/site-packages/gevent/_threading.py", line 36 in acquire_with_timeout File "/home/merten/.local/lib/python3.10/site-packages/gevent/_threading.py", line 86 in wait File "/home/merten/.local/lib/python3.10/site-packages/gevent/_threading.py", line 207 in get File "/home/merten/.local/lib/python3.10/site-packages/gevent/threadpool.py", line 195 in run

Thread 0x00007f70067fc640 (most recent call first): File "/home/merten/.local/lib/python3.10/site-packages/gevent/_threading.py", line 36 in acquire_with_timeout File "/home/merten/.local/lib/python3.10/site-packages/gevent/_threading.py", line 86 in wait File "/home/merten/.local/lib/python3.10/site-packages/gevent/_threading.py", line 207 in get File "/home/merten/.local/lib/python3.10/site-packages/gevent/threadpool.py", line 195 in run

Thread 0x00007f7006ffd640 (most recent call first): File "/home/merten/.local/lib/python3.10/site-packages/gevent/threadpool.py", line 173 in __run_task File "/home/merten/.local/lib/python3.10/site-packages/gevent/threadpool.py", line 206 in run

Current thread 0x00007f702155f1c0 (most recent call first): Garbage-collecting File "/home/merten/.local/lib/python3.10/site-packages/grpc/_channel.py", line 2046 in init File "/home/merten/.local/lib/python3.10/site-packages/grpc/init.py", line 2119 in secure_channel File "/home/merten/.local/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 386 in create_channel File "/home/merten/.local/lib/python3.10/site-packages/google/pubsub_v1/services/publisher/transports/grpc.py", line 222 in create_channel File "/home/merten/.local/lib/python3.10/site-packages/google/pubsub_v1/services/publisher/transports/grpc.py", line 165 in init File "/home/merten/.local/lib/python3.10/site-packages/google/pubsub_v1/services/publisher/client.py", line 712 in init File "/home/merten/.local/lib/python3.10/site-packages/google/cloud/pubsub_v1/publisher/client.py", line 139 in init File "/home/merten/pubsub2.py", line 49 in publish File "/home/merten/locustfile.py", line 17 in publish File "/home/merten/.local/lib/python3.10/site-packages/locust/user/task.py", line 489 in execute_task File "/home/merten/.local/lib/python3.10/site-packages/locust/user/task.py", line 372 in execute_next_task File "/home/merten/.local/lib/python3.10/site-packages/locust/user/task.py", line 339 in run File "/home/merten/.local/lib/python3.10/site-packages/locust/user/users.py", line 156 in run File "/home/merten/.local/lib/python3.10/site-packages/locust/user/users.py", line 188 in run_user

Extension modules: greenlet._greenlet, zope.interface._zope_interface_coptimizations, gevent.libev.corecext, gevent._gevent_c_greenlet_primitives, gevent._gevent_c_hub_local, gevent._gevent_c_waiter, gevent._gevent_c_hub_primitives, gevent._gevent_c_ident, gevent._gevent_cgreenlet, gevent._gevent_c_abstract_linkable, gevent._gevent_c_semaphore, gevent._gevent_clocal, gevent._gevent_cevent, gevent._gevent_cqueue, psutil._psutil_linux, psutil._psutil_posix, gevent._gevent_c_imap, msgpack._cmsgpack, zmq.backend.cython._zmq, _brotli, charset_normalizer.md, markupsafe._speedups, geventhttpclient._parser, grpc._cython.cygrpc, google._upb._message, _cffi_backend (total: 26) Segmentation fault

Command line

locust -f locustfile.py --u 2 --headless

Locustfile contents

## -- locustfile--
import grpc.experimental.gevent as grpc_gevent
grpc_gevent.init_gevent()

from gevent import monkey
monkey.patch_all()

from locust import HttpUser, task, run_single_user
from pubsub2 import GCloudSubUser
import os

class MyUser(GCloudSubUser):
    @task
    def publish(self):
        self.client.publish()

## --pubsub user -- 
import os
import time
from google.cloud import pubsub_v1
from locust import User

import faulthandler

faulthandler.enable()

class GCloudSubUser(User):
    abstract = True

    def __init__(self, environment) -> None:
        super().__init__(environment)
        self.client = GCloudSubClient(environment)

class GCloudSubClient:
    def __init__(self, environment):
        self.environment = environment
    def publish(self):
    #"""Publishes multiple message to a Pub/Sub topic."""
    # [START pubsub_quickstart_publisher]
    # [START pubsub_publish]
    #from google.cloud import pubsub_v1

    # TODO(developer)
    # project_id = "your-project-id"
    # topic_id = "your-topic-id"
        publisher = pubsub_v1.PublisherClient()
        project_id = os.getenv("PUBSUB_PROJECT_ID")
        topic_id = os.getenv("PUBSUB_TOPIC_ID")
    # The `topic_path` method creates a fully qualified identifier
    # in the form `projects/{project_id}/topics/{topic_id}`
        topic_path = publisher.topic_path(project_id, topic_id)
        data_str = f"SOME MESSAGE TO PUBLISH"
        # Data must be a bytestring
        data = data_str.encode("utf-8")
        # When you publish a message, the client returns a future.
        cb_time = time.time()
        future = publisher.publish(topic_path, data)
        try:
            print(future.result(2)) # future.result() is the message ID for the published message.
            self.environment.events.request.fire(
                request_type="pubsub",
                name="publish",
                response_time=(time.time() - cb_time ) * 1000,
                response_length=5, #to check how to get real value
                exception=None,
                )
             # fire successful publish event
        except Exception as e:
            print("Error publishing: " + str(e))
            self.environment.events.request.fire(
                request_type="pubsub",
                name="publish",
                response_time=(time.time() - cb_time ) * 1000,
                response_length=5, #to check how to get real value
                exception="Error publishing: " + str(e),
                )
             # fire unsuccessful publish event

Python version

Python 3.11.5 and Python 3.10.12

Locust version

locust 2.26.0

Operating system

RHEL 8.9 and Ubuntu 22.04

[cyberw]

Hi!

Have a look at the example in the docs and try to follow it closely, specifically: import locust as early as possible (before import grpc.experimental.gevent) and dont call monkey patching directly (importing locust will do it you for you)

I hope that helps. If not, try reproducing without locust and file an issue with pubsub.

[merten2000]

Hi, thanks for the feedback; I have made the changes, but still getting the same result

from locust import User, HttpUser, task, run_single_user
import grpc.experimental.gevent as grpc_gevent
grpc_gevent.init_gevent()
from google.cloud import pubsub_v1
import time
import os 

when trying without locust (simply spawning 2 pubsub clients) I dont get any errors .... just to be sure - can you provide the example? thanks!

[merten2000]

Issue fixed when moving the client creation in the init