Locust docker image run failure - dependencies issue

barakbs1 commented 4 days ago

Prerequisites

[X] I am using the latest version of Locust
[X] I am reporting a bug, not asking a question

Description

Hi everyone, I seem to have dependencies issue when running Locust's docker image. When trying to execute the test, I get the following error: Traceback (most recent call last): File "/opt/venv/bin/locust", line 8, in <module> sys.exit(main()) ^^^^^^ File "/opt/venv/lib/python3.11/site-packages/locust/main.py", line 75, in main docstring, _user_classes, shape_class = load_locustfile(_locustfile) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/locust/util/load_locustfile.py", line 58, in load_locustfile imported = source.load_module() ^^^^^^^^^^^^^^^^^^^^ File "<frozen importlib._bootstrap_external>", line 605, in _check_name_wrapper File "<frozen importlib._bootstrap_external>", line 1120, in load_module File "<frozen importlib._bootstrap_external>", line 945, in load_module File "<frozen importlib._bootstrap>", line 290, in _load_module_shim File "<frozen importlib._bootstrap>", line 721, in _load File "<frozen importlib._bootstrap>", line 690, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 940, in exec_module File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed File "/home/locust/tests/test_constant_high_load.py", line 4, in <module> from common.tasks import init_locust_user, register_device, \ File "/home/locust/common/tasks.py", line 12, in <module> import pandas as pd File "/opt/venv/lib/python3.11/site-packages/pandas/__init__.py", line 22, in <module> from pandas.compat import is_numpy_dev as _is_numpy_dev # pyright: ignore # noqa:F401 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/pandas/compat/__init__.py", line 25, in <module> from pandas.compat.numpy import ( File "/opt/venv/lib/python3.11/site-packages/pandas/compat/numpy/__init__.py", line 4, in <module> from pandas.util.version import Version File "/opt/venv/lib/python3.11/site-packages/pandas/util/__init__.py", line 2, in <module> from pandas.util._decorators import ( # noqa:F401 File "/opt/venv/lib/python3.11/site-packages/pandas/util/_decorators.py", line 14, in <module> from pandas._libs.properties import cache_readonly File "/opt/venv/lib/python3.11/site-packages/pandas/_libs/__init__.py", line 13, in <module> from pandas._libs.interval import Interval File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

Looking into the issue, i searched around a little bit for answers, and found this post: https://stackoverflow.com/questions/78650222/valueerror-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expec. It assures that there is a compatibility issue between pandas==2.1.1 and numpy==2.0.0. It States that downgrading to numpy==1.26.4 solved the issue.

Finally, looking into the dependencies which were downloaded by Locust I saw the following: Locust run which had the issue: Downloading numpy-2.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata Downloading pandas-2.0.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata Earlier Locust run which didn't had the issue: Downloading pandas-2.0.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata Downloading numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata

Command line

locust --headless -u 100 -i 50 -f tests/test.py

Locustfile contents

import os, time
from locust import HttpUser, run_single_user, events, SequentialTaskSet, task
from common.tasks import init_locust_user, register_device, \
    multiple_authentications, publish_pods_nodes_usage_metrics, set_server_disk_usage_value, \
    register_mekorot_device, set_mekorot_devices
from common.events import validate_constant_high_load
import locust_plugins

authentications_interval = int(os.getenv('AUTHENTICATION_INTERVAL'))

class SeqTaskSet(SequentialTaskSet):
    @task
    def on_start(self):
        # multiple_authentications(self, 50, authentications_interval)
        if self.client.device_num == "Station_45_Bareket_Station":
            publish_pods_nodes_usage_metrics(self)
        time.sleep(60 * 5)  # every 5 minutes

class AgentRequestor(HttpUser):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        init_locust_user(self)
        self.environment.actual_server_storage_usage_values_dict = {}
        set_server_disk_usage_value(self, 'start')
        self.environment.pods_cpu_usage = {}

    def on_start(self):
        # register_mekorot_device(self)
        pass

    tasks = [SeqTaskSet]

    events.test_start.add_listener(set_mekorot_devices)
    events.test_stopping.add_listener(validate_constant_high_load)

# for debugging purposes
if __name__ == "__main__":
    run_single_user(AgentRequestor)

Python version

3.11

Locust version

2.15.1

Operating system

Don't know

cyberw commented 4 days ago

Your locust version is very old. Please follow the instructions and upgrade to latest before filing bug reports. Also, this seems like a pandas/numpy issue, not Locust?

barakbs1 commented 3 days ago

Hi @cyberw, thank you for your quick answer. Please correct me if I'm wrong, I'm not managing the Locust version as it is set by the official docker image I'm pulling from https://hub.docker.com/r/locustio/locust. Looking into the docker-hub page, it appears this image was updated 2 days ago, so I guess this image is being maintained? Regarding the pandas/numpy, I'm not familiar with what is the correct way to deal with dependencies issues. Nevertheless, please correct me if I'm wrong, doesn't Locust sets itself with dependency versions to use?

cyberw commented 3 days ago

Latest release version of locust is 2.29.1, and that is what is in the docker image too. Perhaps your docker has cached an old version? Either do docker docker pull locustio/locust, or specify an explicit image tag

❯ docker run locustio/locust -V
locust 2.29.1 from /opt/venv/lib/python3.11/site-packages/locust (Python 3.11.9, OpenSSL 3.0.11)
❯ docker run locustio/locust:2.29.1 -V
locust 2.29.1 from /opt/venv/lib/python3.11/site-packages/locust (Python 3.11.9, OpenSSL 3.0.11)

Pandas/numpy is not really part of this project, talk to them if you have issues with that :) Dependencies are typically managed by building your own image based on locustio/locust. https://docs.locust.io/en/stable/running-in-docker.html#use-docker-image-as-a-base-image

barakbs1 commented 3 days ago

Thank you very much for the answer! I guess you are right and docker is using a cached version. I'll try to either clear the cache or run an explicit version. Thank you! I'll update back with the results.

locustio / locust