search

cloud-bulldozer / benchmark-wrapper

Python Library to run benchmarks
https://benchmark-wrapper.readthedocs.io
Apache License 2.0
19 stars 56 forks source link

uperf Docker Image creation fails with dependency issues on ppc64le #228

Closed piyushgupta1551 closed 3 years ago

piyushgupta1551 commented 3 years ago

Hi Team, I am trying to build the uperf image from benchmark-wrapper repository for ppc64le but it keeps failing at the last step - RUN pip3 install -e /opt/snafu/ with errors pointing to missing packages/dependencies as below - 

ModuleNotFoundError: No module named 'numpy'
ModuleNotFoundError: No module named 'cython'
RuntimeError: Broken toolchain: cannot link a simple C program
gcc: error trying to exec 'cc1plus': execvp: No such file or directory

Adding these 'missing' packages is not straight as a new one shows up with every image build failure. The expectation was that all dependencies mentioned in setup.cfg would be sufficient but that isn't the case ppc64le. Main issue seems to be scipy & numpy packages.

As per a workaround suggested on perf-scale slack, I tried removing scipy & numpy in setup.cfg but it didn't help and the build failed with error as numpy is again required by prometheus_api_client - 

Collecting numpy (from prometheus_api_client->snafu==0.0.1)
  Downloading https://files.pythonhosted.org/packages/c5/63/a48648ebc57711348420670bb074998f79828291f68aebfff1642be212ec/numpy-1.19.4.zip (7.3MB)
    100% |████████████████████████████████| 7.3MB 175kB/s 
Collecting pandas>=1.0.0 (from prometheus_api_client->snafu==0.0.1)
  Downloading https://files.pythonhosted.org/packages/09/39/fb93ed98962d032963418cd1ea5927b9e11c4c80cb1e0b45dea769d8f9a5/pandas-1.1.4.tar.gz (5.2MB)
    100% |████████████████████████████████| 5.2MB 243kB/s 
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 346, in get_provider
        module = sys.modules[moduleOrReq]
    KeyError: 'numpy'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-6u08iba0/pandas/setup.py", line 792, in <module>
        setup_package()
      File "/tmp/pip-build-6u08iba0/pandas/setup.py", line 762, in setup_package
        ext_modules=maybe_cythonize(extensions, compiler_directives=directives),
      File "/tmp/pip-build-6u08iba0/pandas/setup.py", line 521, in maybe_cythonize
        numpy_incl = pkg_resources.resource_filename("numpy", "core/include")
      File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1132, in resource_filename
        return get_provider(package_or_requirement).get_resource_filename(
      File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 348, in get_provider
        __import__(moduleOrReq)
    ModuleNotFoundError: No module named 'numpy'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-6u08iba0/pandas/
Error: error building at STEP "RUN pip3 install -e /opt/snafu/": error while running runtime: exit status 1

Please help suggest a fix or a workaround for the same.

piyushgupta1551 commented 3 years ago

Update - I did manage to eventually build the uperf image on ppc64le. What I did additionally to make it happen -

Since this requirement is specific to ppc64le systems only, can we probably add this info as a small Note in the README doc somewhere for future reference?

jtaleric commented 3 years ago

Update - I did manage to eventually build the uperf image on ppc64le. What I did additionally to make it happen -

  • Install a couple of rpm packages before scipy and numpy from the Dockerfile - atlas-devel libffi-devel gcc gcc-c++ gcc-gfortran libffi-devel make openssl-devel (added them in line 3 of the Dockerfiles) - this seems to be a known requirement for ppc64le in general for both scipy & numpy
  • I had to add this to the Dockerfile as the same somehow doesn't work by adding as dependency in setup.cfg - RUN python3 -m pip install --upgrade cython numpy
  • I had to add a couple of python packages to setup.cfg -importlib_metadata & urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1(version clash and eventual failure while running workload if version specifics not added)

Since this requirement is specific to ppc64le systems only, can we probably add this info as a small Note in the README doc somewhere for future reference?

Interesting on the ppc64 specific needs. I wonder if we can have a uperf Dockerfile that is ppc64 specific?

Would you like to do a PR against benchmark-wrapper to include this ppc64. And if you have a container image, maybe you can share that, so I can push to Quay?

piyushgupta1551 commented 3 years ago

@jtaleric , yes I will raise the PR for ppc64le Dockerfiles. In my opinion all dockerfiles except iperf will need these additions to be ppc64le compliant. So far I've created 4 images for ppc64le -

How do you want me to add in the python packages that I added in setup.cfg? Should I add them in the Dockerfiles directly? Adding them in setup.cfg might create a conflict as they are not really required for x86.

piyushgupta1551 commented 3 years ago

Hi @jtaleric @vasukulkarni Below are the images that could be created for benchmark-operator and benchmark-wrapper workloads for ppc64le arch - 

quay.io/piyushgupta1551/benchmark-operator:ppc64le
quay.io/piyushgupta1551/iperf3-m:ppc64le
quay.io/piyushgupta1551/fs-drift-m:ppc64le
quay.io/piyushgupta1551/fio-m:ppc64le
quay.io/piyushgupta1551/ycsb-m:ppc64le
quay.io/piyushgupta1551/smallfile-m:ppc64le
quay.io/piyushgupta1551/upgrade-m:ppc64le
quay.io/piyushgupta1551/scale-openshift-m:ppc64le
quay.io/piyushgupta1551/uperf-m:ppc64le

The remaining (oslat, hammerdb, sysbench, pgbench, cyclictest, vegeta) could not be created as dependent rpms, installers etc were not available for the required distros for ppc64le arch.

piyushgupta1551 commented 3 years ago

Open Issues for ppc64le workload images -

FYI @vasukulkarni