Problem installing dependency on unix platform

[BSouzaDock]

I had problems installing the package on AWS during deploy:

To my surprise, I tried to install it in a docker container and had the same error:

On Windows, I was able to use it normally.

Does anyone know how I can fix this problem?

[BSouzaDock]

Full log:

# pip install bonsai
Collecting bonsai
  Downloading bonsai-1.5.1.tar.gz (149 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 149.6/149.6 kB 1.0 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: bonsai
  Building wheel for bonsai (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for bonsai (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [69 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-311
      creating build/lib.linux-x86_64-cpython-311/bonsai
      copying src/bonsai/ldapurl.py -> build/lib.linux-x86_64-cpython-311/bonsai
      copying src/bonsai/utils.py -> build/lib.linux-x86_64-cpython-311/bonsai
      copying src/bonsai/ldapentry.py -> build/lib.linux-x86_64-cpython-311/bonsai
      copying src/bonsai/pool.py -> build/lib.linux-x86_64-cpython-311/bonsai
      copying src/bonsai/errors.py -> build/lib.linux-x86_64-cpython-311/bonsai
      copying src/bonsai/ldapdn.py -> build/lib.linux-x86_64-cpython-311/bonsai
      copying src/bonsai/ldapconnection.py -> build/lib.linux-x86_64-cpython-311/bonsai
      copying src/bonsai/ldif.py -> build/lib.linux-x86_64-cpython-311/bonsai
      copying src/bonsai/ldapvaluelist.py -> build/lib.linux-x86_64-cpython-311/bonsai
      copying src/bonsai/__init__.py -> build/lib.linux-x86_64-cpython-311/bonsai
      copying src/bonsai/ldapreference.py -> build/lib.linux-x86_64-cpython-311/bonsai
      copying src/bonsai/ldapclient.py -> build/lib.linux-x86_64-cpython-311/bonsai
      creating build/lib.linux-x86_64-cpython-311/bonsai/active_directory
      copying src/bonsai/active_directory/sid.py -> build/lib.linux-x86_64-cpython-311/bonsai/active_directory
      copying src/bonsai/active_directory/acl.py -> build/lib.linux-x86_64-cpython-311/bonsai/active_directory
      copying src/bonsai/active_directory/__init__.py -> build/lib.linux-x86_64-cpython-311/bonsai/active_directory
      creating build/lib.linux-x86_64-cpython-311/bonsai/asyncio
      copying src/bonsai/asyncio/aioconnection.py -> build/lib.linux-x86_64-cpython-311/bonsai/asyncio
      copying src/bonsai/asyncio/aiopool.py -> build/lib.linux-x86_64-cpython-311/bonsai/asyncio
      copying src/bonsai/asyncio/__init__.py -> build/lib.linux-x86_64-cpython-311/bonsai/asyncio
      creating build/lib.linux-x86_64-cpython-311/bonsai/gevent
      copying src/bonsai/gevent/geventconnection.py -> build/lib.linux-x86_64-cpython-311/bonsai/gevent
      copying src/bonsai/gevent/__init__.py -> build/lib.linux-x86_64-cpython-311/bonsai/gevent
      creating build/lib.linux-x86_64-cpython-311/bonsai/tornado
      copying src/bonsai/tornado/tornadoconnection.py -> build/lib.linux-x86_64-cpython-311/bonsai/tornado
      copying src/bonsai/tornado/__init__.py -> build/lib.linux-x86_64-cpython-311/bonsai/tornado
      creating build/lib.linux-x86_64-cpython-311/bonsai/trio
      copying src/bonsai/trio/trioconnection.py -> build/lib.linux-x86_64-cpython-311/bonsai/trio
      copying src/bonsai/trio/__init__.py -> build/lib.linux-x86_64-cpython-311/bonsai/trio
      running egg_info
      writing bonsai.egg-info/PKG-INFO
      writing dependency_links to bonsai.egg-info/dependency_links.txt
      writing requirements to bonsai.egg-info/requires.txt
      writing top-level names to bonsai.egg-info/top_level.txt
      reading manifest file 'bonsai.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      warning: no files found matching '*.css' under directory 'docs'
      warning: no previously-included files matching '*' found under directory 'docs/_build'
      adding license file 'LICENSE'
      writing manifest file 'bonsai.egg-info/SOURCES.txt'
      copying src/bonsai/py.typed -> build/lib.linux-x86_64-cpython-311/bonsai
      running build_ext
      creating /tmp/tmpkfom83qv/tmp
      creating /tmp/tmpkfom83qv/tmp/tmpkfom83qv
      gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/include/sasl -I/usr/include -I/usr/local/include/python3.11 -c /tmp/tmpkfom83qv/test_krb5.c -o /tmp/tmpkfom83qv/tmp/tmpkfom83qv/test_krb5.o
      gcc /tmp/tmpkfom83qv/tmp/tmpkfom83qv/test_krb5.o -L/usr/lib -L/usr/local/lib -L/usr/lib -L/usr/local/lib -lkrb5 -lgssapi -o /tmp/tmpkfom83qv/test_krb5
      creating /tmp/tmp7y4wx2ft/tmp
      creating /tmp/tmp7y4wx2ft/tmp/tmp7y4wx2ft
      gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/include/sasl -I/usr/include -I/usr/local/include/python3.11 -c /tmp/tmp7y4wx2ft/test_krb5.c -o /tmp/tmp7y4wx2ft/tmp/tmp7y4wx2ft/test_krb5.o
      gcc /tmp/tmp7y4wx2ft/tmp/tmp7y4wx2ft/test_krb5.o -L/usr/lib -L/usr/local/lib -L/usr/lib -L/usr/local/lib -lkrb5 -lgssapi_krb5 -o /tmp/tmp7y4wx2ft/test_krb5
      building 'bonsai._bonsai' extension
      creating build/temp.linux-x86_64-cpython-311
      creating build/temp.linux-x86_64-cpython-311/src
      creating build/temp.linux-x86_64-cpython-311/src/_bonsai
      gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DHAVE_KRB5=1 -I/usr/include/sasl -I/usr/include -I/usr/local/include/python3.11 -c src/_bonsai/bonsaimodule.c -o build/temp.linux-x86_64-cpython-311/src/_bonsai/bonsaimodule.o
      In file included from src/_bonsai/utils.h:8,
                       from src/_bonsai/ldapconnection.h:9,
                       from src/_bonsai/bonsaimodule.c:5:
      src/_bonsai/ldap-xplat.h:23:10: fatal error: ldap.h: No such file or directory
         23 | #include <ldap.h>
            |          ^~~~~~~~
      compilation terminated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for bonsai
Failed to build bonsai
ERROR: Could not build wheels for bonsai, which is required to install pyproject.toml-based projects
#   

[noirello]

Sure, there aren't any wheels available on PyPI for Linux, so you have to compile the module by your own. The required packages are listed in the docs.

[BSouzaDock]

Hello @noirello , Thanks for the response.

I need help to be able to run this module on an AWS Lambda. I need to upload my project with this module and the binaries together, since I can't do pip install or install Linux packages within the lambda.

I installed it inside a Docker container and used the python3 setup.py build command.

After this I copied the bonsai folder "build/lib.linux-x86_64-3.8/bonsai" to the root of my project. Inside the bonsai folder, there is the binary "_bonsai.cpython-38-x86_64-linux-gnu.so".

But when I start my project on lambda, I get the following error: [ERROR] Runtime.ImportModuleError: Unable to import module 'init': No module named 'bonsai._bonsai'

Do you know what I can do to solve the problem?

I had something similar when using the paramiko library for SFTP use, but copying the binary from that library worked. In the bonsai case, it does not seem to find this file, even though it is in the folder.

[noirello]

That's a tricky one, and my AWS Lambda experience is negligible. I think you have to copy the bonsai source directory to the same level of your project. Something like this:

.
├── bonsai
│   ├── active_directory
│   ├── asnycio
│   ├── gevent
│   ├── tornado
│   ├── trio
│   ├── __init__.py
│   ├── _bonsai.cpython-311-x86_64-linux-gnu.so
│   ├── ldapclient.py
│   ├── ...
│   └── ...
├── other-python-package
│   └── ...
├── your_lambda_thing.py
└── README.md

[BSouzaDock]

In fact it is like this. The problem occurs when there is an attempt to import the existing library in the .so file, then I receive the following error: [ERROR] Runtime.ImportModuleError: Unable to import module 'init': No module named 'bonsai._bonsai'

I'm researching how to read these .so files through lambda. I created this topic: https://stackoverflow.com/questions/77211100/using-so-file-libraries-in-aws-lambda

[noirello]

The bunch of native dependencies of the module that makes this (or providing a linux wheel on the first place) tricky.

First I'd have recommended using a custom docker image with a multi stage build for your Lambda function, but looking at the official docker image (public.ecr.aws/lambda/python:3.11), you might've every necessary library provided to make this work.

Did you use this image to build the module? It's vital to have the same libraries with the same path on the runtime environment just like on the build environment.

[BSouzaDock]

I use lambda with version 3.9 of python.

The issue is that I'm not able to get the lambda to read the .so file. I tried creating a layer as I saw another similar case but it didn't work either

[noirello]

The attached image of your project on StackOverflow shows that the module build for 3.8. If you use that .so file on 3.9 Lambda runtime, that can lead to the same error message that you've got (or at least it did when I tried to use the module that was built using the python:3.11 docker container with python:3.10).

[BSouzaDock]

I understood what you said, it made sense.

After changing the project to python 3.8, now the file is recognized, but it is asking for another file: Unable to import module 'init': libldap-2.5.so.0: cannot open shared object file: No such file or directory

I didn't find this file inside the docker build folder. I believe it should be located in another location. Can you tell me where?

[noirello]

Are you sure you used the right docker image to build the module? These AWS Lambda images all have older libldap installed (2.4.* something).

Here's how I'd do:

$ docker run --name build --entrypoint bash -ti public.ecr.aws/lambda/python:3.8
bash-4.2# yum install -y openldap-devel gcc
bash-4.2# pip install bonsai
bash-4.2# ldd /var/lang/lib/python3.8/site-packages/bonsai/_bonsai.cpython-38-x86_64-linux-gnu.so
    linux-vdso.so.1 (0x00007ffcf9ead000)
    libldap.2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007f549e3d6000)
    liblber.2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007f549e1c7000)
    ...

The ldd shows which shared libraries are linked to this one. When you run this so file on AWS Lambda, it'd expect to have these libraries with same version on the same path. Otherwise you'll get the mentioned ...cannot open shared object... error message.

To get the artefacts, open another console and copy them from the running container:

$ docker cp build:/var/lang/lib/python3.8/site-packages/bonsai .

Like I said before, I think (based on the container image) the Lambda runtime has every necessary library (libldap, liblber, libsasl2, libssl, etc.), so you don't have to copy these to your zip file. But if I'm wrong, and something is missing, you can still create a custom container image and used that instead of a zip file (if you're still want to use bonsai for your project after all this cumbersome activities).

[BSouzaDock]

In fact, during the creation of the lambda I only chose the runtime that already uses a standard image.

There is the option to upload a custom image, but I didn't try it.

I even imported the .so files 1 by 1 that were being requested, but I ended up stopping at this: Unable to import module 'init': /lib64/libc.so.6: version GLIBC_2.34' not found (required by /var/task/bonsai/_bonsai.cpython-38-x86_64-linux-gnu.so)

I even created the lib64 folder with the requested file but it didn't work. I believe that this libc.so.6 file in the default image does not have this GLIBC_2.34 version as stated in the message.

I'll try to upload it via a custom docker image.

[noirello]

That's odd. The Amazon's container quite old, the so file built on that should been linked to glibc 2.26 or an earlier version.

[BSouzaDock]

I managed to do it locally by downloading the docker python3.8 model from aws ecr and installing linux packages.

In the dockerfile, I would like to configure all the commands, including those for installing the bonsai, but I am not able to.

If I leave a RUN bonsai/setup.py build I get an error because the setup.py file is opening local files, and when I do a with open("README.rst") as file: for example, it ends up searching in the root folder and not in the local folder, causing a file not found error.

So after the build, I need to do these commands manually: _#docker run -it ad-scripts /bin/bash

cd bonsai

python3 setup.py build

python3 setup.py install_

Can you tell me if there is a way to leave these commands in the dockerfile without returning an error?

[noirello]

You can use the following command: RUN cd bonsai && setup.py build && setup.py install

[BSouzaDock]

I managed using a lambda with a custom docker image. I'll leave the dockerfile I used here, in case anyone has the same problem:

# Define custom function directory
ARG FUNCTION_DIR="/function"

FROM public.ecr.aws/docker/library/python:3.8.15

# Include global arg in this stage of the build
ARG FUNCTION_DIR

# Intall bonsai linux dependencies
RUN apt-get update &&  \
    apt-get install -y \
    gcc \
    libldap2-dev \
    libsasl2-dev

# Copy function code
RUN mkdir -p ${FUNCTION_DIR}
COPY . ${FUNCTION_DIR}

# Install the function's dependencies
RUN pip install awslambdaric
COPY util/requirements.txt ./
RUN pip install \
    --target ${FUNCTION_DIR} \
    -r requirements.txt

# Install bonsai
RUN cd ${FUNCTION_DIR}/bonsai && python3 setup.py build && python3 setup.py install

# Set working directory to function root directory
WORKDIR ${FUNCTION_DIR}

# Set runtime interface client as default command for the container runtime
ENTRYPOINT [ "/usr/local/bin/python", "-m", "awslambdaric" ]

# Pass the name of the function handler as an argument to the runtime
CMD [ "init.main" ]