cannot import name 'RunOptions' when executing in AWS lambda

[ahmed-shariff]

Describe the bug I get an error cannot import name 'RunOptions' when I am trying to import onnxruntime in an aws lambda function.

Full stack trace:

Traceback (most recent call last):
File "/var/task/lambda_function.py", line 8, in <module>
  import onnxruntime
File "/var/task/onnxruntime/__init__.py", line 21, in <module>
  from onnxruntime.capi._pybind_state import RunOptions, SessionOptions, get_device, NodeArg, ModelMetadata
ImportError: cannot import name 'RunOptions'

System information

• OS Platform and Distribution: amazon linux 2018.03
• ONNX Runtime installed from (source or binary): source
• ONNX Runtime version: 0.4.0
• Python version: 3.6

To Reproduce execute the following script in a ec2 instance with amazon linux 2018.03/in a docker container with amazon linux 2018.03

dev_install () {
    sudo yum -y update
    sudo yum -y upgrade
    sudo yum install -y \
         wget \
         gcc \
         gcc-c++ \
         python36-devel \
         python36-virtualenv \
         python36-pip \
         findutils \
         zlib-devel \
         zip
}

pip_install () {
    rm -r env
    virtualenv env -p python3.6
    source env/bin/activate

    pip install --upgrade pip wheel
    pip install numpy -U
    pip install onnxruntime -U
    pip install imageio -U
    deactivate
}

gather_pack () {
    # packing
    source env/bin/activate

    rm -r lambdapack
    mkdir lambdapack
    cd lambdapack

    cp -R $VIRTUAL_ENV/lib/python3.6/site-packages/* .
    cp -R $VIRTUAL_ENV/lib64/python3.6/site-packages/* .

    mkdir lib
    cp /usr/lib64/libstdc++.so.6 lib/
    cp /usr/lib64/libgomp.so.1 lib/
    cp /lib64/libgcc_s.so.1 lib/
    cd ../
    cp lambda_function.py lambdapack/lambda_function.py
    echo "original size $(du -sh lambdapack | cut -f1)"

    # cleaning libs
    cd lambdapack
    rm -r external
    find . -type d -name "tests" -exec rm -rf {} +

    # cleaning
    find -name "*.so" | xargs strip
    find -name "*.so.*" | xargs strip
    # find . -name tests -type d -print0|xargs -0 rm -r --
    # find . -name test -type d -print0|xargs -0 rm -r --    
    rm -r pip
    rm -r pip-*
    rm -r wheel
    rm -r wheel-*
    rm easy_install.py
    find . -name \*.pyc -delete
    # find . -name \*.txt -delete
    echo "stripped size $(du -sh ../lambdapack | cut -f1)"

    # compressing
    zip -FS -r9 ../function.zip * > /dev/null
    echo "compressed size $(du -sh ../function.zip | cut -f1)"
}

main () {
    dev_install

    pip_install

    gather_pack
}

main

Content of lambda_function.py:

import json
import onnxruntime

def lambda_handler(event, context):
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

Following running above script, upload the function.zip file to aws lambda and execute the function Expected behavior The lambda function executes normally.

I am bundling the libstdc++.so.6, libgomp.so.1 and libgcc_s.so.1 with the function. Is there any other library I should include here?

[pranavsharma]

I can't reproduce this. See below.

pranav@pranav-HP-Z440-Workstation:~$ virtualenv env2 -p python3.6
Running virtualenv with interpreter /usr/bin/python3.6
Using base prefix '/usr'
New python executable in /home/pranav/env2/bin/python3.6
Also creating executable in /home/pranav/env2/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
pranav@pranav-HP-Z440-Workstation:~$ source env2/bin/activate
(env2) pranav@pranav-HP-Z440-Workstation:~$ pip install numpy -U
Collecting numpy
  Using cached https://files.pythonhosted.org/packages/87/2d/e4656149cbadd3a8a0369fcd1a9c7d61cc7b87b3903b85389c70c989a696/numpy-1.16.4-cp36-cp36m-manylinux1_x86_64.whl
Installing collected packages: numpy
Successfully installed numpy-1.16.4
(env2) pranav@pranav-HP-Z440-Workstation:~$ pip install onnxruntime -U
Collecting onnxruntime
  Using cached https://files.pythonhosted.org/packages/fa/e1/1548ef61a5c3b583c7ce777c9f38e30b7fdeda309ddcc886b5883cba5771/onnxruntime-0.4.0-cp36-cp36m-manylinux1_x86_64.whl
Installing collected packages: onnxruntime
Successfully installed onnxruntime-0.4.0
(env2) pranav@pranav-HP-Z440-Workstation:~$ python
Python 3.6.8 (default, Dec 24 2018, 19:24:27)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import onnxruntime
>>>

[jywu-msft]

the onnxruntime python whl was built on ubuntu 16.04 (gcc 5.4.0) it looks like amazon linux dist is closest to rhel/centos 7 which isn't an officially supported due to its default compiler being gcc 4.8 see: https://github.com/microsoft/onnxruntime/blob/master/BUILD.md

you might be able to build from source, but you'll need to use gcc >= 5.0

[pranavsharma]

@ahmed-shariff can this be closed based on the above comment?

[ahmed-shariff]

It still doesn't resolve the issue.

I built onnx from source using gcc72 and cmake-3.14.4 for python3.6 using a docker image of amazonlinux:1 Since the size limit of a function in lambda is 256MB, and the size of onnxruntime_pybind11_state.so alone was 263MB, i ran strip on it which reduced it's size to 6MB (which was interesting).

I uploaded the function with the onnxruntime and still am getting the same error. Is there any other way in which I can debug this?

[pranavsharma]

Can you run ldd onnxruntime_pybind11_state.so and paste the output here? This will show if you're missing any dependent libraries. Having said that we only support building on Ubuntu 16.04 with gcc 5.x. See build instructions here: https://github.com/microsoft/onnxruntime/blob/master/BUILD.md.

[ahmed-shariff]

When I run ldd in the docker image:

        linux-vdso.so.1 =>  (0x00007fff113bc000)
    libpython3.6m.so.1.0 => /usr/lib64/libpython3.6m.so.1.0 (0x00007f0b631a7000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f0b62fa3000)
    libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f0b62c1e000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f0b6291c000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f0b62706000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f0b624ea000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f0b6211d000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f0b63ed7000)
    libutil.so.1 => /lib64/libutil.so.1 (0x00007f0b61f1a000)

I also uploaded the onnxruntime with to the lambda function and ran ldd through python's subprocess. This is what I initially got:

/var/task/onnxruntime/capi/onnxruntime_pybind11_state.so: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /var/task/onnxruntime/capi/onnxruntime_pybind11_state.so)
/var/task/onnxruntime/capi/onnxruntime_pybind11_state.so: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /var/task/onnxruntime/capi/onnxruntime_pybind11_state.so)
/var/task/onnxruntime/capi/onnxruntime_pybind11_state.so: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /var/task/onnxruntime/capi/onnxruntime_pybind11_state.so)
/var/task/onnxruntime/capi/onnxruntime_pybind11_state.so: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.11' not found (required by /var/task/onnxruntime/capi/onnxruntime_pybind11_state.so)
/var/task/onnxruntime/capi/onnxruntime_pybind11_state.so: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /var/task/onnxruntime/capi/onnxruntime_pybind11_state.so)
/var/task/onnxruntime/capi/onnxruntime_pybind11_state.so: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by /var/task/onnxruntime/capi/onnxruntime_pybind11_state.so)
    linux-vdso.so.1 =>  (0x00007ffd145b3000)
    libpython3.6m.so.1.0 => /var/lang/lib/libpython3.6m.so.1.0 (0x00007fd3a0c15000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fd3a0a11000)
    libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007fd3a070c000)
    libm.so.6 => /lib64/libm.so.6 (0x00007fd3a040a000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fd3a01f4000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fd39ffd8000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fd39fc0b000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fd3a176d000)
    libutil.so.1 => /lib64/libutil.so.1 (0x00007fd39fa08000)
    librt.so.1 => /lib64/librt.so.1 (0x00007fd39f800000)

The default LD_LIBRARY_PATH on lambda was:

/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task/lib:/var/task:/opt/lib

Since the libstdc++.so.6 I packaged with the function is unziped to /var/task/lib/, I manually set LD_LIBRARY_PATH to:

/var/task/lib:/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/opt/lib

After which, when I run ldd through pythons subprocess in lambda I get the following:

        linux-vdso.so.1 =>  (0x00007ffccb52e000)
    libpython3.6m.so.1.0 => /var/lang/lib/libpython3.6m.so.1.0 (0x00007f74d16f3000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f74d14ef000)
    libstdc++.so.6 => /var/task/lib/libstdc++.so.6 (0x00007f74d116a000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f74d0e68000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f74d0c52000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f74d0a36000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f74d0669000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f74d224b000)
    libutil.so.1 => /lib64/libutil.so.1 (0x00007f74d0466000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f74d025e000)

So I tried importing onnxruntime after that and got this for the lambda function:

/var/task/onnxruntime/capi/_pybind_state.py:12: UserWarning: Cannot load onnxruntime.capi. Error: '/usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /var/task/onnxruntime/capi/onnxruntime_pybind11_state.so)'
  warnings.warn("Cannot load onnxruntime.capi. Error: '{0}'".format(str(e)))
cannot import name 'RunOptions': ImportError
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 15, in lambda_handler
    import onnxruntime
  File "/var/task/onnxruntime/__init__.py", line 21, in <module>
    from onnxruntime.capi._pybind_state import RunOptions, SessionOptions, get_device, NodeArg, ModelMetadata
ImportError: cannot import name 'RunOptions'

[mayeut]

Probably related to #1001

[snnn]

/var/task/onnxruntime/capi/_pybind_state.py:12: UserWarning: Cannot load onnxruntime.capi. Error: '/usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /var/task/onnxruntime/capi/onnxruntime_pybind11_state.so)'

It says, you are still using the old C++ runtime for gcc 4.x .

[pranavsharma]

As mentioned before, we don't support this OS/compiler version combination. Please see system requirements.

[ahmed-shariff]

@snnn yes, it seems so. Still trying to figure out how to load the library before running the lambda function script. Will update here if I found a solution.

[ahmed-shariff]

So I did find a solution. I had to load the libstdc++.so.6 file I bundle with the function/layer needs to be manually loaded using ctypes.cdll.LoadLibrary before importing onnxruntime.

Example function:

import json
import os
import boto3
import numpy as np
from ctypes import cdll
# I packaged onnxruntime as a layer. 
# Hence the '/opt/lib/' directory is already in LD_LIBRARY_PATH  
cdll.LoadLibrary('/opt/lib/libstdc++.so.6')
import onnxruntime

bucket_name = os.environ['BUCKET_NAME']
onnx_file_location = os.environ['ONNX_FILE']
s3 = boto3.resource('s3')
onnx_file = s3.Object(bucket_name, onnx_file_location).download_file('/tmp/model.onnx')
model = onnxruntime.InferenceSession("/tmp/model.onnx")
def lambda_handler(event, context):
    w = np.ndarray((4, 3, 224, 224)).astype(np.float32())
    w = {model.get_inputs()[0].name: w}
    out = model.run(None, w)
    return {
        'statusCode': 200,
        'body': json.dumps('successfully done!')
    }

[ahmed-shariff]

Update after 2 month, it seems the problem has been fixed. I am able to build lambda functions using pip and don't seem to need the fix with cdll I have mentioned above. Cheers guys.