Importing Python modules and volume binding

[NickDarvey]

Hi,

I'm having trouble importing local Python modules when using volume binding.

I understand the Python library I'm trying to use in my example is unrelated to lambci/lambda, but I am opening this here because it does work when I COPY the files into the Docker image, but not when I use volume binding.

(I'm not so familiar with this ecosystem, so forgive my use incorrect terminology or misunderstanding.)

Repro

Download from here, or

1. Create a test folder

2. Create a file, lambda.py

   import os
   import sys
   
   def handler(event, context):
     path = os.path.dirname(os.path.realpath(__file__))
     if os.path.join(path, 'lib/python2.7/site-packages') not in sys.path:
         sys.path.insert(0, os.path.join(path, 'lib/python2.7/site-packages'))
   
     from osgeo import gdal
   
     print('Hello')

3. Download and extract this folder so that you have this structure:

   testfolder
   - lib
   - lambda.py

Test 1

1. Create a file, Dockerfile

   FROM lambci/lambda:python2.7
   WORKDIR /var/task
   COPY . .

2. Build the image, docker build -t plz .
3. Run test 1, docker run --rm plz lambda.handler

Test 2

1. Run test 2, docker run --rm -v /c/Users/me/source/repos/my-test-lambda:/var/task lambci/lambda:python2.7 lambda.handler

Expected

Test 1:
hello

Test 2:
hello

Actual

Test 1:
hello

Test 2:
/var/task/lib/python2.7/site-packages/osgeo/_gdal.so: cannot open shared object file: No such file or directory: ImportError
Traceback (most recent call last):
  File "/var/task/lambda.py", line 9, in handler
    from osgeo import gdal
  File "/var/task/lib/python2.7/site-packages/osgeo/__init__.py", line 23, in <module>
    _gdal = swig_import_helper()
  File "/var/task/lib/python2.7/site-packages/osgeo/__init__.py", line 19, in swig_import_helper
    _mod = imp.load_module('_gdal', fp, pathname, description)
ImportError: /var/task/lib/python2.7/site-packages/osgeo/_gdal.so: cannot open shared object file: No such file or directory

System Info

• OS version: Windows 10 Pro 1803 17134.523
• Docker version: 18.06.1-ce-win73 (19507)
• lambci image version lambci/lambda:python2.7 9e29188f3f15

What I've tried

• Looking for permission issues by accessing the files via Python, I can read the files fine. e.g. os.listdir, open() (So I don't think it's the same as #114.)
• Looking for issues resolving dependencies with ldd, all of the depdencies seem to resolve and refer to the same libraries. Comparing the working: docker run --rm --entrypoint ldd fusv /var/task/lib/python2.7/site-packages/osgeo/_gdal.so with the not working: docker run --rm -v /c/Users/me/source/repos/my-test-lambda:/var/task --entrypoint ldd lambci/lambda:python2.7 /var/task/lib/python2.7/site-packages/osgeo/_gdal.so
• Looking for references to the library, gdal, with LD_DEBUG=all on.
• Setting the LD_LIBRARY_PATH docker run --rm -e "LD_LIBRARY_PATH=/var/task/lib/python2.7/site-packages:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib" -v /c/Users/me/source/repos/my-test-lambda:/var/task lambci/lambda:python2.7 lambda.handler

[mhart]

Might be easiest to open an interactive Docker session and see what's going on.

You can do that with:

docker run --rm -it -v /c/Users/me/source/repos/my-test-lambda:/var/task --entrypoint bash lambci/lambda:python2.7

Then you can have a look around at the filesystem and permissions – and compare them with the COPY situation

[mhart]

Any luck with the above?

[NickDarvey]

Unfortunately I've had to move on from this. If I recall correctly, the permissions I could see with ls -la looked the same with each method. 😅 I'm sorry I can't clarify further We found python-lambda-local ended up doing the trick, though we'd rather be using this to accurately emulate the environment. Hopefully I can reopen this sometime in the future!

[iwpnd]

I know this is a closed issue but one thing that is worth a try is to check your docker settings/shared drives. As you said. When you copy the files into the container it works. If you mount it, it doesn't work because docker does not have the authority to access your drive by default afaik. I faced the problem when I changed my OS login credentials and docker no longer had access to my drives.