handler unable to find python module from handler's requirements.txt

[Jeff-Lowrey]

When I add a non-default python to my handler requirements.txt, my function code is unable to import them.

Git repo to replicate is at https://github.com/Jeff-Lowrey/openfaas-import-test

When deployed to openshift, the container does not launch, and the pod stays in "CrashLoopBackOff" state.

[Jeff-Lowrey]

The pod log shows The log of the pod shows 2020/01/06 16:02:16 Started logging stderr from function. 2020/01/06 16:02:16 Started logging stdout from function. Forking - python [index.py] 2020/01/06 16:02:16 OperationalMode: http 2020/01/06 16:02:16 Timeouts: read: 10s, write: 10s hard: 10s. 2020/01/06 16:02:16 Listening on port: 8080 2020/01/06 16:02:16 Writing lock-file to: /tmp/.lock 2020/01/06 16:02:16 Metrics listening on port: 8081 2020/01/06 16:02:16 stderr: Traceback (most recent call last): 2020/01/06 16:02:16 stderr: File "index.py", line 6, in 2020/01/06 16:02:16 stderr: from function import handler 2020/01/06 16:02:16 stderr: File "/home/app/function/handler.py", line 1, in 2020/01/06 16:02:16 stderr: import blackhole 2020/01/06 16:02:16 stderr: ModuleNotFoundError: No module named 'blackhole' 2020/01/06 16:02:16 Forked function has terminated: exit status 1

[Jeff-Lowrey]

The faas-cli build shows faas-cli build -f ./import-test.yml [0] > Building import-test. Clearing temporary build folder: ./build/import-test/ Preparing ./import-test/ ./build/import-test//function Building: jefflowrey/general-repo:import-test with python3-http template. Please wait.. Sending build context to Docker daemon 10.24kB Step 1/32 : FROM openfaas/of-watchdog:0.7.2 as watchdog 0.7.2: Pulling from openfaas/of-watchdog 4b59300cc38b: Pull complete Digest: sha256:a0103df565c756ff71cc9c814b05b6f7f371534329e6401e6412ba7632b9c19b Status: Downloaded newer image for openfaas/of-watchdog:0.7.2 ---> bea76c093965 Step 2/32 : FROM python:3.7-alpine 3.7-alpine: Pulling from library/python e6b0cf9c0882: Pull complete da0e9bf0cc60: Pull complete b9340422295e: Pull complete 3c651fc490ae: Pull complete aa5f53becb54: Pull complete Digest: sha256:860856fce0f6f17e48b1a9ca89ec8ac99293af6bbc73e4e46a1ebce86daf122b Status: Downloaded newer image for python:3.7-alpine ---> 16a54299f91e Step 3/32 : COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog ---> 2d45fe430a68 Step 4/32 : RUN chmod +x /usr/bin/fwatchdog ---> Running in 514d13a31d6f Removing intermediate container 514d13a31d6f ---> d38827d1f126 Step 5/32 : ARG ADDITIONAL_PACKAGE ---> Running in ba27156e8320 Removing intermediate container ba27156e8320 ---> ab7bbb101aeb Step 6/32 : RUN apk --no-cache add ${ADDITIONAL_PACKAGE} ---> Running in 9ac3d13d2928 fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz OK: 18 MiB in 35 packages Removing intermediate container 9ac3d13d2928 ---> c8aa077a7040 Step 7/32 : RUN addgroup -S app && adduser app -S -G app ---> Running in 5edbb34dd990 Removing intermediate container 5edbb34dd990 ---> 9d0565a1a8ef Step 8/32 : RUN chown app /home/app ---> Running in 6c63cf4981c3 Removing intermediate container 6c63cf4981c3 ---> 6ddb03e8be0a Step 9/32 : USER app ---> Running in ed47a597490f Removing intermediate container ed47a597490f ---> 772bc347c4fa Step 10/32 : ENV PATH=$PATH:/home/app/.local/bin ---> Running in 112fd4a66103 Removing intermediate container 112fd4a66103 ---> c91f4f73dab2 Step 11/32 : WORKDIR /home/app/ ---> Running in 8d81469e6ace Removing intermediate container 8d81469e6ace ---> 04d12b526edf Step 12/32 : COPY index.py . ---> d9a8a73378e2 Step 13/32 : COPY requirements.txt . ---> b15aba87c404 Step 14/32 : USER root ---> Running in a16625901c47 Removing intermediate container a16625901c47 ---> b832c45524b4 Step 15/32 : RUN pip install -r requirements.txt ---> Running in 0d27f5967721 Collecting flask Downloading https://files.pythonhosted.org/packages/9b/93/628509b8d5dc749656a9641f4caf13540e2cdec85276964ff8f43bbb1d3b/Flask-1.1.1-py2.py3-none-any.whl (94kB) Collecting waitress Downloading https://files.pythonhosted.org/packages/6b/0d/589f9e46222fd5b300dc4550427452ccd5fbe9720dad4148f227ba8e3bf7/waitress-1.4.2-py2.py3-none-any.whl (148kB) Collecting click>=5.1 Downloading https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB) Collecting itsdangerous>=0.24 Downloading https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl Collecting Jinja2>=2.10.1 Downloading https://files.pythonhosted.org/packages/65/e0/eb35e762802015cab1ccee04e8a277b03f1d8e53da3ec3106882ec42558b/Jinja2-2.10.3-py2.py3-none-any.whl (125kB) Collecting Werkzeug>=0.15 Downloading https://files.pythonhosted.org/packages/ce/42/3aeda98f96e85fd26180534d36570e4d18108d62ae36f87694b476b83d6f/Werkzeug-0.16.0-py2.py3-none-any.whl (327kB) Collecting MarkupSafe>=0.23 Downloading https://files.pythonhosted.org/packages/b9/2e/64db92e53b86efccfaea71321f597fa2e1b2bd3853d8ce658568f7a13094/MarkupSafe-1.1.1.tar.gz Building wheels for collected packages: MarkupSafe Building wheel for MarkupSafe (setup.py): started Building wheel for MarkupSafe (setup.py): finished with status 'done' Created wheel for MarkupSafe: filename=MarkupSafe-1.1.1-cp37-none-any.whl size=12629 sha256=070c10033032f4d33e442f5d7d73a3032a8c30920b16bfceb42e8d1877ddf35a Stored in directory: /root/.cache/pip/wheels/f2/aa/04/0edf07a1b8a5f5f1aed7580fffb69ce8972edc16a505916a77 Successfully built MarkupSafe Installing collected packages: click, itsdangerous, MarkupSafe, Jinja2, Werkzeug, flask, waitress Successfully installed Jinja2-2.10.3 MarkupSafe-1.1.1 Werkzeug-0.16.0 click-7.0 flask-1.1.1 itsdangerous-1.1.0 waitress-1.4.2 Removing intermediate container 0d27f5967721 ---> 67626c43d171 Step 16/32 : USER app ---> Running in 95b6c95fcfeb Removing intermediate container 95b6c95fcfeb ---> 073b169d754e Step 17/32 : RUN mkdir -p function ---> Running in e87d4bbdc9dc Removing intermediate container e87d4bbdc9dc ---> 9d3fe74f3f2d Step 18/32 : RUN touch ./function/init.py ---> Running in 9119f790600f Removing intermediate container 9119f790600f ---> c64170d33ab9 Step 19/32 : WORKDIR /home/app/function/ ---> Running in 4f2961014681 Removing intermediate container 4f2961014681 ---> 8d931affe5df Step 20/32 : COPY function/requirements.txt . ---> 0e9b4f5e518e Step 21/32 : RUN pip install --user -r requirements.txt ---> Running in 27f5e8b4ad84 Collecting blackhole Downloading https://files.pythonhosted.org/packages/e9/72/db09a16c592e38b8bf7e56dce983679cab8f712dc5bef58644b3e46107d5/blackhole-2.1.13-py2.py3-none-any.whl (45kB) Installing collected packages: blackhole Successfully installed blackhole-2.1.13 Removing intermediate container 27f5e8b4ad84 ---> 1dcf644599ff Step 22/32 : WORKDIR /home/app/ ---> Running in e84b6b56c45b Removing intermediate container e84b6b56c45b ---> b3b771756fa2 Step 23/32 : USER root ---> Running in b55b00844a04 Removing intermediate container b55b00844a04 ---> b1c0ef55c772 Step 24/32 : COPY function function ---> 0a042f327c2b Step 25/32 : RUN chown -R app:app ./ ---> Running in 0cbd72f95931 Removing intermediate container 0cbd72f95931 ---> 42553c05dbba Step 26/32 : USER app ---> Running in 82972565a7bf Removing intermediate container 82972565a7bf ---> d00d211b88eb Step 27/32 : ENV fprocess="python index.py" ---> Running in f4fc395ec184 Removing intermediate container f4fc395ec184 ---> fd1e5b4ea091 Step 28/32 : ENV cgi_headers="true" ---> Running in 822ef717ad40 Removing intermediate container 822ef717ad40 ---> 9582fc72a9c8 Step 29/32 : ENV mode="http" ---> Running in e1072d332ac9 Removing intermediate container e1072d332ac9 ---> a0b204a4771f Step 30/32 : ENV upstream_url="http://127.0.0.1:5000" ---> Running in 4864c0b72f0e Removing intermediate container 4864c0b72f0e ---> eca01ac8a495 Step 31/32 : HEALTHCHECK --interval=5s CMD [ -e /tmp/.lock ] || exit 1 ---> Running in 6deabf1a1167 Removing intermediate container 6deabf1a1167 ---> 2e12c1c44b8f Step 32/32 : CMD ["fwatchdog"] ---> Running in 943d6c5fc1a8 Removing intermediate container 943d6c5fc1a8 ---> f742aaeddbe9 Successfully built f742aaeddbe9 Successfully tagged jefflowrey/general-repo:import-test SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories. Image: jefflowrey/general-repo:import-test built. [0] < Building import-test done. [0] worker done.

[LucasRoesler]

I am not sure what is happening, but I can build and run this function locally without any issue

This is my build output, for comparison:

faas-cli build -f import-test.yml
[0] > Building import-test.
Clearing temporary build folder: ./build/import-test/
Preparing ./import-test/ ./build/import-test//function
Building: jefflowrey/general-repo:import-test with python3-http template. Please wait..
#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 1.08kB 0.0s done
#1 DONE 0.0s

#4 [internal] load metadata for docker.io/openfaas/of-watchdog:0.7.2
#4 ...

#3 [internal] load metadata for docker.io/library/python:3.7-alpine
#3 DONE 1.3s

#4 [internal] load metadata for docker.io/openfaas/of-watchdog:0.7.2
#4 DONE 1.4s

#6 [watchdog 1/1] FROM docker.io/openfaas/of-watchdog:0.7.2@sha256:a0103df5...
#6 DONE 0.0s

#5 [stage-1 1/18] FROM docker.io/library/python:3.7-alpine@sha256:860856fce...
#5 DONE 0.0s

#13 [internal] load build context
#13 transferring context: 2.10kB done
#13 DONE 0.0s

#12 [stage-1 7/18] WORKDIR /home/app/
#12 CACHED

#16 [stage-1 10/18] RUN pip install -r requirements.txt
#16 CACHED

#8 [stage-1 3/18] RUN chmod +x /usr/bin/fwatchdog
#8 CACHED

#17 [stage-1 11/18] RUN mkdir -p function
#17 CACHED

#15 [stage-1 9/18] COPY requirements.txt   .
#15 CACHED

#18 [stage-1 12/18] RUN touch ./function/__init__.py
#18 CACHED

#9 [stage-1 4/18] RUN apk --no-cache add ${ADDITIONAL_PACKAGE}
#9 CACHED

#14 [stage-1 8/18] COPY index.py           .
#14 CACHED

#7 [stage-1 2/18] COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog
#7 CACHED

#10 [stage-1 5/18] RUN addgroup -S app && adduser app -S -G app
#10 CACHED

#11 [stage-1 6/18] RUN chown app /home/app
#11 CACHED

#19 [stage-1 13/18] WORKDIR /home/app/function/
#19 CACHED

#20 [stage-1 14/18] COPY function/requirements.txt  .
#20 DONE 0.0s

#21 [stage-1 15/18] RUN pip install --user -r requirements.txt
#21 0.931 Collecting blackhole
#21 1.204   Downloading https://files.pythonhosted.org/packages/e9/72/db09a16c592e38b8bf7e56dce983679cab8f712dc5bef58644b3e46107d5/blackhole-2.1.13-py2.py3-none-any.whl (45kB)
#21 1.352 Installing collected packages: blackhole
#21 1.396 Successfully installed blackhole-2.1.13
#21 DONE 1.7s

#22 [stage-1 16/18] WORKDIR /home/app/
#22 DONE 0.0s

#23 [stage-1 17/18] COPY function   function
#23 DONE 0.0s

#24 [stage-1 18/18] RUN chown -R app:app ./
#24 DONE 0.4s

#25 exporting to image
#25 exporting layers 0.1s done
#25 writing image sha256:6cbfad433d8167ea46596c79ce90af2225c90cfeb7b72378167caa569cfadf87
#25 writing image sha256:6cbfad433d8167ea46596c79ce90af2225c90cfeb7b72378167caa569cfadf87 done
#25 naming to docker.io/jefflowrey/general-repo:import-test done
#25 DONE 0.1s
Image: jefflowrey/general-repo:import-test built.
[0] < Building import-test done in 3.99s.
[0] Worker done.

Total build time: 3.99

And then I can run it manually

docker run --rm -it -p 8080:8080 jefflowrey/general-repo:import-test
Forking - python [index.py]
2020/01/06 16:12:59 Started logging stderr from function.
2020/01/06 16:12:59 Started logging stdout from function.
2020/01/06 16:12:59 OperationalMode: http
2020/01/06 16:12:59 Timeouts: read: 10s, write: 10s hard: 10s.
2020/01/06 16:12:59 Listening on port: 8080
2020/01/06 16:12:59 Writing lock-file to: /tmp/.lock
2020/01/06 16:12:59 Metrics listening on port: 8081
2020/01/06 16:13:17 POST / - 200 OK - ContentLength: 20

And then curl it, as in the screenshot above

curl http://localhost:8080 -d "stuf"
Hello from OpenFaaS!

@Jeff-Lowrey the security warning about the windows image on a non-Windows Docker is a little interesting. I don't have a lot of experience with Docker on Windows though

[LucasRoesler]

@Jeff-Lowrey can you try my example here, which I have already tested as working, https://github.com/LucasRoesler/botopush. I created it in response to your original question on slack, which mentioned boto.

[Jeff-Lowrey]

Perhaps it's SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories. Image: jefflowrey/general-repo:import-test built. ?

[LucasRoesler]

I guess it is possible, but I am not sure. Given that the issue is specifically with the extra packages, those are not copied in from the build context, and your build output does not show any errors try to read or write any files.

[Jeff-Lowrey]

Your function fails as well. But that's from a failure to pull from the docker repo.

'Failed to pull image "theaxer/botopush:latest": rpc error: code = Unknown desc = repository docker.io/theaxer/botopush not found: does not exist or no pull access'

I also didn't have permissions/access to push the image. 'denied: requested access to the resource is denied 2020/01/06 10:27:26 ERROR - Could not execute command: [docker push theaxer/botopush:latest-2454dcb]'

[Jeff-Lowrey]

I'm going to try to build my sample from linux instead of windows.

[alexellis]

@Jeff-Lowrey you cannot push an image from another user without updating the username in stack.yml to your own. Please try this, the error you received there is expected.

For me (alexellis2), I'd put:

   image: alexellis2/botopush

Instead of theaxer/botopush

You can also run faas-cli build on its own.

I would also recommend that you use Git Bash and not Windows CMD or Powershell for all OpenFAaS activities. We define this as a requirement in the workshop - https://github.com/openfaas/workshop

[alexellis]

Additionally, since you're on OpenShift, you may need to have some changes made to the template like I did with @LucasRoesler for the python3 template to cope with the random UID it assigns.

https://github.com/openfaas-incubator/python-flask-template/blob/master/template/python3-http/Dockerfile

https://github.com/openfaas/templates/blob/master/template/python3/Dockerfile#L25

[Jeff-Lowrey]

Aha! Success! Changing the Dockerfile to the #L25 version worked. My code now deploys and my pods find the imports.

[alexellis]

Can you elaborate please?

[alexellis]

Reopening as no resolution or fix has been made

[Jeff-Lowrey]

I replaced the contents of the dockerfile in template/python3-http with the contents of the L25 file, and the deploy and import worked.

[alexellis]

Ok, it sounded like you replaced the whole file. Do you want to send us a PR with the fix or for us to take it? You can reproduce this error outside of OpenShift using docker run and a high UID.

[alexellis]

@rgee0 may remember the PR where we fixed this fit the main template

[Jeff-Lowrey]

Actually, replacing the whole file wasn't the right thing to do. It caused openshift or the openfaas gateway or something to route to "/function/\<fn>" instead of "/system/function/\<fn>?namespace=openfaas-fn" Where '\<fn>' is the function name. (among other things, it switched to using classic-watchdog instead of of-watchdog).

I've got a working Dockerfile at https://github.com/Jeff-Lowrey/openfaas-import-test/blob/master/import_test/template/python3-http/Dockerfile

I'm not confident enough in my work to submit it as a PR. It's at a "works for me" state, but I don't have the time or resources to do a full test cycle on it.

[alexellis]

I agree, I wasn't suggesting that you replace the whole dockerfile, but you can use the python3 template if you want. We'll find someone to take this.

[alexellis]

Closing as inactive.