Add support for using git tokens in python dependencies

CC007 commented 2 years ago

My actions before raising this issue

[x] Followed the troubleshooting guide
[x] Read/searched the docs
[x] Searched past issues

Why do you need this?

PIP has a feature to use a git repo directly as a dependency. Since python doesn't need to be compiled before running it, this allows you to add dependencies, without having to post the code to a library repository like PyPi. In order to use a private repo, you can define the repository in the following way:

git+https://${GITHUB_TOKEN}@github.com/user/project.git@{version}

Here GITHUB_TOKEN is read as an environment variable. In order to use this functionality, I need the ability to specify an environment variable in such a way that it doesn't end up inside the final container (as this would be a security vulnerability).

Expected Behaviour

GIT_TOKEN can be supplied as a build argument
this GIT_TOKEN is made available as an environment variable while pip install is running
the GIT_TOKEN is not made available in the final image

Current Behaviour

To my knowledge, the current python3 template doesn't support providing this GIT_TOKEN.

Are you a GitHub Sponsor (Yes/No?)

Check at: https://github.com/sponsors/openfaas

[ ] Yes
[x] No

List All Possible Solutions and Workarounds

you could install the git repo as a submodule, but that feels janky and doesn't handle transitive dependencies
you can create a private python module repo, but you need to setup some kind of infrastructure, like a nexus server. This request would allow you to completely bypass that. Also, to be able to connect to such a private repo you'd have to configure a token or other kind of auth anyway, so that solution wouldn't have any benefits compared to the github repo solution.

Which Solution Do You Recommend?

Due to the reasons I explained above, I would prefer the ability to use a private git repo in my requirements.txt directly, which requires the GIT_TOKEN env var to be available.

Context

I am a software engineer that runs a local k8s cluster for hobby projects. This cluster also uses OpenFaaS. I am now working on a project where I am using python functions. A lot of these functions share some common code that I now want to separate out into a library, without having to setup too much extra infrastructure.

Your Environment

FaaS-CLI version ( Full output from: faas-cli version ): 0.13.9
Docker version docker version (e.g. Docker 17.0.05 ): 20.10.14
Are you using OpenFaaS on Kubernetes or faasd? On Kubernetes
Operating System and version (e.g. Linux, Windows, MacOS): Windows + WSL host, ubuntu k8s vm's.

alexellis commented 2 years ago

Thanks for your interest in OpenFaaS.

@LucasRoesler what thoughts do you have here?

LucasRoesler commented 2 years ago

@alexellis i like the idea

VladoPortos commented 2 years ago

The python3 template does not have git installed probably in first place, at least this is what I found today when I tried to use git repo in requirements. ( It would probably not work anyway, since I was trying to install Parrot: git+https://github.com/PrithivirajDamodaran/Parrot.git into arm64 rPi4 😆 )

#21 8.223   ERROR: Error [Errno 2] No such file or directory: 'git' while executing command git version
#21 8.225 ERROR: Cannot find command 'git' - do you have 'git' installed and in your PATH?

Also, this when importing only pip library "sentencepiece" on other hand this might work in python3-dlrs since Transformers seems to be included there.

#21 142.3       ./build_bundled.sh: line 8: git: not found
#21 142.3       ./build_bundled.sh: line 10: git: not found
#21 142.3       ./build_bundled.sh: cd: line 12: can't cd to sentencepiece: No such file or directory
#21 142.3       ./build_bundled.sh: line 15: cmake: not found
#21 142.3       ./build_bundled.sh: line 16: make: not found
#21 142.3       ./build_bundled.sh: line 17: make: not found

CC007 commented 2 years ago

The python3 template does not have git installed probably in first place

That's right. That's why in the pull request I added a builder stage where I did install git. That way the final image doesn't need to have git installed, but you can still use it during the build

VladoPortos commented 2 years ago

@CC007 I just found that I can actually add what I need into the Docker file in template for python3 locally and have it build with the stuff I need. It is actually running right now, I'm so curious if it is going to build it ok and how long it will take :D (numpy, Transformers, etc... )

I should probably use the --virtual for apk, and have it removed after... but maybe later :) ( I will look at your pull request )

openfaas / faas