Adding binary wheels to a Function App

[matiaslahticgi]

I'm having issues trying to add a custom, binary wheel (GDAL) to an Azure Functions app on Consumption. Unfortunately, manylinux wheels for GDAL are not provided in PyPI and building it is a bit of a chore (developing on Windows) so we have used a pre-built wheel in the function app directory to be able to use it.

Previously, deployment used to work with requirements.txt containing a path to the wheel:

./GDAL-2.4.0-cp36-cp36m-manylinux1_x86_64.whl

Which caused building with --build-native-deps to actually install the wheel as expected from the function app directory.

This does not seem to work any more and the change seems to have happened in Azure/azure-functions-core-tools@a8f620398818cb03ff59a2b96bbae922784ac666 - but the rather drastic change in behaviour is not documented anywhere as far as I can see.

Expected behavior

func azure functionapp publish succeeds.

Actual behavior

func azure functionapp publish fails, with

Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/GDAL-2.4.0-cp36-cp36m-manylinux1_x86_64.whl'

Tried workarounds

I did try building a .zip myself, as discussed in #182, but deploying the .zip to Consumption Function App using az functionapp deployment source config-zip fails with Failed to retrieve Scm Uri (due to Kudu not being available on Consumption plans?)

There is discussion in Azure/azure-functions-core-tools#1052 concerning func pack and deploying the .zip from func pack, but --no-build isn't really applicable here - I would like to bundle the app normally, but just inject this wheel at some point.

I did try with --additional-packages, but unfortunately the GDAL package in apt is way too old to build the wheel.

Known workarounds

Currently, after digging through what feels like dozens of bug reports across different repositories (Azure/azure-functions-core-tools#1230 being the crucial one, and realizing to follow the commit referencing it..), I did find the (rather undocumented) environment option FUNCTIONS_PYTHON_DOCKER_RUN_COMMAND allowing me to (essentially manually) mount the Function App directory ($ENV:FUNCTIONS_PYTHON_DOCKER_RUN_COMMAND="run --rm -d -v ${PWD}:/mounted mcr.microsoft.com/azure-functions/python:2.0.12410-python3.6-buildenv sleep infinity") and allowing pip install to find the required wheel. However, this is not really a solution, as I guess I'll need to update the docker image manually when it gets updated?

My question then would be; what is the recommended way of bringing binary wheels into an Azure Function on Linux Consumption? Not everything, after all, can be built on deploy and sometimes binary wheels are the only way to go. Consider, for example, having to interface with a third party provided binary library.

At the very least, consider this a bug report on the documentation - it could be heavily improved. There are little pieces and very ugly hacks easy to find, but actual documentation is, well, non-existent. For example, issue #6 is very easy to find, but seems to be mostly brainstorming about how to do it which isn't immediately clear.

[ivana61]

I have Python Azure Functions on a Linux Container and I can deploy it from my local computer with the command “func azure functionapp publish $APP_NAME --build-native-deps” using docker for linux containers. Due to some restrictions I can not use -build-native-deps in my automatic build and release pipelines, therefore I tried to run func azure functionapp publish $APP_NAME, but this fails with the following error: "binary dependencies without wheels are not supported", because I have libraries like "pyodbc" in my requirements.txt.

Is there another way to publish a function app and add binary wheels to it?

[gangularamya]

going through azure func core tools code.. I think we need to add extra code at below link to check if the wheel file exists in custom folder or wheelhouse where customers can upload in project.

instead of just failing on pip download. Please fix this ASAP... we are seeing many customers with this issue

https://github.com/Azure/azure-functions-core-tools/blob/dev/tools/python/packapp/__main__.py#L154

[mebibou]

Same issue for me with GDAL as well, as reported here: https://github.com/Azure/azure-functions-core-tools/issues/1358#issuecomment-503013415

[maiqbal11]

I believe @ankitkumarr's suggestion should still be applicable here: https://github.com/Azure/azure-functions-python-worker/issues/182#issuecomment-478064747. We recognize a well known path .python_packages/lib/python3.6/site-packages in the function app folder which can have any dependencies installed in it, custom or otherwise. A Linux environment would be needed for installation. @kulkarnisonia16 , we've seen instances of this before as well, would be worth documenting a process for installing custom dependencies.

[mebibou]

@maiqbal11 the problem with GDAL is that it requires a binary file installed under /usr/bin/ogr2ogr which does not get published, so even when doing:

apt-get install -y gdal-bin libgdal-dev
pip install --target=".python_packages/lib/python3.6/site-packages" GDAL==2.1.3 --global-option=build_ext --global-option="-I/usr/include/gdal"

this installs the wheel package correctly but we are still missing the binary file on execution

[maiqbal11]

@mebibou, got it. I don't believe we have an out-of-the-box experience to currently support this on the consumption plan. Custom containers would be the recommended approach.

[robertlagrant]

@maiqbal11 this is a pretty big deal - e.g. SQLAlchemy suffers from the same problem - I get ModuleNotFound with that in --build local, --build remote and --build-native-deps. Not supporting something such a well-known library is a bit worrying.

[Hazhzeng]

Hey @robertlagrant, which SQLAlchemy version are you using?

[AlBlanc]

So sad to see we can't deploy our azure function because of gdal/geos... Would it be available soon or will we have to find another cloud provider ? :/

[anirudhgarg]

@AlBlanc is it possible for you to share a small repro and the exact binary needed. We can then quickly evaluate and see if we can add the binary to the base Consumption image. It is important for us to enable more customers - we have similarly enabled XGBoost/OpenCV and other libraries in the last few months and we can evaluate this too.

[AlBlanc]

@anirudhgarg Thank for your interest.

I just built a small Django project using geodjango, here is a link of the repo

Please provide a valid postgres (with postgis) settings with environment variables: POSTGRES_HOST, POSTGRES_USER, POSTGRES_PASS and POSTGRES_DBNAME in order to be able to start the project.

Here is a list of required depedencies, can you confirm me that only geo-related libraries are missing? (proj4, gdal and geos)

For the moment, I've only found windows wheels here, I am still looking for a manylinux wheel.

In case I don't find any and you are still up to include it in azure functions, let me know and I'll take some time to build it.

[AlBlanc]

I may have found it 😃 https://manthey.github.io/large_image_wheels/

There is plenty of interesting wheels here, we just require one using python3 but not python 3.7, about the GDAL version we only requires >2.4, 3.x are fines

[AlBlanc]

@anirudhgarg Did I provide enough information? Will you study this case?

[AlBlanc]

Pray for @anirudhgarg, I hope his absence is not related to COVID-19!

[anirudhgarg]

Ha ha - I am ok. Thanks for the concern. :). But things are taking longer as we are all adjusting to working completely online. Just to understand that you are trying to run a python function app in the consumption plan with the following libraries:

asgiref==3.2.3 astroid==2.3.3 azure-functions==1.2.0 azure-functions-wsgi-adapter==0.2 Django==3.0.4 isort==4.3.21 lazy-object-proxy==1.4.3 mccabe==0.6.1 psycopg2-binary==2.8.4 ptvsd==4.3.2 pylint==2.4.4 pytz==2019.3 six==1.14.0 sqlparse==0.3.1 wrapt==1.11.2

and this is not working - out of these libraries do you know which one is causing issues ?

[AlBlanc]

Actually, let me come back with a simpler project. I thought it would be nice to show the real concern by showing what our project look likes but binary dependencies are "hidden" by Django.

As mentioned above, when using geodjango (In few words, this is a geospatial extension for Django), it requires these binary librairies: proj4, gdal and geos. (All related for geospatial stuffs, you'll find more details here).

I hope this is not too confusing, I'll try to come back with a simpler project that put more in evidence the issue.

[laurentran]

Did we get to a conclusion on this? What's the recommended path forward? We're seeing the exact same blocker with a partner who is trying to leverage GDAL in a Python Azure Function deployment.

[fiveoaks-tech]

We also have a requirement to use GDAL/OGR within an Azure function. This can be done with Lambda functions and we are porting the solution from AWS to Azure (hopefully).

[Hazhzeng]

Please consider include the direct https:// (can include basic-auth authentication here) link in your requirements.txt and use func azure functionapp publish <appname> to publish your function app. Our remote build server will resolve the dependencies for you.

Here is a small example project using custom GDAL wheel from manthey.github.io PythonWheelExample.zip

What happen if I don't have an external website link to my wheel Consider uploading your custom wheel into your Azure Storage Blob Container, you can generate an SAS link from the blob which allow you to access the blob from remote build. As long as you're not sharing your SAS link to others, your blob is safe.

Why I cannot run my project locally now A single wheel is OS specific and Python version specific. If you're using a manylinux wheel, it will NOT work on a Windows/MacOS environment.

This is a similar issue when you're using PyTorch in Azure Functions, which the CPU version is not presented in PyPi. In order to make requirements.txt point to different wheels based on OS platform, consider building your own index.html and make your requirements.txt using the new index (pytorch index example),

-f https://download.pytorch.org/whl/torch_stable.html
torch==1.5.0+cpu

or modifying your requirements.txt pointing to different wheel based on sys_platform and python_version

https://download.pytorch.org/whl/cpu/torch-1.4.0%2Bcpu-cp36-cp36m-win_amd64.whl; sys_platform == 'win32' and python_version == '3.6'
https://download.pytorch.org/whl/cpu/torch-1.4.0%2Bcpu-cp36-cp36m-linux_x86_64.whl; sys_platform == 'linux' and python_version == '3.6'
https://download.pytorch.org/whl/cpu/torch-1.4.0%2Bcpu-cp37-cp37m-win_amd64.whl; sys_platform == 'win32' and python_version == '3.7'    
https://download.pytorch.org/whl/cpu/torch-1.4.0%2Bcpu-cp37-cp37m-linux_x86_64.whl; sys_platform == 'linux' and python_version == '3.7' 
https://download.pytorch.org/whl/cpu/torch-1.4.0%2Bcpu-cp38-cp38-win_amd64.whl; sys_platform == 'win32' and python_version == '3.8' 
https://download.pytorch.org/whl/cpu/torch-1.4.0%2Bcpu-cp38-cp38-linux_x86_64.whl; sys_platform == 'linux' and python_version == '3.8'

Why my project does not work on Azure Functions runtime Possibly, there's a misalignment between your wheels and our runtime environment. We are using Debian Stretch for v2 functions, and Debian Buster for v3 functions. We recommend using the manylinux wheels.

Also, please check your Python runtime version in https://resources.azure.com/subscriptions/<subscription uuid>/resourceGroups/<resource group name>/providers/Microsoft.Web/sites/<function app name>/config. Search for LinuxFxVersion field. Make sure it matches your custom wheel.

[ghost]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.