search

Azure / azure-functions-python-worker

Python worker for Azure Functions.
http://aka.ms/azurefunctions
MIT License
335 stars 103 forks source link

Missing binary to run numpy in py38 and function v4 #1096

Closed pievalentin closed 2 years ago

pievalentin commented 2 years ago

Investigative information

Please provide the following:

Repro steps

Provide the steps required to reproduce the problem:

Set numpy as a dependecy in the requirements.txt Have the python function import numpy like import numpy as np This is very similar to this issue but using the ubuntu vm as a builder didn't solve the issue

Expected behavior

To be able to import Numpy

Actual behavior

Fail to import numpy. This is the output of my durable function.

{                                                                                                                                                                                                                                                                                                                             
    "createdTime": "2022-08-24T21:27:52Z",                                                                                                                                                                                                                                                                                    
    "customStatus": null,                                                                                                                                                                                                                                                                                                     
    "input": "{\"hello\": \"world\"}",                                                                                                                                                                                                                                                                                        
    "instanceId": "d9005b995aec46b6a15c66c8ec67fbbd",                                                                                                                                                                                                                                                                         
    "lastUpdatedTime": "2022-08-24T21:27:52Z",                                                                                                                                                                                                                                                                                
    "name": "orchestrator-function",                                                                                                                                                                                                                                                                                          
    "output": "Orchestrator function 'orchestrator-function' failed: Activity function 'processing' failed:  ImportError: Unable to import required dependencies:\nnumpy: \n\nIMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!\n\nImporting the numpy C-extensions failed. This error can happen for\nmany r
easons, often due to issues with your setup or how NumPy was\ninstalled.\n\nWe have compiled some common reasons and troubleshooting tips at:\n\n    https://numpy.org/devdocs/user/troubleshooting-importerror.html\n\nPlease note and check the following:\n\n  * The Python version is: Python3.8 from \"/usr/local/bin/pyt
hon\"\n  * The NumPy version is: \"1.23.2\"\n\nand make sure that they are the versions you expect.\nPlease carefully study the documentation linked above for further help.\n\nOriginal error was: No module named 'numpy.core._multiarray_umath'\n. Please check the requirements.txt file for the missing module. For more 
info, please refer the troubleshooting guide: https://aka.ms/functions-modulenotfound \n {\"$type\":\"System.Exception, System.Private.CoreLib\",\"ClassName\":\"System.Exception\",\"Message\":\" ImportError: Unable to import required dependencies:\\nnumpy: \\n\\nIMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE 
THIS ISSUE!\\n\\nImporting the numpy C-extensions failed. This error can happen for\\nmany reasons, often due to issues with your setup or how NumPy was\\ninstalled.\\n\\nWe have compiled some common reasons and troubleshooting tips at:\\n\\n    https://numpy.org/devdocs/user/troubleshooting-importerror.html\\n\\nPle
ase note and check the following:\\n\\n  * The Python version is: Python3.8 from \\\"/usr/local/bin/python\\\"\\n  * The NumPy version is: \\\"1.23.2\\\"\\n\\nand make sure that they are the versions you expect.\\nPlease carefully study the documentation linked above for further help.\\n\\nOriginal error was: No modu
le named 'numpy.core._multiarray_umath'\\n. Please check the requirements.txt file for the missing module. For more info, please refer the troubleshooting guide: https://aka.ms/functions-modulenotfound\",\"Data\":null,\"InnerException\":{\"$type\":\"Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException, Microsoft.A
zure.WebJobs.Script\",\"IsUserException\":false,\"RemoteStackTrace\":\"  File \\\"/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/dispatcher.py\\\", line 355, in _handle__function_load_request\\n    func = loader.load_function(\\n  File \\\"/azure-functions-host/workers/python/3.8/LINUX/X64/
azure_functions_worker/utils/wrappers.py\\\", line 42, in call\\n    raise extend_exception_message(e, message)\\n  File \\\"/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/utils/wrappers.py\\\", line 40, in call\\n    return func(*args, **kwargs)\\n  File \\\"/azure-functions-host/workers/p
ython/3.8/LINUX/X64/azure_functions_worker/loader.py\\\", line 127, in load_function\\n    mod = importlib.import_module(fullmodname)\\n  File \\\"/usr/local/lib/python3.8/importlib/__init__.py\\\", line 127, in import_module\\n    return _bootstrap._gcd_import(name[level:], package, level)\\n  File \\\"<frozen impor
tlib._bootstrap>\\\", line 1014, in _gcd_import\\n  File \\\"<frozen importlib._bootstrap>\\\", line 991, in _find_and_load\\n  File \\\"<frozen importlib._bootstrap>\\\", line 975, in _find_and_load_unlocked\\n  File \\\"<frozen importlib._bootstrap>\\\", line 671, in _load_unlocked\\n  File \\\"<frozen importlib._b
ootstrap_external>\\\", line 843, in exec_module\\n  File \\\"<frozen importlib._bootstrap>\\\", line 219, in _call_with_frames_removed\\n  File \\\"/home/site/wwwroot/processing/__init__.py\\\", line 6, in <module>\\n    from src.datadrift import report_data_drift\\n  File \\\"/home/site/wwwroot/src/datadrift.py\\\"
, line 1, in <module>\\n    import pandas as pd\\n  File \\\"/home/site/wwwroot/.python_packages/lib/site-packages/pandas/__init__.py\\\", line 16, in <module>\\n    raise ImportError(\\n\",\"RemoteMessage\":\"ImportError: Unable to import required dependencies:\\nnumpy: \\n\\nIMPORTANT: PLEASE READ THIS FOR ADVICE O
N HOW TO SOLVE THIS ISSUE!\\n\\nImporting the numpy C-extensions failed. This error can happen for\\nmany reasons, often due to issues with your setup or how NumPy was\\ninstalled.\\n\\nWe have compiled some common reasons and troubleshooting tips at:\\n\\n    https://numpy.org/devdocs/user/troubleshooting-importerro
r.html\\n\\nPlease note and check the following:\\n\\n  * The Python version is: Python3.8 from \\\"/usr/local/bin/python\\\"\\n  * The NumPy version is: \\\"1.23.2\\\"\\n\\nand make sure that they are the versions you expect.\\nPlease carefully study the documentation linked above for further help.\\n\\nOriginal err
or was: No module named 'numpy.core._multiarray_umath'\\n. Please check the requirements.txt file for the missing module. For more info, please refer the troubleshooting guide: https://aka.ms/functions-modulenotfound\",\"RemoteTypeName\":null,\"Message\":\"Result: Failure\\nException: ImportError: Unable to import re
quired dependencies:\\nnumpy: \\n\\nIMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!\\n\\nImporting the numpy C-extensions failed. This error can happen for\\nmany reasons, often due to issues with your setup or how NumPy was\\ninstalled.\\n\\nWe have compiled some common reasons and troubleshooting
 tips at:\\n\\n    https://numpy.org/devdocs/user/troubleshooting-importerror.html\\n\\nPlease note and check the following:\\n\\n  * The Python version is: Python3.8 from \\\"/usr/local/bin/python\\\"\\n  * The NumPy version is: \\\"1.23.2\\\"\\n\\nand make sure that they are the versions you expect.\\nPlease carefu
lly study the documentation linked above for further help.\\n\\nOriginal error was: No module named 'numpy.core._multiarray_umath'\\n. Please check the requirements.txt file for the missing module. For more info, please refer the troubleshooting guide: https://aka.ms/functions-modulenotfound\\nStack:   File \\\"/azur
e-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/dispatcher.py\\\", line 355, in _handle__function_load_request\\n    func = loader.load_function(\\n  File \\\"/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/utils/wrappers.py\\\", line 42, in call\\n    raise extend_excep
tion_message(e, message)\\n  File \\\"/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/utils/wrappers.py\\\", line 40, in call\\n    return func(*args, **kwargs)\\n  File \\\"/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/loader.py\\\", line 127, in load_function\\n
    mod = importlib.import_module(fullmodname)\\n  File \\\"/usr/local/lib/python3.8/importlib/__init__.py\\\", line 127, in import_module\\n    return _bootstrap._gcd_import(name[level:], package, level)\\n  File \\\"<frozen importlib._bootstrap>\\\", line 1014, in _gcd_import\\n  File \\\"<frozen importlib._bootstr
ap>\\\", line 991, in _find_and_load\\n  File \\\"<frozen importlib._bootstrap>\\\", line 975, in _find_and_load_unlocked\\n  File \\\"<frozen importlib._bootstrap>\\\", line 671, in _load_unlocked\\n  File \\\"<frozen importlib._bootstrap_external>\\\", line 843, in exec_module\\n  File \\\"<frozen importlib._bootst
rap>\\\", line 219, in _call_with_frames_removed\\n  File \\\"/home/site/wwwroot/processing/__init__.py\\\", line 6, in <module>\\n    from src.datadrift import report_data_drift\\n  File \\\"/home/site/wwwroot/src/datadrift.py\\\", line 1, in <module>\\n    import pandas as pd\\n  File \\\"/home/site/wwwroot/.python
_packages/lib/site-packages/pandas/__init__.py\\\", line 16, in <module>\\n    raise ImportError(\\n\",\"Data\":{\"$type\":\"System.Collections.ListDictionaryInternal, System.Private.CoreLib\"},\"InnerException\":null,\"HelpLink\":null,\"Source\":\"System.Private.CoreLib\",\"HResult\":-2146233088,\"StackTrace\":\"   
at Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.InvokeCore(Object[] parameters, FunctionInvocationContext context) in /src/azure-functions-host/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs:line 96\\n   at Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Obje
ct[] parameters) in /src/azure-functions-host/src/WebJobs.Script/Description/FunctionInvokerBase.cs:line 82\\n   at Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator.Coerce[T](Task`1 src) in /src/azure-functions-host/src/WebJobs.Script/Description/FunctionGenerator.cs:line 225\\n   at Microsoft.Azure.WebJo
bs.Host.Executors.FunctionInvoker`2.InvokeAsync(Object instance, Object[] arguments) in C:\\\\projects\\\\azure-webjobs-sdk-rqm4t\\\\src\\\\Microsoft.Azure.WebJobs.Host\\\\Executors\\\\FunctionInvoker.cs:line 52\\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker in
voker, ParameterHelper parameterHelper, CancellationTokenSource timeoutTokenSource, CancellationTokenSource functionCancellationTokenSource, Boolean throwOnTimeout, TimeSpan timerInterval, IFunctionInstance instance) in C:\\\\projects\\\\azure-webjobs-sdk-rqm4t\\\\src\\\\Microsoft.Azure.WebJobs.Host\\\\Executors\\\\F
unctionExecutor.cs:line 581\\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance, ParameterHelper parameterHelper, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in C:\\\\projects\\\\azure-webjobs-sdk-rqm4t\\\\src\\\\Microsoft.
Azure.WebJobs.Host\\\\Executors\\\\FunctionExecutor.cs:line 527\\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, Cancellation
Token cancellationToken) in C:\\\\projects\\\\azure-webjobs-sdk-rqm4t\\\\src\\\\Microsoft.Azure.WebJobs.Host\\\\Executors\\\\FunctionExecutor.cs:line 306\\n   at Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.InvokeCore(Object[] parameters, FunctionInvocationContext context) in /src/azure-functions-
host/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs:line 96\\n   at Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) in /src/azure-functions-host/src/WebJobs.Script/Description/FunctionInvokerBase.cs:line 82\\n   at Microsoft.Azure.WebJobs.Script.Description.
FunctionGenerator.Coerce[T](Task`1 src) in /src/azure-functions-host/src/WebJobs.Script/Description/FunctionGenerator.cs:line 225\\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync(Object instance, Object[] arguments) in C:\\\\projects\\\\azure-webjobs-sdk-rqm4t\\\\src\\\\Microsoft.Azure.Web
Jobs.Host\\\\Executors\\\\FunctionInvoker.cs:line 52\\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker, ParameterHelper parameterHelper, CancellationTokenSource timeoutTokenSource, CancellationTokenSource functionCancellationTokenSource, Boolean throwOnTim
eout, TimeSpan timerInterval, IFunctionInstance instance) in C:\\\\projects\\\\azure-webjobs-sdk-rqm4t\\\\src\\\\Microsoft.Azure.WebJobs.Host\\\\Executors\\\\FunctionExecutor.cs:line 581\\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance, ParameterHel
per parameterHelper, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in C:\\\\projects\\\\azure-webjobs-sdk-rqm4t\\\\src\\\\Microsoft.Azure.WebJobs.Host\\\\Executors\\\\FunctionExecutor.cs:line 527\\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFuncti
onInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in C:\\\\projects\\\\azure-webjobs-sdk-rqm4t\\\\src\\\\Microsoft.Azure.WebJobs.Host\\\\Executors\\\\FunctionExecutor.cs:line 306\"},\"
HelpURL\":null,\"StackTraceString\":null,\"RemoteStackTraceString\":null,\"RemoteStackIndex\":0,\"ExceptionMethod\":null,\"HResult\":-2146233088,\"Source\":null,\"WatsonBuckets\":null}",                                                                                                                                    
    "runtimeStatus": "Failed"                                                                                                                                                                                                                                                                                                 
}

Known workarounds

reinstall numpy on the host. Pip should make sure the c libraries are presents. Thing is I don't have access to the host as its serverless

Related information

Provide any related information 

# processing.py
import numpy as np

In other word is there any workaround to make sure that the binary necessary for numpy are present on the host?

martin-greentrax commented 2 years ago

We face the same problem with another package (nest-asyncio). Have a look here: https://docs.microsoft.com/en-us/answers/questions/976311/modulenotfounderror-on-external-packages.html Sadly the proposed solution there made things worse for us. The function did not even start up

pdthummar commented 2 years ago

@pievalentin I have tried deploying the function through VSCode to dedicated app and tested basic function with utilizing NumPy and it is working as expected. 1) Are you still facing this issue? 2) If yes, could you please provide more details for reproducing issue?

Thanks

ghost commented 2 years ago

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

pievalentin commented 1 year ago

@pdthummar I found a fix for my issue. In the azure-pipelines.yaml building my Azure function zip, I now run this command: pip install --only-binary=:all: --target="./.python_packages/lib/site-packages" -r ./requirements.txt

Specifically asking for --only-binary=:all: This solved my issue of missing binaries once the function is running

pievalentin commented 1 year ago

@martin-greentrax It's been a while but maybe my solution will help you

martin-greentrax commented 1 year ago

@pievalentin Thank you for your solution. Back then we found this proposed workaround on-line and we have not seen the issue afterwards. Adding this to the main entry point file of our function:

sys.path.append("/home/site/wwwroot/.python_packages/lib/site-packages")
sys.path.append("/home/site/wwwroot")
sys.path.append("/tmp/functions/standby/wwwroot")

Last week we were notified by Microsoft support staff that this work around would no longer be necessary. Since this issue only occurred occasionally, was not able to be triggered by artificial traffic and affected our production systems we are a bit hesitant to experiment though. :)

vrdmr commented 1 year ago

Similar to #1116. Will post updates in there.