search

Azure / azure-functions-python-worker

Python worker for Azure Functions.
http://aka.ms/azurefunctions
MIT License
335 stars 103 forks source link

libgomp.so serror #493

Closed bszewczyk closed 4 years ago

bszewczyk commented 5 years ago

Cannot import python library xgboost. When I run my function app locally project everything works fine. After publish I get his error

Result: Failure Exception: XGBoostError: XGBoost Library (libxgboost.so) could not be loaded. Likely causes:

It looks like that it need to be configured on linux machine but there is no option for that in azure function apps hosted on linux

https://xansons4cod.com/xansons4cod/forum_thread.php?id=72

Investigative information

Please provide the following:

  1. Create python scripts using xgboost library
  2. Publish to Azure App Function

Expected behavior

Function should run

Actual behavior

Getting error

mathieuisabel commented 5 years ago

I'm hitting the same issue. Have you found a way around this?

bszewczyk commented 5 years ago

Unfortunately no, waiting for Microsoft team to fix it.

mathieuisabel commented 5 years ago

Do you have an opened case for this or are you in touch with the product group? I’ve reached out to MSFT through the MVP program and I’m pending an answer. I’ll try the Docker route to see if I can use this as a workaround.

Thanks!

Get Outlook for iOShttps://aka.ms/o0ukef

From: Beata Szewczyk notifications@github.com Sent: Wednesday, July 24, 2019 2:20:45 AM To: Azure/azure-functions-python-worker azure-functions-python-worker@noreply.github.com Cc: Mathieu Isabel Mathieu.Isabel@globalexcel.com; Comment comment@noreply.github.com Subject: Re: [Azure/azure-functions-python-worker] libgomp.so serror (#493)

Unfortunately no, waiting for Microsoft team to fix it.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-functions-python-worker%2Fissues%2F493%3Femail_source%3Dnotifications%26email_token%3DADYSYLH534FEZQUFHEMUV2LQA7YD3A5CNFSM4IFNAJHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2VJREA%23issuecomment-514496656&data=02%7C01%7Cmathieu.isabel%40globalexcel.com%7Cda4078473e864ce89fef08d70fff1088%7Ccbe28c2f7e6d4e81bdac76a2e9f7256c%7C1%7C0%7C636995460479537748&sdata=948aAGDIlaZPfYZH0Uo9Pz3GHXB4EZihggMgdSo0EGs%3D&reserved=0, or mute the threadhttps://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FADYSYLAA3FSDNFVANY4MGKDQA7YD3ANCNFSM4IFNAJHA&data=02%7C01%7Cmathieu.isabel%40globalexcel.com%7Cda4078473e864ce89fef08d70fff1088%7Ccbe28c2f7e6d4e81bdac76a2e9f7256c%7C1%7C0%7C636995460479547749&sdata=QhQoC0J8toUhX3ILKKaVsi07MYks7uElbiWOsYzLWzk%3D&reserved=0.

---------- The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material; unauthorized use of this information is prohibited. If you have received this in error, please contact the sender and delete the material immediately. L'information transmise ne s'adresse qu'au particulier ou à l'organisme a qui elle est dirigée. Elle peut contenir des renseignements de nature privilégiée et/ou confidentielle. Toute utilisation non autorisée est interdite. Si vous avez reçu ce courriel par erreur, SVP le retourner à l'expéditeur et le détruire. ----------

mathieuisabel commented 5 years ago

Quick update on this. I was able to run my function app successfully by following these steps:

1) In your function app run: func init --docker-only 2) Modified generated dockerfile as follow:

FROM mcr.microsoft.com/azure-functions/python:2.0

ENV AzureWebJobsScriptRoot=/home/site/wwwroot \ AzureFunctionsJobHostLoggingConsole__IsEnabled=true RUN apt-get update && apt-get install -y gcc

COPY . /home/site/wwwroot

RUN cd /home/site/wwwroot && \ pip install -r requirements.txt

3) Push image to Azure Container Registry 4) Create Linux Docker function app 5) Pick the image you pushed to ACR 6) Wait a bit (took a few minutes for the app to come online)

iajoiner commented 5 years ago

I have the same problem when using LightGBM.

Starting Live Log Stream ---
2019-08-12T06:41:32  No new trace in the past 1 min(s).
2019-08-12T06:42:32  No new trace in the past 2 min(s).

2019-08-12T06:42:41.172446703Z [2019-08-12 06:42:41,166] ERROR in app: Exception on / [GET]
2019-08-12T06:42:41.172482304Z Traceback (most recent call last):
2019-08-12T06:42:41.172489004Z   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2311, in wsgi_app
2019-08-12T06:42:41.172494004Z     response = self.full_dispatch_request()
2019-08-12T06:42:41.172498404Z   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1834, in full_dispatch_request
2019-08-12T06:42:41.172502904Z     rv = self.handle_user_exception(e)
2019-08-12T06:42:41.172507304Z   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1737, in handle_user_exception
2019-08-12T06:42:41.172511704Z     reraise(exc_type, exc_value, tb)
2019-08-12T06:42:41.172515804Z   File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 36, in reraise
2019-08-12T06:42:41.172527704Z     raise value
2019-08-12T06:42:41.172532104Z   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1832, in full_dispatch_request
2019-08-12T06:42:41.172536304Z     rv = self.dispatch_request()
2019-08-12T06:42:41.172540304Z   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1818, in dispatch_request
2019-08-12T06:42:41.172544504Z     return self.view_functions[rule.endpoint](**req.view_args)
2019-08-12T06:42:41.172548404Z   File "/home/site/wwwroot/app.py", line 16, in hello_world
2019-08-12T06:42:41.172552504Z     res, date = predict.predict_today_crimes()
2019-08-12T06:42:41.172556504Z   File "/home/site/wwwroot/predict.py", line 350, in predict_today_crimes
2019-08-12T06:42:41.172560604Z     lgbm_final = pickle.load(open('lgbm_reg.p','rb'))
2019-08-12T06:42:41.172564604Z   File "/home/site/wwwroot/antenv/lib/python3.7/site-packages/lightgbm/__init__.py", line 8, in <module>
2019-08-12T06:42:41.172569204Z     from .basic import Booster, Dataset
2019-08-12T06:42:41.172573304Z   File "/home/site/wwwroot/antenv/lib/python3.7/site-packages/lightgbm/basic.py", line 34, in <module>
2019-08-12T06:42:41.172578004Z     _LIB = _load_lib()
2019-08-12T06:42:41.172582004Z   File "/home/site/wwwroot/antenv/lib/python3.7/site-packages/lightgbm/basic.py", line 29, in _load_lib
2019-08-12T06:42:41.172586204Z     lib = ctypes.cdll.LoadLibrary(lib_path[0])
2019-08-12T06:42:41.172590204Z   File "/usr/local/lib/python3.7/ctypes/__init__.py", line 434, in LoadLibrary
2019-08-12T06:42:41.172594304Z     return self._dlltype(name)
2019-08-12T06:42:41.172598204Z   File "/usr/local/lib/python3.7/ctypes/__init__.py", line 356, in __init__
2019-08-12T06:42:41.172602204Z     self._handle = _dlopen(self._name, mode)
2019-08-12T06:42:41.172606204Z OSError: libgomp.so.1: cannot open shared object file: No such file or directory
horihiro commented 4 years ago

Let me re-post same comment as https://github.com/MicrosoftDocs/azure-docs/issues/40856#issuecomment-548985233 because these seem to be same issues.

It seems that xgboost tried to load a library libgomp.so.1 in runtime, but container image mcr.microsoft.com/azure-functions/python for Python Azure Function App doesn't have it.

I have faced similar situation when opencv-python-headless tries to load libglib-2.0.so.0 and libgthread-2.0.so.0 in runtime.

A following repository might help you. https://github.com/horihiro/azure-functions-python-opencv-example

anirudhgarg commented 4 years ago

Folks - we are committed to try to fix this issue for you. It would be very useful for us if you could give us a repo of a sample app. That will help us validate our fix. @bszewczyk @horihiro @mathyingzhou ?

lmmaia commented 4 years ago

@anirudhgarg is this the same error you guys are looking into? image

I'm using --build-native-deps and building my environment from anaconda.

joelhulen commented 4 years ago

@anirudhgarg Please see the following for my repo that encounters this same error: https://github.com/joelhulen/serverless-microservices/tree/features/ai/python/RideshareAI

The error I receive when executing my function is: Failure Exception: OSError: libgomp.so.1: cannot open shared object file: No such file or directory

anirudhgarg commented 4 years ago

@joelhulen I see that your app does not use XGBoost,. do you know which module is throwing that error. We tested adding the libcomp.so.1 dependency and testing with your app. We dont see that error but we see this error now:

System.Private.CoreLib: Exception while executing function: Functions.scoring. System.Private.CoreLib: Result: Failure Exception: ModuleNotFoundError: No module named 'sklearn.preprocessing.imputation'

As we add this dependency trivial app passes. If someone could share an app that uses xgboost then that would help a lot in confirming that the addition for this system dependency is all we need.

joelhulen commented 4 years ago

@anirudhgarg I've resolved the error you listed, which was due to a newer version of the scikit-learn library requiring a new dependency. I pinned a slightly older version. I've updated my repo if you want to give it a try.

Now when I try to execute the function, I receive the same error about the libgomp.so.1 dependency:

Exception while executing function: Functions.scoring <--- Result: Failure Exception: OSError: libgomp.so.1: cannot open shared object file: No such file or directory

It appears as though the module throwing the error is the automl library, via the azureml-train-automl package reference. The full error with the stack trace is appended at the end of this comment (if you love to read ;)

Since I cannot run the apt-get install libgomp1 on the host, is my only option to have you and the engineering team load that dependency in the base host image?

------- FULL ERROR OUTPUT -------

Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Functions.scoring ---> Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException: Result: Failure
Exception: OSError: libgomp.so.1: cannot open shared object file: No such file or directory
Stack:   File "/azure-functions-host/workers/python/3.6/LINUX/X64/azure_functions_worker/dispatcher.py", line 242, in _handle__function_load_request
    func_request.metadata.entry_point)
  File "/azure-functions-host/workers/python/3.6/LINUX/X64/azure_functions_worker/loader.py", line 66, in load_function
    mod = importlib.import_module(fullmodname)
  File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/site/wwwroot/scoring/__init__.py", line 8, in <module>
    from .scoring_service import predict_battery_failure
  File "/home/site/wwwroot/scoring/scoring_service.py", line 9, in <module>
    from azureml.train import automl
  File "/home/site/wwwroot/.python_packages/lib/python3.6/site-packages/azureml/train/automl/__init__.py", line 25, in <module>
    from ._automl import fit_pipeline
  File "/home/site/wwwroot/.python_packages/lib/python3.6/site-packages/azureml/train/automl/_automl.py", line 18, in <module>
    from azureml.automl.core import data_transformation, fit_pipeline as fit_pipeline_helper
  File "/home/site/wwwroot/.python_packages/lib/python3.6/site-packages/azureml/automl/core/data_transformation.py", line 20, in <module>
    from .data_context import RawDataContext, TransformedDataContext
  File "/home/site/wwwroot/.python_packages/lib/python3.6/site-packages/azureml/automl/core/data_context.py", line 9, in <module>
    from azureml.automl.core.automl_base_settings import AutoMLBaseSettings
  File "/home/site/wwwroot/.python_packages/lib/python3.6/site-packages/azureml/automl/core/automl_base_settings.py", line 20, in <module>
    from azureml.automl.core.faults_verifier import VerifierManager
  File "/home/site/wwwroot/.python_packages/lib/python3.6/site-packages/azureml/automl/core/faults_verifier.py", line 15, in <module>
    from azureml.automl.core.automl_run_context import AutoMLAbstractRunContext
  File "/home/site/wwwroot/.python_packages/lib/python3.6/site-packages/azureml/automl/core/automl_run_context.py", line 13, in <module>
    from .onnx_convert import OnnxConverter
  File "/home/site/wwwroot/.python_packages/lib/python3.6/site-packages/azureml/automl/core/onnx_convert/__init__.py", line 10, in <module>
    from .operator_converters import _AbstractOperatorConverter
  File "/home/site/wwwroot/.python_packages/lib/python3.6/site-packages/azureml/automl/core/onnx_convert/operator_converters/__init__.py", line 10, in <module>
    from ._utilities \
  File "/home/site/wwwroot/.python_packages/lib/python3.6/site-packages/azureml/automl/core/onnx_convert/operator_converters/_utilities.py", line 179, in <module>
    from onnxmltools.convert.lightgbm.operator_converters.LightGbm import convert_lightgbm       # noqa: E402
  File "/home/site/wwwroot/.python_packages/lib/python3.6/site-packages/onnxmltools/convert/lightgbm/__init__.py", line 7, in <module>
    from .convert import convert
  File "/home/site/wwwroot/.python_packages/lib/python3.6/site-packages/onnxmltools/convert/lightgbm/convert.py", line 10, in <module>
    from ._parse import parse_lightgbm
  File "/home/site/wwwroot/.python_packages/lib/python3.6/site-packages/onnxmltools/convert/lightgbm/_parse.py", line 10, in <module>
    from lightgbm import LGBMClassifier, LGBMRegressor
  File "/home/site/wwwroot/.python_packages/lib/python3.6/site-packages/lightgbm/__init__.py", line 8, in <module>
    from .basic import Booster, Dataset
  File "/home/site/wwwroot/.python_packages/lib/python3.6/site-packages/lightgbm/basic.py", line 34, in <module>
    _LIB = _load_lib()
  File "/home/site/wwwroot/.python_packages/lib/python3.6/site-packages/lightgbm/basic.py", line 29, in _load_lib
    lib = ctypes.cdll.LoadLibrary(lib_path[0])
  File "/usr/local/lib/python3.6/ctypes/__init__.py", line 426, in LoadLibrary
    return self._dlltype(name)
  File "/usr/local/lib/python3.6/ctypes/__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)

   at Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.InvokeCore(Object[] parameters, FunctionInvocationContext context) in /src/azure-functions-host/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs:line 81
   at Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) in /src/azure-functions-host/src/WebJobs.Script/Description/FunctionInvokerBase.cs:line 85
   at Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator.Coerce[T](Task`1 src) in /src/azure-functions-host/src/WebJobs.Script/Description/FunctionGenerator.cs:line 225
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync(Object instance, Object[] arguments)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeAsync(IFunctionInvoker invoker, ParameterHelper parameterHelper, CancellationTokenSource timeoutTokenSource, CancellationTokenSource functionCancellationTokenSource, Boolean throwOnTimeout, TimeSpan timerInterval, IFunctionInstance instance) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 585
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance, ParameterHelper parameterHelper, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 532
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, ParameterHelper parameterHelper, IFunctionOutputDefinition outputDefinition, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 470
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 278
   --- End of inner exception stack trace ---
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 322
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsyncCore(IFunctionInstanceEx functionInstance, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 117

Thank you for looking into this!

ahmelsayed commented 4 years ago

that works too with just libgomp1. so it should be good.

anirudhgarg commented 4 years ago

Thanks everyone for reporting and your help in testing this. We are adding libgomp.so1 now - should be available early January at the latest.

prashanth-ach commented 4 years ago

Hello @anirudhgarg - Do you have any update on this fix ?

anirudhgarg commented 4 years ago

We are waiting for a deployment for this fix. Deployment was delayed due to some reasons. We hope to have this now by the end of next week latest.

eafpres commented 4 years ago

I am having the same problem with xgboost: 2020-01-14T22:17:00.663067008Z xgboost.core.XGBoostError: XGBoost Library (libxgboost.so) could not be loaded. 2020-01-14T22:17:00.663070609Z Likely causes: 2020-01-14T22:17:00.663074109Z * OpenMP runtime is not installed (vcomp140.dll or libgomp-1.dll for Windows, libgomp.so for UNIX-like OSes) 2020-01-14T22:17:00.663077709Z * You are running 32-bit Python on a 64-bit OS

This is a Flask Python app deployed via Azure CLI: az webapp up -n -l centralus

It fails in deploy with the above errors

anirudhgarg commented 4 years ago

We have added support for the missing library libgomp.so.1 now. @prashanth-ach @bszewczyk can you please confirm that it is working for you know.

eafpres commented 4 years ago

I am having the same problem with xgboost: It has been clarified by support to me that Azure Functions and Azure Web App are completely separate, so my issue presumably does not apply here. Support has declined so far to actually do anything regarding this similar issue w/Azure Web Apps, so I'll try to post my issue in the appropriate repo.

Hazhzeng commented 4 years ago

Hi @eafpres , Sorry for your inconvenient. The Web App uses Oryx runtime which is not shared with Azure Functions App. We have a separate repository for monitoring Oryx runtime environment issues, please report your concerns here. Thanks.

eafpres commented 4 years ago

@Hazhzeng Hey Roger, thank you so much. I could not find the right repo myself.

Have a great day and thank you for reaching out!

anirudhgarg commented 4 years ago

Closing this out as we have added the missing system dependency and xgboost and lightgbm both are loading fine now. Please report if things are not working.