[BUG] On "func start" - Python worker fails to import 'builder' from 'google.protobuf.internal' (even though it's there) while importing tensorflow

jaroslavknotek commented 1 year ago

Investigative information

Please provide the following:

Timestamp: 20230420
Function App name: n/a
Function name(s) (as appropriate): myfn
Core Tools version: 4.0.5095

Repro steps

Provide the steps required to reproduce the problem:

Create new function and add import:

mkdir aztftest
cd aztftest
func init

# virtualenv
python -m venv .venv
. .venv/bin/activate
pip install - r requirements.txt #default requirements
pip install tensorflow==2.12.0

func new --name myfn --template "HTTP trigger"

#add import
sed '1 i\from tensorflow.keras.model import Model' myfn/__init__.py

Just as a sanity check, I run

>python3 --version
Python 3.9.16

>python3 -c 'from tensorflow.keras.models import Model;print("done")'

2023-04-20 20:17:47.040057: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-04-20 20:17:47.041210: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-04-20 20:17:47.064363: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-04-20 20:17:47.064678: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-04-20 20:17:47.568547: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
done

Expected behavior

Provide a description of the expected behavior.

I expect to run func start and not see any error.

Actual behavior

Provide a description of the actual behavior observed.

Running func start

> func start

Found Python version 3.9.16 (python3).
Core Tools Version:       4.0.5095 Commit hash: N/A  (64-bit)
Function Runtime Version: 4.16.5.20396

....

[2023-04-20T18:11:50.035Z] Error in load_function. ...

[2023-04-20T18:11:50.038Z] Worker failed to load function: 'myfn' with functionId: 'a6354057-27c3-40d7-b532-68c65ca5e00b'.
[2023-04-20T18:11:50.038Z] Result: Failure
[2023-04-20T18:11:50.038Z] Exception: ImportError: cannot import name 'builder' from 'google.protobuf.internal' (/home/jry/apps/azure-function-cli-new/workers/python/3.9/LINUX/X64/google/protobuf/internal/__init__.py). Please check the requirements.txt file for the missing module. For more info, please refer the troubleshooting guide: https://aka.ms/functions-modulenotfound
[2023-04-20T18:11:50.038Z] Stack:   File "/home/jry/apps/azure-function-cli-new/workers/python/3.9/LINUX/X64/azure_functions_worker/dispatcher.py", line 357, in _handle__function_load_request
[2023-04-20T18:11:50.038Z]     func = loader.load_function(
[2023-04-20T18:11:50.038Z]   File "/home/jry/apps/azure-function-cli-new/workers/python/3.9/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 48, in call
[2023-04-20T18:11:50.039Z]     raise extend_exception_message(e, message)
[2023-04-20T18:11:50.039Z]   File "/home/jry/apps/azure-function-cli-new/workers/python/3.9/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 44, in call
[2023-04-20T18:11:50.039Z]     return func(*args, **kwargs)
[2023-04-20T18:11:50.039Z]   File "/home/jry/apps/azure-function-cli-new/workers/python/3.9/LINUX/X64/azure_functions_worker/loader.py", line 132, in load_function
[2023-04-20T18:11:50.039Z]     mod = importlib.import_module(fullmodname)
[2023-04-20T18:11:50.039Z]   File "/home/jry/.pyenv/versions/3.9.16/lib/python3.9/importlib/__init__.py", line 127, in import_module
[2023-04-20T18:11:50.039Z]     return _bootstrap._gcd_import(name[level:], package, level)
[2023-04-20T18:11:50.039Z]   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
[2023-04-20T18:11:50.039Z]   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
[2023-04-20T18:11:50.039Z]   File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
[2023-04-20T18:11:50.039Z]   File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
[2023-04-20T18:11:50.039Z]   File "<frozen importlib._bootstrap_external>", line 850, in exec_module
[2023-04-20T18:11:50.039Z]   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
[2023-04-20T18:11:50.039Z]   File "/home/jry/source/jaroslavknotek/az-micro-precipitates/segment-precipiates/__init__.py", line 4, in <module>
[2023-04-20T18:11:50.039Z]     from tensorflow.keras.models import Model
[2023-04-20T18:11:50.039Z]   File "/home/jry/source/jaroslavknotek/az-micro-precipitates/.venv/lib/python3.9/site-packages/tensorflow/__init__.py", line 37, in <module>
[2023-04-20T18:11:50.039Z]     from tensorflow.python.tools import module_util as _module_util
[2023-04-20T18:11:50.040Z]   File "/home/jry/source/jaroslavknotek/az-micro-precipitates/.venv/lib/python3.9/site-packages/tensorflow/python/__init__.py", line 37, in <module>
[2023-04-20T18:11:50.040Z]     from tensorflow.python.eager import context
[2023-04-20T18:11:50.040Z]   File "/home/jry/source/jaroslavknotek/az-micro-precipitates/.venv/lib/python3.9/site-packages/tensorflow/python/eager/context.py", line 28, in <module>
[2023-04-20T18:11:50.040Z]     from tensorflow.core.framework import function_pb2
[2023-04-20T18:11:50.040Z]   File "/home/jry/source/jaroslavknotek/az-micro-precipitates/.venv/lib/python3.9/site-packages/tensorflow/core/framework/function_pb2.py", line 5, in <module>
[2023-04-20T18:11:50.040Z]     from google.protobuf.internal import builder as _builder

The func starts fails even though it's run from within the same virtual environment, with the same pytohon. For some reason, it looks for protobuf's builder in

/home/jry/apps/azure-function-cli-new/workers/python/3.9/LINUX/X64/google/protobuf/internal/__init__.py

and not

lib/python3.10/site-packages/...

I checked your guide and found protobuf wheels in

.venv/lib/python3.10/site-packages/keras/protobuf 
.venv/lib/python3.10/site-packages/tensorflow/protobuf

even running

>python3 -c 'from google.protobuf.internal import builder as _builder'

works fine.

Contents of the requirements.txt file:

Provide the requirements.txt file to help us find out module related issues.

Just default:

azure-functions

Related information

Provide any related information

Summary:

python 3.9.16 (python3).
func Core Tools Version: 4.0.5095 Commit hash: N/A (64-bit)
func Function Runtime Version: 4.16.5.20396
tensorflow = 2.12.0
protobuf =4.22.3/4.22.1 both fail
System archlinux 6.2.1-arch1-1

What can I do to resolve this problem? Searching for solution leads nowhere as I went throght

guide
Similar problems at stack overflow here and here

EDIT:

There is a workaround that works for me - isolate dependencies with PYTHON_ISOLATE_WORKER_DEPENDENCIES=1

PYTHON_ISOLATE_WORKER_DEPENDENCIES=1 func start

empowerVictor commented 1 year ago

Just ran into the same problem! I am trying to use Python's protobuf.

As the Azure Functions Core Tools contains an older version from protobuf, my function fails because it tries to load it from there, not from the .venv with the proper requirements.txt

func host start sys path:

['C:\\Program Files\\Microsoft\\Azure Functions Core Tools\\workers\\python\\3.10/WINDOWS/X64', 
'C:\\Program Files\\Microsoft\\Azure Functions Core Tools\\workers\\python\\3.10\\WINDOWS\\X64', 
'C:\\Users\\user\\miniconda3\\python310.zip',
'C:\\Users\\user\\miniconda3\\DLLs',
'C:\\Users\\user\\miniconda3\\lib', 
'C:\\Users\\user\\miniconda3', 
'c:\\dev\\function\\.venv', 
'c:\\dev\\function\\.venv\\lib\\site-packages',
'C:\\dev\\function']

Exception: ImportError: cannot import name 'builder' from 'google.protobuf.internal' (C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.10/WINDOWS/X64\google\protobuf\internal\__init__.py)

jaroslavknotek commented 1 year ago

I found a workaround (and added to the issue's description). Set env var PYTHON_ISOLATE_WORKER_DEPENDENCIES=1 before running. It work's for the local runner. However, it doesn't solve the issue with protobuf.

bhagyshricompany commented 1 year ago

thanks for informing will check and update you

bhagyshricompany commented 1 year ago

use this protobuf~=3.19.3 in requirements.txt file as its working fine .We are investigating for latest version why its failing. Thanks

ghost commented 1 year ago

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

Azure / azure-functions-python-worker