search

Azure / azure-functions-python-worker

Python worker for Azure Functions.
http://aka.ms/azurefunctions
MIT License
336 stars 104 forks source link

ModuleNotFoundError: No module named 'azure.storage 'error #561

Closed Viswa-Ranjan-Sahu closed 5 years ago

Viswa-Ranjan-Sahu commented 5 years ago

Investigative information

Please provide the following:

Repro steps

Provide the steps required to reproduce the problem:

Expected behavior

Provide a description of the expected behavior.

Actual behavior

Provide a description of the actual behavior observed.

Known workarounds

Provide a description of any known workarounds.

Related information

Provide any related information

Viswa-Ranjan-Sahu commented 5 years ago

Hi All, i had an python application which i converted into an azure function and trying to run from local using azure function core tools and VsCode . The debug flow constantly stops at
"from azure.storage.blob import BlockBlobService, ContainerPermissions" code section where it errors out saying

[10/10/2019 2:44:40 PM] Executing 'Functions.scrape_fund_pdfs' (Reason='New queue message detected on 'samplequeue'.', Id=d8e4171b-11e4-48aa-863a-2ba21fd00874) [10/10/2019 2:44:40 PM] Trigger Details: MessageId: c8328c4a-f870-4902-8ad1-ebb718f394f4, DequeueCount: 5, InsertionTime: 10/10/2019 2:44:39 PM +00:00 [10/10/2019 2:44:40 PM] Executed 'Functions.scrape_fund_pdfs' (Failed, Id=d8e4171b-11e4-48aa-863a-2ba21fd00874) [10/10/2019 2:44:40 PM] System.Private.CoreLib: Exception while executing function: Functions.scrape_fund_pdfs. System.Private.CoreLib: Result: Failure Exception: ModuleNotFoundError: No module named 'azure.storage' Stack: File "C:\Users\vsahu003\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\python\deps\azure_functions_worker\dispatcher.py", line 240, in _handlefunction_load_request func_request.metadata.entry_point) File "C:\Users\vsahu003\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\python\deps\azure_functions_worker\loader.py", line 66, in load_function mod = importlib.import_module(fullmodname) File "C:\Program Files\Python36\lib\importlib__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "E:\DIP_PwC_FundAudit_DSSln\scrape_fund_pdfs__init__.py", line 5, in from scrape_fund_pdfs.extract_data.main import main as prog_main File "E:\DIP_PwC_FundAudit_DSSln\scrape_fund_pdfs\init__.py", line 5, in from scrape_fund_pdfs.extract_data.main import main as prog_main File "E:\DIP_PwC_FundAudit_DSSln\scrape_fund_pdfs\extract_data\main.py", line 4, in from azure.storage.blob import BlockBlobService, ContainerPermissions.

Now i am using a virtual env based on python 3.6.8 and all my storage packages are also available in the path. I have selected the correct interpreter as the Vnev and can physically check that the storage packages are available .

this is my requirements .txt file which clearly has azure storage . Also as i said , i can see the blockblobservice .py in the lib->site-packages->azure->storage->blob path.not sure why it is failing with the issue . When i do run the same command on command prompt by activating the venv , it runs fine without issue. i guess it is more to do with azure functions than python. absl-py==0.7.1 applicationinsights==0.11.9 asn1crypto==0.24.0 astor==0.8.0 azure-common==1.1.23 azure-functions==1.0.0b5 azure-storage-blob==2.1.0 azure-storage-common==2.1.0 beautifulsoup4==4.8.0 certifi==2019.6.16 cffi==1.12.3 chardet==3.0.4 cryptography==2.7 cycler==0.10.0 Cython==0.29.13 gast==0.2.2 ghostscript==0.6 grpcio==1.23.0 h5py==2.9.0 idna==2.8 Keras-Applications==1.0.8 Keras-Preprocessing==1.1.0 kiwisolver==1.1.0 Markdown==3.1.1 matplotlib==3.1.1 mock==3.0.5 numpy==1.17.0 pandas==0.25.0 pdf2image==1.6.0 pdfminer.six==20181108 Pillow==6.1.0 protobuf==3.9.1 pycparser==2.19 pycryptodome==3.8.2 pyparsing==2.4.2 python-dateutil==2.8.0 pytz==2019.2 requests==2.22.0 six==1.12.0 sortedcontainers==2.1.0 soupsieve==1.9.3 tensorboard==1.13.1 tensorflow==1.13.1 tensorflow-estimator==1.13.0 termcolor==1.1.0 tika==1.19 urllib3==1.25.3 Werkzeug==0.15.5 python-dotenv==0.7.1

anirudhgarg commented 5 years ago

Please make sure to update your CoreTools version to the latest

Viswa-Ranjan-Sahu commented 5 years ago

HI Anirudh , i have just setup the whole setup yesterday so i assume the core tools should be latest steps done:

  1. got node.js
  2. npm install -g azure-functions-core-tools this is as of yesterday ,is there an update command or version which i should try to get. I am pretty sure it ia Az Fn issue as , the normal python without the AZFn form works fine.
Viswa-Ranjan-Sahu commented 5 years ago

Quick Update i just tried to update AZFn core function tools using install or update core tools .But still the issue remains.

azure-functions-core-tools@2.7.1704 postinstall C:\Users\vsahu003\AppData\Roaming\npm\node_modules\azure-functions-core-tools node lib/install.js

attempting to GET "https://functionscdn.azureedge.net/public/2.7.1704/Azure.Functions.Cli.win-x64.2.7.1704.zip"

azure-functions-core-tools@2.7.1704 postinstall C:\Users\vsahu003\AppData\Roaming\npm\node_modules\azure-functions-core-tools node lib/install.js

attempting to GET "https://functionscdn.azureedge.net/public/2.7.1704/Azure.Functions.Cli.win-x64.2.7.1704.zip"

anirudhgarg commented 5 years ago

Core tools look correct. Did you do a pip install on the requirements.txt when not using venv ? If not, please try that first.

Viswa-Ranjan-Sahu commented 5 years ago

Tried that too and the exact same issue still exists. steps:

deactivated the venev deleted the whole venv folder recreated the venv without activating, went to venev\scripts> pip install -r path\requirements.txt post everything started vscode and initiated the function and still the same issue

is it running on your side on local? i am literally confounded

maiqbal11 commented 5 years ago

@Viswa-Ranjan-Sahu, you are using the pip install from within the venv it seems. You would need to run python -m pip install -r requirements.txt from with a non-venv directory.

Viswa-Ranjan-Sahu commented 5 years ago

Quick Update: Thnx Maiqbal11 ,your trick worked and the function App started working locally. However, Now i am thinking, isn't it that the whole purpose of having a Virtualenv is defeated if i have to restore all related packages in virtualenv as well as in global env. Also, what happens if my virtualenv has a lower version of packages (3.6) as compared to 3.7 in global env? steps:

  1. open CMD in admin mode
  2. pip install -r pathToRequirements.txt\requirements.txt
  3. start azFn in VScode and select the venv and BOOM!!
anirudhgarg commented 5 years ago

Well you have to decide whether you want to use VirtualEnv or Global Env. By definition Virtual Environments are there in the first place so that you can have your own environment separate from the global environment. We have done the work that you dont actually need the VirtualEnv to do local debugging.

mail2atulm commented 4 years ago

@Viswa-Ranjan-Sahu, you are using the pip install from within the venv it seems. You would need to run python -m pip install -r requirements.txt from with a non-venv directory.

This worked for me

zaiquiri72 commented 4 years ago

@Viswa-Ranjan-Sahu nearly a year later and I am experiencing the exact same issue you reported.

Although the pip -m approach suggested by @anirudhgarg and @mail2atulm seems to fix the issue in the short term, I'm not very comfortable with that solution.

One of the things I observed is that although my Azure function will not debug locally due to the unfound package issue, it runs perfectly when deployed to my function app in Azure. The packages deployed to the Azure function app appear to be controlled by the virtual environment that's specific to the VSCode project (exactly as you would expect).

The root cause of the problem appears to be the fact that the local debugging environment and the deployed Azure environment (for the same code version) are not the same. @anirudhgarg if you happen to read this, if that's indeed the case I can't underscore how serious this issue is, it means that we can't trust that things which run/debug successfully in our local environment will run successfully when deployed up to Azure (and vice-versa, we can't be assured that anything which fails locally won't actually work when deployed to Azure). It really undercuts the whole purpose of having a local test/debug environment in the first place.

Digging a little deeper to try to understand what's going on here, I noticed that when I'm inside VSCode working on my function application, the blue status bar at the bottom indicates that VSCode is using Python 3.8.2 64 bit, and that the virtual environment is '.venv", as expected. If I run python --version in the VSCode terminal window, it reports as expected that the Python version is 3.8.2 64 bit. Also, my command prompt in the terminal window correctly indicates that my virtual environment is set to .venv.

However..

If I fire up "debug" or "run" from the IDE menu, or execute func start from the terminal window, I get the message:

Found Python version 3.6.8 :: Anaconda, Inc. (python3.6)

Which begs the question, why is func start out "looking" for a version of Python at all?

Shouldn't it be using the one that's specified by the virtual environment that's associated with the VSCode workspace, since that's the environment that every other part of the IDE is using and/or referencing?

A little research proves that the version of Python that func start is finding on my machine is the default python version when no virtual environment is activated, which means func start is either oblivious to, or is not not respecting, the virtual environment setting of the workspace at all.

I'm not sure if there's some setting in VSCode or elsewhere that's controlling this behavior. If there is, maybe @anirudhgarg can give us some pointers on what it is and how to change it.

anirudhgarg commented 4 years ago

@zaiquiri72 - thanks for your question. Firstly I want to assure you that the actual function host and language worker bits that are running on your local box are exactly the same as the ones running in the cloud. The issues that people seem to be hitting are because of the use of virtual environments or different environments like Ananconda. Firstly in Azure Functions we do not support Ananconda. Second virtual environments exist to replace the global environments if that is indeed you want that - they are a pure local convenience. We are not using virtual environments in the cloud. The workaround that was given is the best as it installs the packages globally and that is also what is used in the cloud.

themaxx75 commented 4 years ago

If we need to use a virtual environment, what is the process for it to work? I work on multiple projects with different versions of python and packages, and virtual environments are mandatory in my case. It seems to me that running func start is not using currently activated python. I would expect python3 -m venv .venv and source .venv/bin/activate and pip install -r requirements.txt and func start to work, but it fails when importing packages that are installed in the activated virtual env. Thanks