search

Azure / azure-sdk-for-python

This repository is for active development of the Azure SDK for Python. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/python/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-python.
MIT License
4.57k stars 2.79k forks source link

ImportError: cannot import name 'ArrowDialect' #14459

Closed frammnm closed 3 years ago

frammnm commented 3 years ago

Describe the bug "from azure.storage.filedatalake import DataLakeServiceClient" gives "ImportError: cannot import name 'ArrowDialect'", trying to work with datalake gen2 snapshots, tried the blob sdk but its not working there, i read somewhere that this library is the proper one to work with datalake. Im fairly certain that is a python libraries related problem, but i have no clue what the correct versions should be, i have been working with different libraries and components of azure for more than a year and some version changes seem to break things in other SDKs. How could i approach this? the full stacktrace:

Traceback (most recent call last): File ".\snapshot_test.py", line 84, in from azure.storage.filedatalake import DataLakeServiceClient File "C:\Users\user1\AppData\Local\Programs\Python\Python36-32\lib\site-packages\azure\storage\filedatalake__init__.py", line 7, in from ._download import StorageStreamDownloader File "C:\Users\user1\AppData\Local\Programs\Python\Python36-32\lib\site-packages\azure\storage\filedatalake_download.py", line 6, in from ._deserialize import from_blob_properties File "C:\Users\user1\AppData\Local\Programs\Python\Python36-32\lib\site-packages\azure\storage\filedatalake_deserialize.py", line 15, in from ._models import FileProperties, DirectoryProperties, LeaseProperties File "C:\Users\user1\AppData\Local\Programs\Python\Python36-32\lib\site-packages\azure\storage\filedatalake_models.py", line 18, in from azure.storage.blob import ArrowDialect as BlobArrowDialect ImportError: cannot import name 'ArrowDialect'

To Reproduce Steps to reproduce the behavior:

  1. Import library
  2. Execute import

Expected behavior Expected to import with no problems, it seems there are clashing libraries.

Additional context Relevant context would be my pip freeze:

PS {path} > pip freeze adal==1.2.2 aiohttp==3.6.2 astroid==2.3.3 async-timeout==3.0.1 attrs==19.3.0 azure==4.0.0 azure-applicationinsights==0.1.0 azure-batch==4.1.3 azure-common==1.1.18 azure-core==1.8.2 azure-cosmosdb-nspkg==2.0.2 azure-cosmosdb-table==1.0.6 azure-datalake-store==0.0.48 azure-eventgrid==1.3.0 azure-eventhub==1.3.1 azure-eventhub-checkpointstoreblob-aio==1.0.0 azure-functions==1.0.5 azure-graphrbac==0.40.0 azure-keyvault==1.1.0 azure-loganalytics==0.1.0 azure-mgmt==4.0.0 azure-mgmt-advisor==1.0.1 azure-mgmt-applicationinsights==0.1.1 azure-mgmt-authorization==0.50.0 azure-mgmt-batch==5.0.1 azure-mgmt-batchai==2.0.0 azure-mgmt-billing==0.2.0 azure-mgmt-cdn==3.1.0 azure-mgmt-cognitiveservices==3.0.0 azure-mgmt-commerce==1.0.1 azure-mgmt-compute==4.6.2 azure-mgmt-consumption==2.0.0 azure-mgmt-containerinstance==1.5.0 azure-mgmt-containerregistry==2.8.0 azure-mgmt-containerservice==4.4.0 azure-mgmt-cosmosdb==0.4.1 azure-mgmt-datafactory==0.6.0 azure-mgmt-datalake-analytics==0.6.0 azure-mgmt-datalake-nspkg==3.0.1 azure-mgmt-datalake-store==0.5.0 azure-mgmt-datamigration==1.0.0 azure-mgmt-devspaces==0.1.0 azure-mgmt-devtestlabs==2.2.0 azure-mgmt-dns==2.1.0 azure-mgmt-eventgrid==1.0.0 azure-mgmt-eventhub==2.6.0 azure-mgmt-hanaonazure==0.1.1 azure-mgmt-iotcentral==0.1.0 azure-mgmt-iothub==0.5.0 azure-mgmt-iothubprovisioningservices==0.2.0 azure-mgmt-keyvault==1.1.0 azure-mgmt-loganalytics==0.2.0 azure-mgmt-logic==3.0.0 azure-mgmt-machinelearningcompute==0.4.1 azure-mgmt-managementgroups==0.1.0 azure-mgmt-managementpartner==0.1.1 azure-mgmt-maps==0.1.0 azure-mgmt-marketplaceordering==0.1.0 azure-mgmt-media==1.0.0 azure-mgmt-monitor==0.5.2 azure-mgmt-msi==0.2.0 azure-mgmt-network==2.7.0 azure-mgmt-notificationhubs==2.1.0 azure-mgmt-nspkg==3.0.2 azure-mgmt-policyinsights==0.1.0 azure-mgmt-powerbiembedded==2.0.0 azure-mgmt-rdbms==1.9.0 azure-mgmt-recoveryservices==0.3.0 azure-mgmt-recoveryservicesbackup==0.3.0 azure-mgmt-redis==5.0.0 azure-mgmt-relay==0.1.0 azure-mgmt-reservations==0.2.1 azure-mgmt-resource==2.2.0 azure-mgmt-scheduler==2.0.0 azure-mgmt-search==2.1.0 azure-mgmt-servicebus==0.5.3 azure-mgmt-servicefabric==0.2.0 azure-mgmt-signalr==0.1.1 azure-mgmt-sql==0.9.1 azure-mgmt-storage==2.0.0 azure-mgmt-subscription==0.2.0 azure-mgmt-trafficmanager==0.50.0 azure-mgmt-web==0.35.0 azure-nspkg==3.0.2 azure-servicebus==0.21.1 azure-servicefabric==6.3.0.0 azure-servicemanagement-legacy==0.20.6 azure-storage-blob==12.5.0 azure-storage-common==1.4.0 azure-storage-file==1.4.0 azure-storage-file-datalake==12.2.0b1 azure-storage-queue==1.4.0 cachetools==3.1.1 certifi==2019.11.28 cffi==1.12.2 chardet==3.0.4 Click==7.0 colorama==0.4.1 configparser==4.0.2 cryptography==2.8 databricks-cli==0.9.1 future==0.18.2 google-api-python-client==1.7.11 google-auth==1.7.1 google-auth-httplib2==0.0.3 httplib2==0.14.0 idna==2.8 idna-ssl==1.1.0 isodate==0.6.0 isort==4.3.21 lazy-object-proxy==1.4.3 mccabe==0.6.1 msrest==0.6.19 msrestazure==0.6.0 multidict==4.7.4 oauth2client==4.1.3 oauthlib==3.0.1 pyasn1==0.4.8 pyasn1-modules==0.2.7 pycparser==2.19 PyJWT==1.7.1 pylint==2.4.4 pyodbc==4.0.28 python-certifi-win32==1.4 python-dateutil==2.8.0 python-tds==1.10.0 python-twitter==3.5 requests==2.21.0 requests-oauthlib==1.2.0 rsa==4.0 setuptools-scm==4.1.2 six==1.14.0 tabulate==0.8.6 typed-ast==1.4.0 typing-extensions==3.7.4.1 uamqp==1.1.0 uritemplate==3.0.0 urllib3==1.24.1 virtualenv==16.7.9 wrapt==1.11.2 yarl==1.4.2

ghost commented 3 years ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @southpolesteve, @zfoster

ghost commented 3 years ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @sumantmehtams.

ghost commented 3 years ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

frammnm commented 3 years ago

I have been testing and creating a new and clean python env theres no clashing libraries so the problem doesn't exist, most probably the problem was related to azure-datalake-store==0.0.48, azure-storage-common==1.4.0 or azure-storage-file==1.4.0. Im closing the issue.

aminekaabachi commented 3 years ago

I still have the same issue:

pip freeze

aiohttp==3.7.2 async-timeout==3.0.1 attrs==20.2.0 azure-core==1.8.2 azure-functions==1.4.0 azure-functions-durable==1.0.0b10 azure-storage-blob==12.4.0 azure-storage-file-datalake==12.1.2 certifi==2020.6.20 cffi==1.14.3 chardet==3.0.4 cryptography==3.2 environs==8.0.0 furl==2.1.0 idna==2.10 isodate==0.6.0 marshmallow==3.8.0 msrest==0.6.19 multidict==5.0.0 oauthlib==3.1.0 orderedmultidict==1.0.1 pycparser==2.20 pyfiglet==0.8.post1 python-dateutil==2.8.1 python-dotenv==0.14.0 requests==2.24.0 requests-oauthlib==1.3.0 six==1.15.0 tweepy==3.9.0 typing-extensions==3.7.4.3 urllib3==1.25.11 yarl==1.6.2

frammnm commented 3 years ago

I still have the same issue:

pip freeze

aiohttp==3.7.2 async-timeout==3.0.1 attrs==20.2.0 azure-core==1.8.2 azure-functions==1.4.0 azure-functions-durable==1.0.0b10 azure-storage-blob==12.4.0 azure-storage-file-datalake==12.1.2 certifi==2020.6.20 cffi==1.14.3 chardet==3.0.4 cryptography==3.2 environs==8.0.0 furl==2.1.0 idna==2.10 isodate==0.6.0 marshmallow==3.8.0 msrest==0.6.19 multidict==5.0.0 oauthlib==3.1.0 orderedmultidict==1.0.1 pycparser==2.20 pyfiglet==0.8.post1 python-dateutil==2.8.1 python-dotenv==0.14.0 requests==2.24.0 requests-oauthlib==1.3.0 six==1.15.0 tweepy==3.9.0 typing-extensions==3.7.4.3 urllib3==1.25.11 yarl==1.6.2

i fixed my issue with a new and clean , it seems some azure libraries can affect others in a bad way when together i bet your issue can be fixed with this. @aminekaabachi

Fl4v commented 3 years ago

I also encountered the same issue when I installed: azure-storage-file-datalake==12.2.0b1. I resolved it by manually installing: azure-storage-blob==12.6.0b1, I believe that pip won't install the latter since it's a pre-release version, so it falls back to the more recent stable version 12.5.0.