search

MicrosoftDocs / azure-docs

Open source documentation of Microsoft Azure
https://docs.microsoft.com/azure
Creative Commons Attribution 4.0 International
10.29k stars 21.47k forks source link

Error - module 'fsspec_wrapper' has no attribute 'AzureBlobFileSystem #90345

Closed VasanthiMudunuri closed 2 years ago

VasanthiMudunuri commented 2 years ago

Hi, I am trying fsspec to read/write to adls as mentioned here in the doc

But the path 'abfs[s]:///' (I followed this format for my path) is not working I tried 'abfs:///' it says 'module 'fsspec_wrapper' has no attribute 'AzureBlobFileSystem''. what must be the issue here could someone help?

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

CHEEKATLAPRADEEP-MSFT commented 2 years ago

@VasanthiMudunuri You need to pass the path as shown below:

abfs://<container>/<path-to-file>

image

Hope this helps.

PetrutOPopescu commented 2 years ago

Same problem on my side, couldn't find a fix. Maybe the fsspec version has changed?

VasanthiMudunuri commented 2 years ago

@CHEEKATLAPRADEEP-MSFT I even tried pandas way to read/write data but still the same error 'fsspec_wrapper' has no attribute AzureBlobFileSystem. I am trying the same format for my path with required container name and path to file - abfs:/// its not working.

PetrutOPopescu commented 2 years ago

@VasanthiMudunuri managed to solve it. in the Apache Spark Pools, find Packages within the name of the pool, and you need to add the latest fsspec version.

VasanthiMudunuri commented 2 years ago

@PetrutOPopescu I tried adding the latest version of fsspec in requirement.txt file as fsspec==2022.2.0 but that job failed. How to add the latest version in packages could you please let me know the steps. Thank you for taking time to respond.

PetrutOPopescu commented 2 years ago

@VasanthiMudunuri just put fsspec without ==.... because it will fetch automatically what it needs. and give it a bit of time to take effect.

VasanthiMudunuri commented 2 years ago

@PetrutOPopescu I tried it twice but the SystemReservedJob-LibraryManagement job is failing with below error. [plugins..systemreservedpool-librarymanagement.5 WorkspaceType: CCID:<>] [Monitoring] Livy Endpoint=[https://hubservice1.***]. Livy Id=[0] Job failed during run time with state=[dead]. I am not sure if this is environment issue as I don't see any issue with adding package it should actually work when I run below I am seeing fsspec version is old import pkg_resources for d in pkg_resources.working_set: print(d) Not sure how to update to latest @PetrutOPopescu did you see any such error before? @CHEEKATLAPRADEEP-MSFT can you also help with this?

PetrutOPopescu commented 2 years ago

@VasanthiMudunuri don't know what to say. maybe try to remove the whole configuration file to see if it works

CHEEKATLAPRADEEP-MSFT commented 2 years ago

@VasanthiMudunuri This document helps to install packages on Sparks pools.

We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.

siddharth-redseer commented 2 years ago

@CHEEKATLAPRADEEP-MSFT getting the same error on Azure synapse. fsspec version is 2022.7.1, it was working perfectly a month ago. I guess synapse spark fetched a newer version when I uploaded a new package list. I do see Azure blob in fsspec docs. How is synapse missing that!?

Hmr-ramzi commented 2 years ago

@siddharth-redseer I am experiencing the same issue on synapse. It was working perfectly fine like yesterday :D any idea about a possible resolution? I tried to upgrade the fsspec but looks like this was not enough to fix the problem. Any hint?

dj-petrovic commented 2 years ago

@siddharth-redseer I am experiencing the same issue on synapse. It was working perfectly fine like yesterday :D any idea about a possible resolution? I tried to upgrade the fsspec but looks like this was not enough to fix the problem. Any hint?

Could you try updating adlfs to latest version? I was reading trough some changset notes and see they added support for 3 days ago, could be connected. see https://github.com/fsspec/adlfs/blob/1d905740b8f1af3df83cce9262b9da9395ed2be8/CHANGELOG.md

siddharth-redseer commented 2 years ago

I changed the version to 22.5.0 and it’s working again.

From: Hmr-ramzi @.> Sent: Friday, September 9, 2022 5:58:57 PM To: MicrosoftDocs/azure-docs @.> Cc: Siddharth Poonia @.>; Mention @.> Subject: Re: [MicrosoftDocs/azure-docs] Error - module 'fsspec_wrapper' has no attribute 'AzureBlobFileSystem (Issue #90345)

@siddharth-redseerhttps://ind01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsiddharth-redseer&data=05%7C01%7Csiddharth.poonia%40redseerconsulting.com%7Cd12548f914214753362908da925edf23%7C00a9ff8c98304847ae514579ec092cb4%7C0%7C0%7C637983233420238302%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nKlfbIdwTacYQlRs3gzbn%2Fwm%2BIXZO6zYqzv8apPNwe0%3D&reserved=0 I am experiencing the same issue on synapse. It was working perfectly fine like yesterday :D any idea about a possible resolution? I tried to upgrade the fsspec but looks like this was not enough to fix the problem. Any hint?

— Reply to this email directly, view it on GitHubhttps://ind01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fazure-docs%2Fissues%2F90345%23issuecomment-1241917423&data=05%7C01%7Csiddharth.poonia%40redseerconsulting.com%7Cd12548f914214753362908da925edf23%7C00a9ff8c98304847ae514579ec092cb4%7C0%7C0%7C637983233420238302%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=91K8aS2ULnzJ1%2FYAVipMMrRS8Smx8wMtbG9D15WZ8n8%3D&reserved=0, or unsubscribehttps://ind01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAXOAOUAG53SDEMPOVGWGQU3V5MUQTANCNFSM5RPLGRKA&data=05%7C01%7Csiddharth.poonia%40redseerconsulting.com%7Cd12548f914214753362908da925edf23%7C00a9ff8c98304847ae514579ec092cb4%7C0%7C0%7C637983233420238302%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FOSBPpaKCdpuVn0il9MdBGtgC%2BVg%2BFVcBZkUwE%2B%2BFAs%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

Siddharth Poonia Data Engineer | Digital 91-7340495290 [Logo]https://redseer.com/ Bangalore | New Delhi | Mumbai | Dubai | Singapore | New York | London Disclaimer and confidentiality notice: This message contains information that may be confidential and proprietary. Unless you are the intended recipient (or authorized to receive this message for the intended recipient), you may not use, copy, disseminate or disclose to anyone the message or any information contained in the message. If you have received the message in error, please advise the sender by reply e-mail, and delete the message immediately.

FWuttke commented 2 years ago

I am facing the same issue. What did you change to this version @siddharth-redseer ?

Hmr-ramzi commented 2 years ago

I upgraded fsspec package to latest and now everything works fine

siddharth-redseer commented 2 years ago

I am facing the same issue. What did you change to this version @siddharth-redseer ?

install fsspec==22.5.0

siddharth-redseer commented 2 years ago

I upgraded fsspec package to latest and now everything works fine

@Hmr-ramzi synapse is not picking up the latest version

Hmr-ramzi commented 2 years ago

I installed pip install fsspec==2022.8.2 and my pipeline worked just fine

siddharth-redseer commented 2 years ago

Looks like the version needs to be specified otherwise it defaults to 2022.7.0

FWuttke commented 2 years ago

I don't know what I am doing wrong but still doesn't work for me... any ideas? image

Hmr-ramzi commented 2 years ago

You need to install the package by uploading a requirements.txt file to the spark pool server not within the spark session. Also give it some time to be applied before you pick up another spark session

FarBo commented 2 years ago

Just add fsspec in a text file (people usually call it requirements.txt) and upload it to your pool and save the changes in the pool. If you already have the requirement file in your pool, you need to add this package to that file and upload and save again. No need to specify the version of the package though.

keen85 commented 1 year ago

@Microsoft: Why do the Spark Pools not come with the correct version out of the box?

When having Data Exfiltration Protection (DEP) the process of updating Python packages (especially when there are dependencies) is not so easy...

ychen4-li commented 1 year ago

@microsoft: Why do the Spark Pools not come with the correct version out of the box?

When having Data Exfiltration Protection (DEP) the process of updating Python packages (especially when there are dependencies) is not so easy...

+1 to above, I am still seeing this nowadays. Although adding fsspec==xxx version workaround it, how can we make sure it's not coming back? @CHEEKATLAPRADEEP-MSFT