Getting issue while doing a coverity scan on blobfuse2 mounted directory Failed to acquire lock

[rahulshrivastava-eaton]

I am trying to run a synopsis coverity scan on a Linux machine , as i wanted to store the coverity scan results to be stored in cloud so i am using a blobfuse2 mounted directory (i checked for file permissions are correct) . I am able so successfully mount the directory , but when i am starting the coverity scan using the command cov-build -- then it is not giving me any errors in blobfuse2 logs but coverity scan is getting hang for 30mins and later giving the below error .

Error : Failed to acquire lock "/__w/test-actions/test-actions/idir_new_stream/emit/243b98c66385/emit-db.creation-lock". Unable to protect emit DB creation!

l checked for the file write permissions are there and there is no issues related to authentication

Logs from blobfuse2:

Mon Dec  2 07:10:45 EST 2024 : blobfuse2[32194] : [/home/administrator/blobfuse1] LOG_TRACE [block_blob.go (260)]: BlockBlob::DeleteFile : name emit/QA-GITHUB/emit-db.creation-lock-b72cf4020f6dfad85255871ba3e
Mon Dec  2 07:10:45 EST 2024 : blobfuse2[32194] : [/home/administrator/blobfuse1] LOG_DEBUG [utils.go (156)]: SDK(Retry) : =====> Try=1 for DELETE https://<>.blob.core.windows.net/coveriont1/emit%2QA-GITHUB%2Femit-db.creation-lock-b72cf4020f6dfabb8efca3e
Mon Dec  2 07:10:45 EST 2024 : blobfuse2[32194] : [/home/administrator/blobfuse1] LOG_DEBUG [utils.go (156)]: SDK(Request) : ==> OUTGOING REQUEST (Try=1)
   DELETE https://<>.blob.core.windows.net/coveritytempcont1/emit%2FPSPL-QA-GITHUB%2Femit-db.creation-lock-b72cf4020f6dfad85255871bb8efca3e
   Accept: application/xml
   Authorization: REDACTED
   User-Agent: Azure-Storage-Fuse/2.3.2 (Ubuntu 22.04.5 LTS) azsdk-go-azblob/v1.4.0 (go1.22.4; linux)
   X-Ms-Date: Mon, 02 Dec 2024 12:10:45 GMT
   x-ms-delete-snapshots: include
   x-ms-version: 2024-05-04

Mon Dec  2 07:10:45 EST 2024 : blobfuse2[32194] : [/home/administrator/blobfuse1] LOG_DEBUG [utils.go (156)]: SDK(Response) : ==> REQUEST/RESPONSE (Try=1/368.828002ms, OpTime=369.266698ms) -- RESPONSE RECEIVED
   DELETE https://<>.blob.core.windows.net/coveritytempcont1/emit%2FPSPL-QA-GITHUB%2Femit-db.creation-lock-b72cf4020f6dfad85255871bb8efca3e
   Accept: application/xml
   Authorization: REDACTED
   User-Agent: Azure-Storage-Fuse/2.3.2 (Ubuntu 22.04.5 LTS) azsdk-go-azblob/v1.4.0 (go1.22.4; linux)
   X-Ms-Date: Mon, 02 Dec 2024 12:10:45 GMT
   x-ms-delete-snapshots: include
   x-ms-version: 2024-05-04
   --------------------------------------------------------------------------------
   RESPONSE Status: 202 Accepted
   Content-Length: 0
   Date: Mon, 02 Dec 2024 12:10:46 GMT
   Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
   X-Ms-Delete-Type-Permanent: true
   X-Ms-Request-Id: 52707bb0-d01e-000f-01b3-441c92000000
   X-Ms-Version: 2024-05-04

I saw that i am getting the 202 here

[vibhansa-msft]

Error : Failed to acquire lock "/__w/test-actions/test-actions/idir_new_stream/emit/243b98c66385/emit-db.creation-lock". Unable to protect emit DB creation!

This error is coming from your application and not directly from Blobfuse2. 202 is a success response only. In your blobfuse log file can you check what kind of error you are getting in blobfuse ?

[rahulshrivastava-eaton]

@vibhansa-msft coverity application is running fine if i am using a local directory only issue happens when i use the blobfuse mounted directory , i saw that 202 is ok and success response. looks like even after a 202 response request to delete /create the log files via coverity (file : emit-db.creation-lock-b72cf4020f6dfad85255871bb8efca3e) files are not getting deleted as i can see them via azure portal . As a azure says 202 response on a delete request means deletion request has been submitted to server successfully but it does not guarantees that resource has been deleted. Is there some way to drill down more to get to root cause of this , i appreciate your help in understanding the issue .

[vibhansa-msft]

From the tool side we can only put a request across to backend. Whether backend deletes the file immediately or schedules it for deletion after some time is totally out of our control. I will try to reach out to backend team to better understand the scenario but from blobfuse side we have no control over this.

That being said, how does this impact your flow or failure in Coverity as '202` is still a success for you. Reason for failure might be something else for which we shall investigate more on logs to see which operation actually failed.

[vibhansa-msft]

Can you confirm couple of things:

• Is your account HNS enabled?
• Is soft-delete enabled on your account?
• Is blob versioning enabled on your account?

When you say delete request was posted and you got 202 but still the blob existed on the portal. Does it get deleted after some time you never see it being deleted?

[rahulshrivastava-eaton]

@vibhansa-msft Is your account HNS enabled? - not enabled , I am not very sure though since i can't see HNS settings for me, I saw there a option 'Data Lake Gen 2 updrade' for my storage account, and as per google Hierarchical Namespace (HNS) feature is specifically designed for Azure Data Lake Storage Gen2, so i can assume that this is not enabled

Is soft-delete enabled on your account? - not enabled
Is blob versioning enabled on your account? - not enabled

[vibhansa-msft]

In portal you can go to overview tab of your storage account and there shall be in option of "Hierarchial Name space" and you need to check whether it says enabled or not.

[vibhansa-msft]

And how about the other question. File deleted but still visible in portal, does it disappear after some time or just remains for eternity.

[rahulshrivastava-eaton]

@vibhansa-msft yes Hierarchical namespace is Disabled and yes deleted files are remains for eternity

[jainakanksha-msft]

@rahulshrivastava-eaton can you please cross-verify the container name/filename, if you are cross-verifying at the right place.

[vibhansa-msft]

@vibhansa-msft yes Hierarchical namespace is Disabled and yes deleted files are remains for eternity

Does that mean backend returned 202 Accepted but files were never deleted? That is highly unlikely to happen, there is something else wrong if that's the observation. From blobfuse end as well we have never seen any such bug report by any of our customers in the past.

[rahulshrivastava-eaton]

ok @vibhansa-msft i will check again and verify again