BlockCache::OpenFile : Failed to get block list

[Niko-CZ]

Which version of blobfuse was used?

blobfuse2-2.3.2-1.x86_64 fuse-2.9.2-11.el7.x86_64 fuse3-3.6.1-4.el7.x86_64 fuse3-libs-3.6.1-4.el7.x86_64

Which OS distribution and version are you using?

Red Hat Enterprise Linux release 8.9 (Ootpa) x86_64 4.18.0-513.11.1.el8_9.x86_64 #1 SMP Thu Dec 7 03:06:13 EST 2023 x86_64 x86_64 x86_64 GNU/Linux

If relevant, please share your mount command.

/usr/bin/blobfuse2 mount /oracle/Blob_BAK --config-file=/etc/Blobfuse2-BlockCacheConfig.yaml; cat /etc/Blobfuse2-BlockCacheConfig.yaml; allow-other: true logging: type: base level: log_debug file-path: /var/log/blobfuse2/blobfuse2.log max-file-size-mb: 64 file-count: 2 track-time: true components: -libfuse -block_cache -attr_cache -azstorage health_monitor: enable-monitoring: true stats-poll-interval-sec: 10 process-monitor-interval-sec: 30 output-path: /mnt/resource/blobfuse2_healthmon monitor-disable-list: -blobfuse_stats libfuse: fuse-trace: true default-permission: 0777 attribute-expiration-sec: 240 entry-expiration-sec: 240 negative-entry-expiration-sec: 120 ignore-open-flags: true block_cache: block-size-mb: 256 mem-size-mb: 20480 prefetch: 20 parallelism: 128 attr_cache: timeout-sec: 7200 azstorage: type: block account-name: xxx endpoint: xxx mode: msi container: xxx honour-acl: false

What was the issue encountered?

Tue Oct 8 13:37:50 UTC 2024 : blobfuse2[159980] : [/oracle/Blob_BAK] LOG_DEBUG [utils.go (156)]: SDK(Retry) : response 200 Tue Oct 8 13:37:50 UTC 2024 : blobfuse2[159980] : [/oracle/Blob_BAK] LOG_DEBUG [utils.go (156)]: SDK(Retry) : exit due to non-retriable status code Tue Oct 8 13:37:50 UTC 2024 : blobfuse2[159980] : [/oracle/Blob_BAK] LOG_ERR [block_cache.go (349)]: BlockCache::OpenFile : Failed to get block list of X28_3965183084/X28_3965183084_20241008_51_1 [<nil>] Tue Oct 8 13:37:50 UTC 2024 : blobfuse2[159980] : [/oracle/Blob_BAK] LOG_ERR [libfuse_handler.go (712)]: Libfuse::libfuse_open : Failed to open X28_3965183084/X28_3965183084_20241008_51_1 [failed to retrieve bloc k list for X28_3965183084/X28_3965183084_20241008_51_1]

In the same time file is being created: ls -lsa /oracle/Blob_BAK/X28_3965183084/X28_3965183084_20241008_51_1 0 -rwxrwxrwx 1 root root 67117056 Oct 8 13:37 /oracle/Blob_BAK/X28_3965183084/X28_3965183084_20241008_51_1

blobfuse2_log_block_cache.log We have backups working with the same configuration, but the streaming cache instead of block. After reading of fresh release notes (https://github.com/Azure/azure-storage-fuse/blob/main/README.md):

streaming mode is being deprecated.

I decided to enable block caching changing configuration part: stream: block-size-mb: 8 max-buffers: 16 buffer-size-mb: 384 To: block_cache: block-size-mb: 256 mem-size-mb: 20480 prefetch: 20 parallelism: 128 But facing issue above with any combination of parameters in block_cache section.

Have you found a mitigation/solution?

No.

Please share logs if available.

File blobfuse2_log_block_cache.log is attached.

[ashruti-msft]

In the same time file is being created:

You mean to say that the file was created successfully but you saw that error message in the logs, is that correct?

[Niko-CZ]

In the same time file is being created:

You mean to say that the file was created successfully but you saw that error message in the logs, is that correct?

Yes, file created successfully with the 64M size.

[bharath-krishna]

Hi,

I am also facing exactly similar issue. Unable to list/create any folders in the container. Here is a debug log,

Mon Nov 11 07:44:50 UTC 2024 : blobfuse2[10940] : [/mnt/XXXXXX] LOG_DEBUG [utils.go (156)]: SDK(Response) : ==> REQUEST/RESPONSE (Try=1/12.735586ms, OpTime=12.753664ms) -- RESPONSE RECEIVED
   HEAD https://XXXXXX.blob.core.windows.net/XXXXXX/data
   Accept: application/xml
   Authorization: REDACTED
   User-Agent: Azure-Storage-Fuse/2.3.2 (Ubuntu 22.04.5 LTS) azsdk-go-azblob/v1.4.0 (go1.22.4; linux)
   X-Ms-Date: Mon, 11 Nov 2024 07:44:50 GMT
   x-ms-version: 2024-05-04
   --------------------------------------------------------------------------------
   RESPONSE Status: 200 OK
   Accept-Ranges: bytes
   Content-Length: 0
   Date: Mon, 11 Nov 2024 07:44:50 GMT
   Etag: "0x8DCE72D5A742E74"
   Last-Modified: Tue, 08 Oct 2024 00:08:35 GMT
   Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
   X-Ms-Acl: user::rwx,group::r-x,other::---
   X-Ms-Blob-Type: BlockBlob
   X-Ms-Creation-Time: Tue, 08 Oct 2024 00:08:35 GMT
   X-Ms-Group: $superuser
   X-Ms-Lease-State: available
   X-Ms-Lease-Status: REDACTED
   X-Ms-Meta-Hdi_isfolder: REDACTED
   X-Ms-Owner: $superuser
   X-Ms-Permissions: rwxr-x---
   X-Ms-Request-Id: b2393362-b01e-0039-440d-34bc54000000
   X-Ms-Resource-Type: directory
   X-Ms-Server-Encrypted: REDACTED
   X-Ms-Version: 2024-05-04

Mon Nov 11 07:44:50 UTC 2024 : blobfuse2[10940] : [/mnt/XXXXXX] LOG_DEBUG [utils.go (156)]: SDK(Retry) : response 200
Mon Nov 11 07:44:50 UTC 2024 : blobfuse2[10940] : [/mnt/XXXXXX] LOG_DEBUG [utils.go (156)]: SDK(Retry) : exit due to non-retriable status code

Is there any fix for this?

[vibhansa-msft]

Can you share the complete log file with debug logs enabled. Above log shows success reply from the backend and does not point to any error condition. Also, can you share list does not work only for some time after mount or it does not work ever ?